New answers tagged robots.txt
2
I know you previously asked to block the bots and now you just want to slow them down.
The only way to force them to slow down is not to serve them content if they are going too fast. The problem with not serving content is it will surely have negative consequences to your search engine rankings.
In case you decide to block some of the less important bots, ...
2
Google sends out a notice whenever it has issues connecting to your site.
The message will be similar to:
Over the last 24 hours, Googlebot encountered 205 errors while
attempting to connect to your site.
I've seen it send notices for outages less than 1 hour but still reference the 24 hour window. If you only received one message on one day, then ...
1
Your robots.txt is just fine. However, it might not be enough to totally prevent indexing: Disallow command in robots.txt will block crawling, but in some cases the URLs themselves will still be indexed because of links or other factors.
Robots.txt is not meant to prevent the indexing of URLs, its purpose is to prevent crawling.
Best way to prevent Google ...
0
The simplest solution would be to disable the 'direct' subdomain. If however you want to use that subdomain you would have to use a more creative approach.
One way to do it is to have a dynamic robots.txt. When the web spider requests robots.txt we redirect it to our dynamic robots page. If the subdomain matches our criteria we send a 'disallow' otherwise ...
0
Your robots.txt tells all bots (User-agent: *) not (Disallow:) to crawl any URL starting with a slash (/), which would be all URLs, of course.
If you want to allow all bots crawling all your pages, remove the slash:
User-agent: *
Disallow:
Note that Allow: is not part of the original robots.txt specification, however, some/many crawlers recognize it. ...
1
Your robots.txt own these lines:
User-agent: *
Disallow: /
It indicates to Googlebot and other search spiders to not index your website.
Just put these lines instead and wait:
User-agent: *
Allow: /
That way, all search spiders (Googlebot included) will index your website.
0
I can confirm that your robots.txt and HTML are not blocking Google from scanning your site, so whats the problem? In short... time.
Google may state that it takes two days for Google to take note of a robots.txt but in no way does this imply that all these pages blocked previously will be unblocked and instantly crawled.
Now that Google can crawl your ...
0
I suspect that with Facebook you meant the facebookexternalhit user-agent string that appears in your access logs? This is not a crawler and as such doesn't respect (or indeed needs to, but that's argumentative) restrictions in robots.txt. This answer from Jeff Sherlock pretty much explains their position on it. So having this in mind, you could simply ...
1
You seem to be suffering from an overly broad rule. To target Google web-search explicitly, you have to put rules that prevent googlebot from crawling whichever directory that you want blocked. With a targeted disallow rule, there is not need to do anything for Facebook.
Now, if the general case is to block everyone but Facebook, you need to put a disallow ...
2
You'll want both in the root of the site, especially robots.txt because that's where Bot's will be looking for it. The Sitemap could go anywhere, but it would make the most sense in your case to put it in the root, rather than having several sitemaps for each additional language you add in the future.
2
The meta tags are sometimes useful as they offer more fine-grained guidelines for robots than robots.txt. They can also be used if you do not have access to the root directory of the server.
However, the robots.txt is more powerful, as it can be used to restrict access to entire directories (while the scope of a meta tag is always constrained to the single ...
0
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710
The noindex meta standard is useful if you don't have root access to
your server, as it allows you to control access to your site on a
page-by-page basis.
This tag is still useful and used by many plugins in WordPress, Joomla, Drupal and so on.. For example Yoast SEO for ...
-2
The html meta index tags have been depreciated in favor of using a robots.txt
As for workflow the nice thing about the robots.txt file is that you can just update the txt file if you want to make a change, so you only have to make it once, for example if you wanted to noindex the whole site this could be done with 2 lines of code, were as with the meta tags ...
0
I'm seeing the same issue across multiple dedicated servers in different datacenters.
Matt Cutts posted on his blog that this looks like a Google issue. I wouldn't freak out unless it continues for another couple days.
0
Problem comes from encoding of robots.txt file. UTF-8 encoding is recommended by Google.
Top 50 recent answers are included
