Hot answers tagged robots.txt
14
As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder.
Here are all the ones I know of:
favicon.ico
Gives your pages an icon in tabs, bookmarks, etc.
robots.txt
Useful ...
13
I would strongly recommend registering your site with webmaster tools. There is a crawler access section under site configuration that will tell you when your robots.txt was last downloaded. The tool also provides a lot of detail as to how the crawlers are seeing your site, what is blocked or not working, and where you are appearing in queries on Google.
...
12
There are 2 main ways to prevent search engines from indexing specific pages:
A Robots.txt file for your domain.
The Meta Robots tag on each page.
Robots.txt should be your first stop for URL patterns that match several files. You can see the syntax here and more detailed here. The robots.txt file must be placed in the root folder of your domain, i.e. at ...
10
Hard to say for sure why you aren't indexed yet but:
1) Number of visitors has no bearing whatsoever on your indexing or rankings. Google doesn't know this information and, even if it they did, it really offers nothing in terms of relevance of any page for search.
2) 20 backlinks is hardly a lot. Even then, unless Google knows about those backlinks they ...
10
You don't want the page to appear in the SERPs at all...
Don't disallow in robots.txt. Add a noindex meta tag (or X-Robots-Tag HTTP header) to your pages instead.
As j0k suggests, your pages could be found somehow. Stats reports, directory listings, etc...
Disallowing in robots.txt prevents the page from being crawled, but could still be indexed and could ...
9
Maybe someone didn't want to pay for spider traffic?
Regardless, you are reading it correctly:
http://www.robotstxt.org/robotstxt.html
Web site owners use the /robots.txt file to give instructions about
their site to web robots; this is called The Robots Exclusion
Protocol. It works likes this: a robot wants to vists a Web site
URL, say ...
9
You really only need the disallow. Search engine crawlers will automatically assume they are allowed everywhere that isn't disallowed.
User-agent: *
Disallow: /templates_c
But to answer your question, according to Google:
At a group-member level, in particular for allow and disallow
directives, the most specific rule based on the length of the ...
9
No. See the Google Robots.txt specifications, specifically the File Formats section.
The expected file format is plain text encoded in UTF-8. The file consists of records (lines) separated by CR, CR/LF or LF.
Only valid records will be considered; all other content will be ignored. For example, if the resulting document is a HTML page, only valid ...
8
No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed.
However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as ...
7
It seems that Google deliberately includes URLs disallowed in robots.txt in their index if there are links to those URLs from other pages they've crawled. To quote their Webmaster Tools help pages:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a ...
6
It doesn't matter. According to the spec:
The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL).
CR = Carriage Return (i.e. \r)
CR/NL = Carriage Return/New Line (i.e. \r\n)
NL = New Line (i.e. \n)
This means you can use \r. \n, or \r\n and it will work.
6
Remeber that no one from internet can see your directory tree, so stackoverflow.com/ and test.stackoverflow.com are completely diffrent sites for us and search robots.
You can do that by checing http host in query.
RewriteCond %{HTTP_HOST} ^test
RewriteRule robots.txt someotherrobots.txt
6
This is a bad idea. This practice, of showing one thing to search engines and another to users, is called cloaking: http://www.google.com/support/webmasters/bin/answer.py?answer=66355
It will cause your site more harm than good, with respect to search engine results; it "may cause your site to be perceived as deceptive and removed from the Google index", ...
6
You can see what the file contains here: http://cdn.attracta.com/sitemap/728687/0.xml.gz
If that doesn't reflect your sites structure I would delete it. However see answer 2 - you may be able to update it to make it accurate.
Probably.
An Attracta Starter Account
($120/year value) is bundled and
integrated into the HostMonster
Control Panel of ...
6
Yes, you can have a sitemap index containing references to other sitemap indexes [source], as long as they're all on the same domain.
Each individual sitemap index can include up to 1,000 references to another <sitemap>. If you need to reference more than that, create multiple sitemap indexes and submit each one.
Each individual sitemap can include ...
6
Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them.
This will depend a bit on the exact implementation of the crawlers robot.txt ...
5
I'll have to go looking for the source of this information but apparently robots.txt will not necessarily prevent a page from being indexed. But the HTTP x-robots-tag header does apparently work.
If you're using Apache you can block pages in bulk using this line in an .htaccess file:
Header set x-robots-tag: noindex
Give that a try and see what happens.
...
5
You can't make them re-download your robots.txt when you want them to. Google will re-crawl it and use the new data whenever they feel it is appropriate for your site. They tend to crawl it regularly so I wouldn't expect it to take long for your updated file to be found and your pages re-crawled and re-indexed. Keep in mind that it may take some time after ...
5
It would seem that Google has probably not yet updated it's cache of your robots.txt file. Your current robots.txt file (above) does not look as if it should be blocking your sitemap URL.
I guess google just hasnt updated its cache.
There is no need to guess. In Google Webmaster Tools (GWT) under "Health" > "Blocked URLs", you can see when your ...
5
robots.txt only specifies how Google should or should not crawl your site and as Mike said if you solely rely on this vector it will take a long time to have the pages removed from the index.
So you will also need to make a removal request if you want it to happen faster and have Google remove the pages from the index. When you are in Webmaster Tools go to ...
4
I had a site with far more links from other reputable, well trafficked sites that took several weeks to be indexed. And it was first caught by Bing, not Google, despite Google's far more prolific crawler. Go figure.
There's a concept called "Domain Aging." Nobody knows exactly what the various search engines look for, but several surmise that older, more ...
4
Google Webmaster tools has a Section called "Crawler access"
This section allows you very easily to create your robots.txt
For example to allow everything except blog a folder called test your robot.txt would look something like
User-agent: *
Disallow: /Test
Allow: /
4
While it's not part of the standard robots.txt protocol (and therefore not globally recognized), Google and Bing both support LIMITED pattern matching. For example:
User-agent: Googlebot
Disallow: /private*/
Will block Googlebot from any directory beginning with "private". Currently, Googlebot supports * and $ (end of string)
Details on Googlebot's ...
4
I think Matt Cutts talked about this. If my memory is correct it had to do with linking.
Here is more: http://www.google.com/support/forum/p/Webmasters/thread?tid=2720810fa226e9c8&hl=en
You can remove them with the Google removal tool.
4
No. Read this(all of it; there's a lot of useful stuff), though of particular relevance here:
Google no longer recommends blocking
crawler access to duplicate content on
your website, whether with a
robots.txt file or other methods.
[...]
Duplicate content on a site is not
grounds for action on that site unless
it appears that the intent ...
4
There are various theories as to how Google knows what to crawl. It could be that someone linked to your mobile version. It could be that Google tried random urls and came across the /m version of your site. I'm not aware that they say they won't use URLs from their analytics data.
Yes they do follow those rules: ...
4
As suggested by Pekka you may want to try to place the Allow directives after the Disallow directives.
But given the differences in interpretations between Google, Bing and others, you may want to use a robots meta tag instead. This will be safer and more granular.
In your disallowed pages:
<meta name="robots" content="noindex" />
In your allowed ...
Only top voted, non community-wiki answers of a minimum length are eligible
