If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the sitemap trump robots.txt? I think so, but am not sure.
4 Answers
No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed.
However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as such evidence) then they may simply decide to add the URL to their index without any content. To quote Google's Webmaster Tools help pages:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results."
Such pages can turn up as search results e.g. for words included in the URL itself, or for words used in links pointing to the page.
Thus, if you both list a page in a sitemap and disallow it in robots.txt, it's likely that Google will index the URL of that page — but not its content.
answered Mar 29, 2013 at 0:11
-
So that would make your answer Yes instead of No, wouldn't it? :) Because it picks up the URLs despite the directory being blocked in robots.txt, and you seem to agree with that. May 26, 2014 at 18:28
Robots.txt defines what conforming bots are allowed or not to request. Even if a particular link is present in a sitemap, a bot is not be allowed to request it if the robots.txt disallows it.
Remember that sitemaps are not necessary and even if one is provided, crawlers may ignore URLs and crawl ones which are not there. If can see this in the Google Webmaster Tools which shows that not all the URLs in a sitemap get crawled and if some URLs are roboted.
Itai's answer is correct so nothing much major to add to that but in reply to your specific question...
A sitemap cannot trump a robots.txt, a sitemap provides no instructions / directives for crawlers on a website. They aren't even comparable. If you have instructed robots not to visit/follow /foo then any bots that are obeying your robots directives will simply not visit that directory regardless of what path they took to get there (sitemap or otherwise).
-
Erm... This is what Google say in their documentation for how they handle crawling. [absoluteURL] points to a Sitemap, Sitemap Index file or equivalent URL. The URL does not have to be on the same host as the robots.txt file. Multiple sitemap entries may exist. As non-group-member records, these are not tied to any specific user-agents and may be followed by all crawlers, provided it is not disallowed. Mar 28, 2013 at 18:04
-
3When Google can properly process a robots.txt file, a URL mentioned in a Sitemap file will never trump a valid disallow directive in the robots.txt file. A URL that is disallowed from crawling should not be crawled by Googlebot. Apr 2, 2013 at 20:17
In Google webmaster: It shows an error in your XML sitemap that "You have put a link which is prevented to Crawl in your robots.txt file. Google prefers robots.txt file rather than Sitemap.
