Sign up ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am storing my sitemaps in my web folder. I want web crawlers (Googlebot etc) to be able to access the file, but I don't necessarily want all and sundry to have access to it.

For example, this site (, has a site index - as specified by its robots.txt file (

However, when you type, you are directed to a 404 page.

How can I implement the same thing on my website?

I am running a LAMP website, also I am using a sitemap index file (so I have multiple site maps for the site). I would like to use the same mechanism to make them unavailable via a browser, as described above.

share|improve this question

migrated from Jan 13 '11 at 2:10

This question came from our site for computer enthusiasts and power users.

Sorry, questions about websites are off-topic on Super User. I recommend you ask this question on our sister site dedicated to Pro Webmasters questions, currently in public beta, where it will be better suited. –  Gnoupi Jul 20 '10 at 19:20

2 Answers 2

The best approach will most likely involve getting the IP address of the visitor to the page, performing a reverse NS lookup, and checking if the domain name matches the known list of web crawlers. As far as I know, this is pretty much foolproof (discounting DNS spoofing which is unlikely to be a major problem).

For the Google web crawler, this is described in the blog post How to verify Googlebot.

Here's a list of the domain name wildcards for the most common spider bots/web crawlers:

  • Google (Googlebot): *
  • Bing (msnbot): (Not resovable, see IP ranges)
  • Yahoo (Yahoo Slurp): *

Though I'm not sure how often the IP address ranges for the various main crawlers, there's also this page which lists such ranges for the three main search engines.

(Note: I believe the bots do set the user-agent HTTP header on requests, but this is very easy to fake of course.)

Hope this helps.

share|improve this answer
@noldorin: +1 for the useful links. I am quite new to all this. I think I will like to set up Apache to take into account the information you have provided. Off the top of my head, I think its the <File> or <FileMatch> directives along with 'Allow from' that I need to set Apache up. But I'm not sure how to 'wire' it all up. Could you please provide a snippet of the section I need to add to my httpd.conf. This will help me get started. Thanks –  morpheous Jul 20 '10 at 10:04
@morpheous: Sure, glad to help... Unfortunately I'm no real expert with regards to Apache and HTTP servers. You might want to ask that as a separate question over on :) –  Noldorin Jul 20 '10 at 11:19
+1 for the fact users can't spoof the IP... if you'd suggested user-agent, that would have been an easy hack for any hacker. –  JeremyB Jan 13 '11 at 2:43
"performing a reverse NS lookup" - Also very easy to fake. It's a better idea to cross-reference the autonomous system which the IP belongs to (and, if you cache that lookup, don't cache it for more than a day or two) –  danlefree Jan 13 '11 at 3:15
some (most? all?) of those spiders will cache copies of the pages in question and provide those cached copies to visitors who request them. –  Sparr Dec 6 '13 at 17:05

You can hide your sitemap files from normal visitors by:

  • Naming the sitemap file something obscure that can't be guessed: sitemap-jk4KnDJ8.xml
  • NOT listing the sitemap file location in robots.txt
  • Submitting the sitemap to the search engines that you want to fetch it through their web sites. For example through Google Webmaster Tools.
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.