I am storing my sitemaps in my web folder. I want web crawlers (Googlebot etc) to be able to access the file, but I don't necessarily want all and sundry to have access to it.

For example, this site (superuser.com), has a site index - as specified by its robots.txt file (https://superuser.com/robots.txt).

However, when you type https://superuser.com/sitemap.xml, you are directed to a 404 page.

How can I implement the same thing on my website?

I am running a LAMP website, also I am using a sitemap index file (so I have multiple site maps for the site). I would like to use the same mechanism to make them unavailable via a browser, as described above.

  • Sorry, questions about websites are off-topic on Super User. I recommend you ask this question on our sister site dedicated to Pro Webmasters questions, currently in public beta, where it will be better suited.
    – Gnoupi
    Jul 20 '10 at 19:20

The best approach will most likely involve getting the IP address of the visitor to the page, performing a reverse NS lookup, and checking if the domain name matches the known list of web crawlers. As far as I know, this is pretty much foolproof (discounting DNS spoofing which is unlikely to be a major problem).

For the Google web crawler, this is described in the blog post How to verify Googlebot.

Here's a list of the domain name wildcards for the most common spider bots/web crawlers:

  • Google (Googlebot): *.googlebot.com
  • Bing (msnbot): (Not resovable, see IP ranges)
  • Yahoo (Yahoo Slurp): *.yahoo.com

Though I'm not sure how often the IP address ranges for the various main crawlers, there's also this page which lists such ranges for the three main search engines.

(Note: I believe the bots do set the user-agent HTTP header on requests, but this is very easy to fake of course.)

Hope this helps.

  • @noldorin: +1 for the useful links. I am quite new to all this. I think I will like to set up Apache to take into account the information you have provided. Off the top of my head, I think its the <File> or <FileMatch> directives along with 'Allow from' that I need to set Apache up. But I'm not sure how to 'wire' it all up. Could you please provide a snippet of the section I need to add to my httpd.conf. This will help me get started. Thanks
    – morpheous
    Jul 20 '10 at 10:04
  • @morpheous: Sure, glad to help... Unfortunately I'm no real expert with regards to Apache and HTTP servers. You might want to ask that as a separate question over on ServerFault.com. :)
    – Noldorin
    Jul 20 '10 at 11:19
  • +1 for the fact users can't spoof the IP... if you'd suggested user-agent, that would have been an easy hack for any hacker.
    – JeremyB
    Jan 13 '11 at 2:43
  • "performing a reverse NS lookup" - Also very easy to fake. It's a better idea to cross-reference the autonomous system which the IP belongs to (and, if you cache that lookup, don't cache it for more than a day or two)
    – danlefree
    Jan 13 '11 at 3:15
  • some (most? all?) of those spiders will cache copies of the pages in question and provide those cached copies to visitors who request them.
    – Sparr
    Dec 6 '13 at 17:05

You can hide your sitemap files from normal visitors by:

  • Naming the sitemap file something obscure that can't be guessed: sitemap-jk4KnDJ8.xml
  • NOT listing the sitemap file location in robots.txt
  • Submitting the sitemap to the search engines that you want to fetch it through their web sites. For example through Google Webmaster Tools.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy