2

I am storing my sitemaps in my web folder. I want web crawlers (Googlebot etc) to be able to access the file, but I don't necessarily want all and sundry to have access to it.

For example, this site (superuser.com), has a site index - as specified by its robots.txt file (https://superuser.com/robots.txt).

However, when you type https://superuser.com/sitemap.xml, you are directed to a 404 page.

How can I implement the same thing on my website?

I am running a LAMP website, also I am using a sitemap index file (so I have multiple site maps for the site). I would like to use the same mechanism to make them unavailable via a browser, as described above.

Improve this question
0

3 Answers 3

Reset to default
4

The best approach will most likely involve getting the IP address of the visitor to the page, performing a reverse NS lookup, and checking if the domain name matches the known list of web crawlers. As far as I know, this is pretty much foolproof (discounting DNS spoofing which is unlikely to be a major problem).

For the Google web crawler, this is described in the blog post How to verify Googlebot.

Here's a list of the domain name wildcards for the most common spider bots/web crawlers:

  • Google (Googlebot): *.googlebot.com
  • Bing (msnbot): (Not resovable, see IP ranges)
  • Yahoo (Yahoo Slurp): *.yahoo.com

Though I'm not sure how often the IP address ranges for the various main crawlers, there's also this page which lists such ranges for the three main search engines.

(Note: I believe the bots do set the user-agent HTTP header on requests, but this is very easy to fake of course.)

Hope this helps.

Improve this answer
5
  • @noldorin: +1 for the useful links. I am quite new to all this. I think I will like to set up Apache to take into account the information you have provided. Off the top of my head, I think its the <File> or <FileMatch> directives along with 'Allow from' that I need to set Apache up. But I'm not sure how to 'wire' it all up. Could you please provide a snippet of the section I need to add to my httpd.conf. This will help me get started. Thanks
    – morpheous
    Commented Jul 20, 2010 at 10:04
  • @morpheous: Sure, glad to help... Unfortunately I'm no real expert with regards to Apache and HTTP servers. You might want to ask that as a separate question over on ServerFault.com. :)
    – Noldorin
    Commented Jul 20, 2010 at 11:19
  • +1 for the fact users can't spoof the IP... if you'd suggested user-agent, that would have been an easy hack for any hacker.
    – JeremyB
    Commented Jan 13, 2011 at 2:43
  • "performing a reverse NS lookup" - Also very easy to fake. It's a better idea to cross-reference the autonomous system which the IP belongs to (and, if you cache that lookup, don't cache it for more than a day or two)
    – danlefree
    Commented Jan 13, 2011 at 3:15
  • some (most? all?) of those spiders will cache copies of the pages in question and provide those cached copies to visitors who request them.
    – Sparr
    Commented Dec 6, 2013 at 17:05
2

You can hide your sitemap files from normal visitors by:

  • Naming the sitemap file something obscure that can't be guessed: sitemap-jk4KnDJ8.xml
  • NOT listing the sitemap file location in robots.txt
  • Submitting the sitemap to the search engines that you want to fetch it through their web sites. For example through Google Webmaster Tools.
Improve this answer
2

This topic is over 9 years old, but maybe it helps someone right now.

Google itself wrote the following about verifying GoogleBot and other Google crawlers in their Google Search Central Documentation:

Alternatively, you can identify Googlebot by IP address by matching the crawler's IP address to the list of Googlebot IP addresses.

For all other Google crawlers, match the crawler's IP address against the complete list of Google IP addresses.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.