1

I have a sitemap file which generates the sitemap in real time and lists every page on the site. Eventually the site will have tens of thousands of pages so generating the sitemap will take a fair bit of resources to do so I don't want just anyone to access the sitemap. What I am trying to achieve is something similar to the way Stack Exchange does it where search engines requesting the sitemap will be able to access it properly but if a person tries to access the sitemap file directly they will be served with a 404 not found error.

Could anyone point me in the right direction for how to do this. I already have a rewrite rule in place to rewrite sitemap.xml to sitemap.php for the generating script so now just need to limit access to the file. I would prefer to do this through a .htaccess file or through the vhost file but if it has to be done in the PHP so be it.

Thanks

||improve this question
  • You would need to write this in code. I do not use PHP, I use another language, however, this can easily be done. You would want a text based map file, slurp file, or some other mechanism that allows you to specify who can see what. You want to go by the sub-domain/domain name and IP address or just the IP address. Domain names by themselves can be easily spoofed and agent names are extremely unreliable. You can use Apache variables to compare to your list. As well, any 404 header can easily given. You just need to research how. Cheers!! – closetnoc Apr 4 '17 at 16:41
1

One way to limit access to search engines is to publish the sitemap at a secret url like /sitemap-poakunmecruight.xml. Instead of directing /sitemap.xml to that, or publishing that URL in robots.txt, only submit the sitemap URL to search engines via their webmaster tools. That way only search engines know where your sitemap is.

It also occurs to me that generating the sitemap in realtime may not be the best solution. If it takes resources to generate (like database queries), it would be better just to generate it once a day. You could fairly easily write a cron job that runs something like curl -s http://example.com/sitemap.php > /var/www/example.com/sitemap.xml

It sounds like you are leaning towards user agent sniffing to only allow bots. You can do that through .htaccess. I took the list of search engine bots from here.

BrowserMatchNoCase adsbot-google search_engine_bot
BrowserMatchNoCase aolbuild search_engine_bot
BrowserMatchNoCase baidu search_engine_bot
BrowserMatchNoCase bingbot search_engine_bot
BrowserMatchNoCase bingpreview search_engine_bot
BrowserMatchNoCase duckduckgo search_engine_bot
BrowserMatchNoCase googlbot search_engine_bot
BrowserMatchNoCase mediapartners-google search_engine_bot
BrowserMatchNoCase msnbot search_engine_bot
BrowserMatchNoCase slurp search_engine_bot
BrowserMatchNoCase teoma search_engine_bot
BrowserMatchNoCase yandex search_engine_bot

<Files "sitemap.php">
    Order Deny,Allow
    Deny from all
    Allow from env=search_engine_bot
</Files>
||improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.