Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a scenario where my sitemaps are generated when visiting /sitemap.xml due to the contents being dynamic in nature. For that reason I'd like to not call the services that are used to create my sitemap when a user visits but it should generate contents when Google or Bingbot visits the link. I have code to detect the bots. I'm concerned if this is considered as bad according to Google or Bing.

share|improve this question
add comment

2 Answers

up vote 2 down vote accepted

I wouldn't worry about "cloaking" on files that are meant only for the consumption of robots. Showing a 403 Forbidden status when the user-agent isn't Googlebot should be fine on a sitemap file. Google cares about cloaking when users see different results than Googlebot. In this case, Google is never going to refer users to the sitemap at all.

I often serve different robots.txt files to different crawlers based on user-agent. That is another case in which I just don't worry about cloaking.

share|improve this answer
    
I talked to other SEO experts over these days and they suggested the same thing. They said that as long as users don't see different contents in the website itself than what Google or Bing is showing them for their search, any form of cloaking is okay. The key was "do not deter the search experience" to be on the good side of search engines. –  Vite Falcon May 23 '13 at 17:14
add comment

If I read your question right is that basically you do not want users or any other bots than Bing and Google having the ability to visit sitemap.xml as the contents of the sitemap are generated real time which could cause additional server load that you want to avoid.

If I'm correct you are approaching this in the wrong manner and there should be no reason resulting in such user agent detecting and serving different results as Google may think this is cloaking if they did an audit on your site using a standard user agent (Doubtful, but ya).

A Possible Solution

Within Google and Bing Webmaster Tools it gives you the option to add a sitemap, this sitemap can be named anything you want... So if you name it top-sercet-sitemap.xml then bots that generally just check to see if /sitemap.xml exists will receive a good old 404 error now the problem that arises is that Google is known for indexing sitemaps, now depending on your server software this can be prevented using Header Tags...

For example in Apache you can use Header set X-Robots-Tag to inject noindex into the header response of the sitemap - this means that Google will crawl the sitemap but won't index the sitemap itself, many people confuse noindex with not having the ability to crawl the page when this isn't true, only robots denied will do that.

For Apache:

<IfModule mod_rewrite.c>
  <Files ~ "^(top-sercet-sitemap1|top-sercet-sitemap2|top-sercet-sitemap3)\.xml$">
   Header set X-Robots-Tag "noindex"
  </Files>
 </IfModule>
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.