1

Are there advantages to listing XML sitemap files in robots.txt vs submitting them directly to Google via Webmaster Tools?

2

It seems these days that a sitemap in the root named sitemap.xml will get picked up by dang near anyone in time.

It all depends on how much control you want.

If you do not want just anyone reading your sitemap including scrapper bots, then name it something unique and submit it to Google, Bing, and who ever else you want. Do not put it in your robots.txt file.

If you are not worried about who reads your sitemap, it is mostly enough to just create a sitemap file in the root named sitemap.xml. Bing, Yandex, and Baidu all found mine just fine without anything in the robots.txt file. However, if you want it more widely known and more easily picked up, then use the robots.txt file. I would still submit it to Google and Bing manually. That way you can see some info about how many pages have been indexed from the sitemap etc. I do not think that Google specifically looks for a sitemap if one was not submitted. Things may have changed recently where they do.

Also consider that some of these domain stat sites look for sitemaps and will report it. So if you make it publicly known, it will really be publicly known to anyone including script kiddies. There are scrapper bots reading the sitemaps and spidering sites.

While I do not have anything in my robots.txt, my sitemap file is in the root and named sitemap.xml. It is being reported on domain stat sites. I generally do not have any regrets over this, but once the genie is let out of the bottle, it is hard to stuff back in. Just keep that in mind.

  • 1
    "see some info" - Yes, I found that you had to manually submit the sitemap for GWT to report anything about it. Simply having an entry in robots.txt was not enough, although I think (?) Googlebot did still request the sitemap (a regularly named "sitemap.xml" file) before this. – MrWhite Apr 17 '15 at 22:48
  • Years ago, I did use a custom named sitemap for Google. The only reason why mine is not this time is because of a brain-fart. ;-) I created the new code here on the dev box and forgot to rename the file output before deploying the new code. Ooopppsss! This is the price of getting old. I do not recommend it. I decided just to leave the code and submit the new standard named sitemap. It is more in the past year or so that others started blindly looking for a sitemap. I have no real heart-burn over this, but it may have been better to stay private to stop the bat rastards. – closetnoc Apr 17 '15 at 22:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.