1

I used the google.com/ping service to make it download 500 sitemaps for a customer of mine, that was about 30 hours ago.

Googlebot reacted immediately, downloaded all 500 sitemaps.

Now I have more than 30,000 downloads from Googlebot in less than two days.
Always the same unchanged sitemaps are downloaded, again and again and again. They are valid, clean and Bing downloads them ONE time and is fine with it.

I am quite worried about that amount of downloads, that will cause significant traffic over time and it's useless.
Those sitemaps are not going to change soon.

The webserver is a normal Apache.

The download count increased to 92000 downloads of unchanged sitemaps, I blocked google from access to sitemaps for half a day. This helped reduce the rate but it is slowly ramping up again.
Though since I blocked the GoogleBot the rates are a lot better than before.

So at this point it seems under control but the incident is really strange.

Improve this question
  • Can't address your main question, but you might want to read regarding whether sitemaps are necessary. webmasters.stackexchange.com/questions/4803/the-sitemap-paradox. – Trebor Jan 3 at 22:28
  • 2
    The website has millions and millions of pages, sitemaps should help enormously to cover them quick and without missing out. random crawls would need a long time – John Jan 3 at 22:34
  • In my experience, Google's crawling isn't totally random, but crawls links to determine content before crawling sitemaps. That's still a lot of pages though. – Trebor Jan 3 at 22:41
  • @Trebor Google crawled about 4000 pages and 31000 times the same sitemaps Something is bugging on their end, I just don't understand what is triggering it – John Jan 3 at 23:19
  • 1
    What if you removed your sitemaps, and added them back in a few at a time? I'm not sure how many a few would be, but what if you added 50 every day or some other increment? I've had similar problems with other apps when I overloaded the number of submissions and they got stuck in a loop trying to process my submissions. You'll probably have to wait for Google to exit out of the current loop before resubmitting. – Trebor Jan 3 at 23:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.