Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?

share|improve this question

2 Answers 2

up vote 4 down vote accepted

It's depends on which search engine you refer to. Most people are interested in google so I'll assume that's what you meant:
Generally speaking, the higher your rank is, and if the frequency in which you change your site (add articles etc) is high - your site will be crawled more frequently.

The following article might interest you.

share|improve this answer

Let's try an empirical approach.

In the access logs for my site, I see 55 sitemap requests over the last 33 days. Out of those 55, 30 are from Googlebot, 21 from msnbot and the remaining four are from Exabot. (I've only submitted the sitemap manually to Google; the others have found it through robots.txt.)

So that's one data point for about "every day", at least for Google and for a smallish site like mine. Although I should note that I regenerate the sitemap daily, so it's possible that Googlebot is simply observing a pattern in the last modification timestamps and adapting to it.

If you want to directly inform search engines that your sitemap has been updated, you can do so by sending an HTTP ping. This may make the search engines reload your sitemap sooner, although of course there are no guarantees.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.