How many times will Google crawler attempt to recrawl a URL that has a 5xx error occurring intermittently? When would it give up and drop the URL from the Index?

  • 2
    If the error is "intermittent" (or even if it's permanent), I can't really think of a reason why it would ever stop trying to crawl that URL (providing there are links to it, etc.). – DocRoot Jun 17 '16 at 10:01

When Google determines that your site is having problems, it will slow down the rate at which it crawls. If your site is less responsive, or is giving Google 500 errors, Google will make fewer request per day to your site as a whole.

Googlebot will never give up on any particular URL. Google will come back and try that URL again. How soon it comes back is determined by how important the URL is. It may take weeks for Google to try an unimportant URL again, but it may try again in a couple hours if the URL is well linked from other sites and has a history of being updated frequently.

  • I thought google gives up on URLs that return a 410 response code. – Mike -- No longer here Jun 19 '16 at 3:09
  • They crawl 410 URLs less frequently, but Googlebot will still check them occasionally. – Stephen Ostermiller Jun 19 '16 at 8:52

There is no easy way to measure this as there doesn't appear to be a hard limit and Google is so complex it's difficult to measure.

However, the more intermittent the service the lower an URL will rank as it accrues a performance penalty. This would lead me to guess that it's based on a curve rather than a binary 'indexed' or 'not indexed'.

The only thing you can be certain of is whether an URL is indexed or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.