An error with our URL rewrite rules caused some pages to be not found (404 errors) when Google crawled our site.

I regularly watch for errors and fix database or coding issues when they arise. We usually avoid them altogether, but sometimes we're surprised to find a 404 and we fix it.

How bad is it? Supposing our goal is to establish "trust" and "authority" and keep the spider coming back for more pages, does anyone know how the spider acts programatically with crawl errors. Is there a point where the crawl frequency is reduced, or the SERPs affected?


How bad is it?

Badness is directly proportional to the number of links to the 404 and the number of visitors who are disappointed when they arrive at your site; Google Webmaster Tools won't show you a 404 unless there's a link to the content somewhere on the internet.

Does anyone know how the spider acts programatically with crawl errors[?]

If your domain expires you can expect it to be removed from search results post haste (i.e. about 48 hours from my experience).

If your domain stops accepting all traffic (i.e. internal server error or a multitude of 404's) it's far less certain how long your rank will remain - my anecdotal experience is 1-2 weeks for extremely low-value search terms (wouldn't let that happen for ranking on high-value terms).

Per this Webmaster World "how long to removal" thread, Google may keep "dead" content in its supplemental index for quite a while, but you're on the right track with fixing the problem as it appears*.

* You are applying a 301 redirect to the correct URL or most relevant content, right? :)

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.