Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

An error with our URL rewrite rules caused some pages to be not found (404 errors) when Google crawled our site.

I regularly watch for errors and fix database or coding issues when they arise. We usually avoid them altogether, but sometimes we're surprised to find a 404 and we fix it.

How bad is it? Supposing our goal is to establish "trust" and "authority" and keep the spider coming back for more pages, does anyone know how the spider acts programatically with crawl errors. Is there a point where the crawl frequency is reduced, or the SERPs affected?

share|improve this question
up vote 3 down vote accepted

How bad is it?

Badness is directly proportional to the number of links to the 404 and the number of visitors who are disappointed when they arrive at your site; Google Webmaster Tools won't show you a 404 unless there's a link to the content somewhere on the internet.

Does anyone know how the spider acts programatically with crawl errors[?]

If your domain expires you can expect it to be removed from search results post haste (i.e. about 48 hours from my experience).

If your domain stops accepting all traffic (i.e. internal server error or a multitude of 404's) it's far less certain how long your rank will remain - my anecdotal experience is 1-2 weeks for extremely low-value search terms (wouldn't let that happen for ranking on high-value terms).

Per this Webmaster World "how long to removal" thread, Google may keep "dead" content in its supplemental index for quite a while, but you're on the right track with fixing the problem as it appears*.

* You are applying a 301 redirect to the correct URL or most relevant content, right? :)

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.