2

How long will Google Webmaster Tools try to crawl old links? We are seeing errors in the crawler that show "linked from" URLs that haven't existed in months. Will it just keep trying to crawl the link until we provide a redirect?

Also, we're seeing 404s that say the linked from source is our homepage, and we have not had those links on the homepage in quite a while.

Improve this question

2 Answers 2

Reset to default
2

There's no set time frame for when Google will stop. But they will eventually stop. They need to keep crawling them for a period of time because those pages may only be unavailable temporarily due to an error on your end. So they want to give them a chance to reappear in those cases.

It is assumed if those pages were intentionally removed you would serve up a 301 redirect to the new URL or a 410 GONE for deleted pages. So in your case you should consider sending 410 GONE statuses for those pages to speed up the process.

Improve this answer
5
  • After doing what @John Conde recommends, you should "Mark as fixed" every non-existent URL reported in Google Webmaster Tools -> Crawl Errors report.
    – Binarysurf
    Jun 24, 2014 at 21:02
  • Actually- DO NOT MARK AS FIXED! It just resets the counter that will eventually expire the link/page. If you keep marking as fixed, Google will think you solved the problem and continue to index the pages. The process begins all over! If you just leave it alone, Google will stop after a certain amount of tries(404 errors)/over time.
    – closetnoc
    Jun 24, 2014 at 21:09
  • 1
    @closetnoc I can't find a page where Google's documentation says something like "The process begins all over", did I misundertood what this doc's page states? support.google.com/webmasters/answer/…
    – Binarysurf
    Jun 24, 2014 at 22:32
  • Interesting, GWMT's documentation talks about fixing over DNS errors page. support.google.com/webmasters/answer/…
    – Binarysurf
    Jun 24, 2014 at 22:35
  • It will not say that- but that is what happens. You are telling Google that you fixed a condition that is actually correct in the first place. Not all notices need to be fixed. They are what the are. For 404's that should be 404's, you leave them alone and they will disappear.
    – closetnoc
    Jun 24, 2014 at 22:36
0

Googlebot will never stop crawling a URL if you once had content there. I have 10 year old sites that still get crawled at old URLs.

You have two options:

  • Return 410 Gone status
  • Redirect the page to another URL

Google treats the 410 Gone much like 404 and it still shows up in Webmaster Tools as an error, unfortunately.

Improve this answer
1
  • This is because there are databases of old Google API data out there being used on spam sites in places like Russia, Poland, and to a lesser degree- China. As well, there are many Google data instances that have to be reconciled and not just one and some are quite a bit slower. Also consider that there are obscure links on obscure pages that still exist. Google will continue to crawl these and follow links, but not always reflect them in Google Webmaster Tools and so forth. They simply do not have enough importance and fly under the radar but still get picked up from time to time.
    – closetnoc
    Jun 24, 2014 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.