Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

About 3 months ago we've completely replaced the content of our domain, practically building up a new website. The sitemap for the new website has been submited to Google Webmaster Tools, the website earned very good rankings for a large set of keywords we were looking for.

However, Google Webmaster Tools keeps on returning some "not found" messages, every 2-3 days, but regarding exclusively pages from the old sitemap, the one of the old website, which does not exist anymore.

Why is Google looking to crawl for those old pages, that are not included anymore in the new sitemap and reports not to find them?

share|improve this question

3 Answers 3

Once Google crawls a URL, it will continue to try to crawl that URL forever and will never quit complaining about it if it goes away. On one of my sites, we changed the URL structure about 15 years ago. To this day, Googlebot still requests a bunch of the old style URLs, even the ones with no links to them.

You can mitigate this situation by returning something other than a 404 status at these pages

  • 410 Gone -- Google treats it like 404 not found except that it goes in a different report which you can ignore
  • 301 Permanent redirect -- If there is a proper replacement for the pages that you removed on your new site, you should redirect each of the individually. This will work unless you redirect anything to the home page. Google treats redirects to your home page as "soft 404" and they still appear in your 404 report.
share|improve this answer
1  
FWIW this is not a problem from Google's POV, it's just that the algorithms are curious to see if they missed anything. –  John Mueller May 29 '13 at 6:23

Presumably you are not attempting to redirect these pages because there is no alternative content?

It is possible other sites are still linking to these old URLs, so as Google crawls these other sites and examines the links they are going to generate 404s on your site.

share|improve this answer

As long as there are links to those pages somewhere, Google will try to crawl it. It takes a long while for Google to be sure a page doesn't exist anymore.

What you can do is 301 redirect those old pages to the new corresponding page. This way Google will know the old page doesn't exist anymore and that it's been replaced. Any "link juice" those pages got from links on other websites will also be transferred after a few months. If you completely rewrote your site and the old pages don't have a corresponding new one, you can just 301 redirect it to the new homepage. This way links on other site will still sort of work and users won't get a 404.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.