What to do about crawler errors after removing non-essential parts of my website?

Question

So, recently I was re-building my website, putting on another CMS, optimizing and so on. Now I visited Google Webmaster tools and it tells I have a lot of Web crawler 404 not found errors.

I am keeping almost all URLs for Articles the same. Almost all the errors now come from insignificant pages that I removed: Archive/date/ and user profile pages. I don't plan to re-create those auxiliary pages for robots now or in near future, but 'never say never'. I know Google crawlers sometimes do crawl some pages for several years while they are long time gone.

Yesterday night I submitted for removal as outdated content several really old non-existing pages crawled for at least some 3 years, but I still have several hundred 404 error pages now. I don't know if I should I submit to remove them all or not.

There may be a small possibility that I may need some of those URLs in the future. If I remove them now, what would happen if I need this URL for something again?

Answer 1

You can perform a 301 redirect for the 404 pages to their relevant equivalent on the new site, or just to the home page. You should be more concerned with human visitors. The web crawlers will eventually stop crawling the old pages.

Answer 2

If the page is gone, then let it 404 error. If you can, then I suggest doing a 410 gone, but if there are quite a few, then it may not be worth the effort. NEVER 301 redirect a page that no longer exists unless you are redirecting to another relevant page. This should be done where ever possible for the sake of the user.

Here is what happens:

Using Google for an example, if a page request results in a 404 error, Google will keep trying for a number of times over a period of time before deciding the page is gone- essentially an equivalent to a 410 gone. Unless there is a link for that page, Google will not request it again. However, if there is a link, then Google will continue to follow that link as long as it exists and continue to request the missing page. This is not a problem. Just let Google discover again that your page is gone. This is where forcing a 410 gone error can really help. I will cut down on the number of requests anyone makes for the missing page.

What is important to realize is that not all crawl errors as Google lists them should be fixed. This is where it gets confusing to people. They think there is a problem when the process is doing exactly what it was designed to do. 404 errors are not a problem unless you get one for a page that should exist. Otherwise, you can ignore them.