1

So, recently I was re-building my website, putting on another CMS, optimizing and so on. Now I visited Google Webmaster tools and it tells I have a lot of Web crawler 404 not found errors.

I am keeping almost all URLs for Articles the same. Almost all the errors now come from insignificant pages that I removed: Archive/date/ and user profile pages. I don't plan to re-create those auxiliary pages for robots now or in near future, but 'never say never'. I know Google crawlers sometimes do crawl some pages for several years while they are long time gone.

Yesterday night I submitted for removal as outdated content several really old non-existing pages crawled for at least some 3 years, but I still have several hundred 404 error pages now. I don't know if I should I submit to remove them all or not.

There may be a small possibility that I may need some of those URLs in the future. If I remove them now, what would happen if I need this URL for something again?

  • Do receive human traffic to the 404 pages? – Sun Sep 12 '14 at 17:10
1

You can perform a 301 redirect for the 404 pages to their relevant equivalent on the new site, or just to the home page. You should be more concerned with human visitors. The web crawlers will eventually stop crawling the old pages.

| improve this answer | |
  • I don't think there will be human visitors; All real URL's for Article/Category pages are preserved; and I don't think there will be human linking to /date/07/12/2013 kind of links.. – Salvis Dišlers Sep 12 '14 at 17:19
1

If the page is gone, then let it 404 error. If you can, then I suggest doing a 410 gone, but if there are quite a few, then it may not be worth the effort. NEVER 301 redirect a page that no longer exists unless you are redirecting to another relevant page. This should be done where ever possible for the sake of the user.

Here is what happens:

Using Google for an example, if a page request results in a 404 error, Google will keep trying for a number of times over a period of time before deciding the page is gone- essentially an equivalent to a 410 gone. Unless there is a link for that page, Google will not request it again. However, if there is a link, then Google will continue to follow that link as long as it exists and continue to request the missing page. This is not a problem. Just let Google discover again that your page is gone. This is where forcing a 410 gone error can really help. I will cut down on the number of requests anyone makes for the missing page.

What is important to realize is that not all crawl errors as Google lists them should be fixed. This is where it gets confusing to people. They think there is a problem when the process is doing exactly what it was designed to do. 404 errors are not a problem unless you get one for a page that should exist. Otherwise, you can ignore them.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.