Why can't GoogleBot forget a 404 File Does Not Exist?

Question

For two months now, googlebot is trying to get a file which does not exist anymore. This is just one example out of many: I had renamed the file to a better name and removed the old file. Now, why does Google insist on getting a file which it already has seen does not exist for months? doesn't it just give up and get on being a happy bot?

My Error log file is filled with these repeating lines all wanting to get that one file, although they know for already long time that its not there:

1. What to do in these situations?
2. Are there automatic redirection rules that handle via 301 to the home page?

My error log:

[Sat Mar 05 01:55:41 2011] [error] [client 66.249.66.177] File does not exist: 
/var/www/vhosts/website.org/httpdocs/extraNeus.php 

[Sat Mar 05 01:58:20 2011] [error] [client 66.249.66.177] File does not exist: 
/var/www/vhosts/website.org/httpdocs/extraNeus.php

[Sat Mar 05 02:03:57 2011] [error] [client 66.249.66.177] File does not exist: 
/var/www/vhosts/website.org/httpdocs/extraNeus.php

on and on ... and on...

Answer 1

put this in .htaccess file (change the path/to/files to your file locations)

Redirect 301 /extraNeus.php http://www.yoursite.com/new_page.php

you can leave it in there forever or next time google indexes the page you should be good to remove it.

Answer 2

You should use a permanent redirect to tell GoogleBot and others where the page has moved to.

If the page no longer exists at all then you can add entries in robots.txt to tell bots not to access the missing page:

User-agent: *     
Disallow: /extraNeus.php

Answer 3

The reason that Googlebot is still trying to hit the page is that there is probably a link to it somewhere. When Google finds a link, it follows it without checking whether it has previous seen a 404 for that page or not. So if this page keeps getting hit, it's probably because of a link.

I agree with Crayon Violet and suggest that you include the 301 redirect from the old page to the new one.