I've been asked to look at the "Not Found" tab in Google Webmaster Tools, to try and eliminate as many of the thousands of 404's not listed. There are several reasons for these, but this post will focus on one type. 404's which cannot be located within our site.
Our WordPress installation will redirect the 404 page to the same page, but with Google Search parameters in the querystring to display a Google search to help the users find what they are looking for.
So if a user goes to:
www.ourdomain.com/pagedoesntexist.php
they will get redirected to
www.ourdomain.com/pagedoesntexist.php?cx=987363876 (etc. this is the Google Search Info)
Now in our "Not Found" tab, it lists the URL with the Google Search code as the page, but lists the first page (with no querystring) as the referring URL.
Where I get lost is here. If I do a complete site crawl of our websites, or search the sitemaps for "www.ourdomain.com/pagedoesntexist.php", I cannot find a reference to it anywhere.
So I am confused as to why Google doesn't detect the original page as the 404, since I am guessing WordPress would be handling 404's the way Google expects them to. I am also confused as to how I can get rid of a 404 that I don't have a proper referrer to, as well as why a complete crawl wouldn't reveal the same 404's that Google finds.
When I crawl the site with Xenu, out of 35,000 pages, it finds approximately 50 404's, most of which are GIF/PNG/JS/CSS glitches and errors.