1

I've been asked to look at the "Not Found" tab in Google Webmaster Tools, to try and eliminate as many of the thousands of 404's not listed. There are several reasons for these, but this post will focus on one type. 404's which cannot be located within our site.

Our WordPress installation will redirect the 404 page to the same page, but with Google Search parameters in the querystring to display a Google search to help the users find what they are looking for.

So if a user goes to:

www.ourdomain.com/pagedoesntexist.php

they will get redirected to

www.ourdomain.com/pagedoesntexist.php?cx=987363876 (etc. this is the Google Search Info)

Now in our "Not Found" tab, it lists the URL with the Google Search code as the page, but lists the first page (with no querystring) as the referring URL.

Where I get lost is here. If I do a complete site crawl of our websites, or search the sitemaps for "www.ourdomain.com/pagedoesntexist.php", I cannot find a reference to it anywhere.

So I am confused as to why Google doesn't detect the original page as the 404, since I am guessing WordPress would be handling 404's the way Google expects them to. I am also confused as to how I can get rid of a 404 that I don't have a proper referrer to, as well as why a complete crawl wouldn't reveal the same 404's that Google finds.

When I crawl the site with Xenu, out of 35,000 pages, it finds approximately 50 404's, most of which are GIF/PNG/JS/CSS glitches and errors.

Improve this question
4
  • Please understand that not all 404's have anything to do with errors on your site. With so many spiders and scraper sites, arrant links are often made on junk sites that then are followed by other spiders, scrapers, and bots- Oh my! You will not be able to predict these. If you do not have an ErrorDocument in your .htaccess you will want to create one. Unfortunately, I am not sure how much you can do with ErrorDocument. You should be able to do something helpful.
    – closetnoc
    Feb 23, 2015 at 17:07
  • Try using a crawler such as Screaming Frog, and set the user agent to 'Google-Bot', this sometimes, but not often finds other links. You could also check your server logs to try and help identity.
    – Max
    Feb 24, 2015 at 2:38
  • @Max, Screaming Frog won't help here, since the website has 35k pages, while SEO Spider has 500-page limit in free version.
    – Oleg
    Feb 24, 2015 at 13:09
  • @Oleg The 500 page limit is for the free version. You can access as many pages you want with a license. ScreamingFrog is great for finding broken internal links on your site and is highly recommended for auditing internal links from time to time. However, most of the time, broken links that are not recognized are coming from scraper sites though not always.
    – closetnoc
    Feb 24, 2015 at 15:50

1 Answer 1

Reset to default
1

You can crawl your website with Site Visualizer software to find 404 pages, find 'pagedoesntexist.php', as well as view all redirects within a site, if any.

After you've downloaded and installed Standard or Pro version (unlimited 30-day trial), click Project -> New, enter your website URL, then uncheck some options as follows:

show website 404s and redirects

Click OK, then Start button. After the crawling is complete, double-click Internal Links report to show all internal links of your site, then click Find command of the report's table context-menu:

find among internal links

Type 'pagedoesntexist' text and click Find Next button. To faster the search, you can uncheck all fields except From URL and To URL. This allows you to find pages that reference to 'pagedoesntexist.php' page, and vice versa.

You can also export the data to an MS Excel sheet or similar. Use Copy Rows commands from the table's context menu.

Improve this answer
1
  • 1
    This appears only to cover broken internal links. It has been my experience that most 404 errors are a result of scraper sites though not always. Your suggestion is very good for scanning a site for internal broken links. Nicely detailed answer.
    – closetnoc
    Feb 24, 2015 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.