Website crawler returns hundreds of 404 errors for pages that do not exist containing duplicate directories

Question

I just completed a website crawl on my site to vet SEO, which should return 119 pages. The results are around 490, with errors being returned showing hundreds of 404 pages with strange URLs. These follow a roughly similar pattern, like so:

Where the actual live page looks like this (please note that this is a dummy URL):

www.example.com/about/careers.html

However, there are many 404 errors being returned when crawling the site that look similar or identical to these:

• www.example.com/about/about/careers.html
• www.example.com/about/about/about/careers.html
• www.example.com/about/about/about/about/careers.html

Currently, I cannot find any specific literature on what may be causing this - at first I thought it may be sitemap related, but attempting to re-run a sitemap generator also indicates over 490 pages which is very odd. I cannot see any of these additional pages on the IIS server system where the site is hosted, only the 119 that should be there.

I am totally stumped at this point! What could be causing this?

Answer 1

You have a broken link somewhere on your site. My guess is that it is in your navigation template that is used on every page on your site. You probably have a link like <a href="about/careers.html">Careers</a> that points to your career page.

That link will work fine on your home page, but it is a relative link that always adds the "about" directory into the URL, even if the "about" directory is already in the URL. When you get to the career page and click on the "careers" link again, it will go to a 404 page: www.example.com/about/about/careers.html.

The solution to the problem is to use "root relative linking" rather than "document relative linking". To do so, start all your links with a slash (/). So your careers link should be: <a href="/about/careers.html">Careers</a>. When a URL starts with a slash, the it is interpreted as relative to your root domain rather than relative to the current page and directory. Linking to /about/careers.html will work from any page on your site, no matter how deep in the directory structure it is.