1

I've been having some indexing issues that I've had a development team working on trying to fix for a week but no progress has been made.

The site has an overwhelming amount of 404 errors as indicated by Google Search Console. This site is about 800 pages but there are almost 1300 404 errors. This site is built on WordPress.

All of said errors are existent on the Desktop portion of GSC but not on the Smartphone portion.

The 404's are all from pages that I have no recollection of ever existing and all follow the same format /URLPath/index.html.

One of these is errors is explicitly /2014/12/index.html. The page /2014/12 exists as do all of the other pages mentioned in the 404 list as long as you drop the /index.html that appends to the end of the URL.

Investigating I can see that the URL was crawled earlier in May and that this page is being linked from 5 other URLS (using the page mentioned above). Of these 5 URLS, none of them link to /2014/12/index.html or /2014/12/ and no changes have been made to this page since it was published.

A similar theme occurs for all of the other 404 errors and there linked from pages.

Is it possible that these crawl errors are contributing to my indexing problem?(A Site: search shows Google has over 1400 pages indexed but there are only about 800)

Why would these pages, appending /index.html, be created without me creating them?

NOTE:

All of these errors appeared over a 2 day period at the start of May.

This is happening for every tag & category as well /tag/index.html, /category/index.html, /category/helloWorld/index.html, etc.

This website used to be www and now uses the non-www version of the site.

A sitemap was submitted at the end of May with only the 800-ish pages that exist and only 98 are recorded as index by GSC.

EDIT:

I looked into the server logs to see who accessed the URLs that append /index.html but the entire server log is empty. There's no trace of anyone visiting a page that 404s.

Improve this question
5
  • 1
    Can you check your server logs and see what IP address(es) is/are attempting to access the non existent pages. If the IP address is associated with a known web crawler then it could be a glitch to do with the crawler itself, however if it is being accessed all from a single or small number of IP addresses and they are associated with consumer IP addresses then it could indicate an attempt to map your website through a private manner. Ordinarily 404's are not much of a concern but with the number you are getting further investigation is warranted.
    – Chris Rutherfurd
    Commented Jun 6, 2016 at 4:44
  • 2
    Just to be clear, the only difference between the nonexistent URLs and existing URLs is the addition of /index.html? Many web servers will treat the default page (typically index.html) the same as just /. Technically, they're two different URLs so could be indexed separately. If you're able to reach the URLs with /index.html added to them and also /, then redirect the former to the later using a 301 redirect (e.g., via a .htaccess) and the errors and indexing should fall off eventually. You could also block access to /index.html using a robots.txt.
    – dan ♦
    Commented Jun 6, 2016 at 7:06
  • You could also block.... .... But if those two pages are copies of each other and are extremely important pages, a redirect is much better, because blocking will frustrate a portion of the users who stored the blocked version of the URL in their browser bookmarks.
    – Mike -- No longer here
    Commented Jun 7, 2016 at 1:59
  • I had my team look into the server log as @PlanetScaleNetworks suggested and have made an edit
    – Memj
    Commented Jun 7, 2016 at 13:52
  • If you are waiting for Google to crawl your site to find the 404s then you have more problems tha you think.
    – symcbean
    Commented Jun 8, 2016 at 11:41

1 Answer 1

Reset to default
1

Based on your edit there is no cause for concern. 404 errors are very common in the search console simply due to the fact that someone somewhere has linked to that page. It could also be the Googlebot having a glitch trying to append index.html to the end of the URL. This is also a common issue due to many URL rewriting configurations leaving out the file extension which under default Apache configurations means each and every page is within a folder and the files are all named index.html and are simply in different folders to denote the site structure. For a static HTML site this is no issue as that is how it may be setup but with a rewritten URL such as what wordpress does this can cause issues only in the sense that you will see a large number of 404 errors. Basically there is no need to worry about it and it isn't going to affect your SERP rankings.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.