Google crawls some pages which do not exist on the website

Question

Google Webmaster Tools is reporting a crawling error.

403 Access Denied for an ".html" such as vy81TYYP/lraynb-20151123-dkd04-2290.html. It has never existed on a my developed website. This is not only once, this happens for more then 100 links.

404 Not Found for links, which do not exist on this website. Similarly, 5000 links are shown in the report.

Its too much complex to get where the issue generated. One of the similar question found and many more, but still not found anything to fix this.

Is there any webmaster issue? or any other? I scanned the wordpress website for malware each and every file also my database. I've not found any fix.

Answer 1

If such a thing not exist in your website then you don't worry about it, may be someone miss-link your content, or may be your competitor do that for negative SEO.

To avoid negative SEO - Discover who links to your site, and disavow those backlinks.

It happens for internal reason as well, so kindly check your website sitemap, and internal link structure of your website.

Answer 2

Googlebot tries to create new URLs from anything (id, class etc.) what it can find on the page.

That is the cause of 404: the existence of 404 in your search console isn't a problem: your server reacts absolutely correctly - it answers with 404, if it receives a GET to non existing URL.

But in case of some URLs your server answers with 403 on try to reach them (where it should answer with 404 too). You should adjust your server to answer correctly, but in case of Googlebot you can't clearly know, how look URLs, which fire 403 - they don't exist. Then i would make a workaround to give 404 answer even to 403 errors.

To do so:

1. create you custom 404 file (404.php) and place before all code of it, just as the first line this code line:

   <?php http_response_code(404); ?>
   

With this line we want send 404 response code in any case.

2. Then create in your htaccess two lines:

   ErrorDocument 403 /error/404.php
   ErrorDocument 404 /error/404.php
   

   Note, that in case of both errors 403 and 404 we answer with the same error file, 404.php. Usually, in case of 403 error it would not help, because regardless of sending 404.php, the header status code would remain 403. But in step 1 we defined the sending of 404 status code with our 404.php, so 404 status code will be sent in any case.

Answer 3

There can be several reasons for this:

1. You had this link on your website, but removed it along with the webpage.
2. You have backlinks pointing to this exact URL. Check your website with MajesticSEO, OneSiteExplorer or Ahrefs.

As Goyllo - disavow this links. However, disavowing them will not remove non-existing pages from the index (at least, it will not happen soon). To remove those pages from index use Remove URLs in Webmaster tools (Google Index -> Remove URLs). There you can remove either pages one by one, or remove the whole directory. Works almost instantly. You can always revert the changes, so don't worry if you have removed wrong pages. Good luck!

Answer 4

Check if the these pages are linked from any other page. Remove those links and mark the errors as fixed. Make sure there are no backlinks on that pages. If there are any backlinks coming to that page make sure you display 404 and submit all those domains to disavow file.