I have a big problem in Google Webmaster Tools. The number of 404 pages in the error report increases so fast that now I have over 1000. When I check for errors, I see that for every page Googlebot tries to crawl URLs without .html. That creates a 404 error each time.

I have tried to find the source of this error. Here is an example: http://ermagazin.com/najgora-nuklearna-katastrofa-u-americkoj-povijesti-za-koju-nikad-niste-culi It has 3 sources that are correct links. One of them is http://ermagazin.com/najgora-nuklearna-katastrofa-u-americkoj-povijesti-za-koju-nikad-niste-culi.html which is the correct URL that Googlebot should be crawling instead the first one without .html.

Check screenshot:

enter image description here

Is there something I can add in robots.txt to prevent Googlebot from crawling the URLs without .html?

My robots.txt is:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
Disallow: /readme.html

Sitemap:   http://ermagazin.com/sitemap_index.xml   
http://ermagazin.com/post-sitemap.xml   2016-02-11 08:57
http://ermagazin.com/page-sitemap.xml   2016-01-14 14:45
http://ermagazin.com/category-sitemap.xml   2016-02-11 08:57
http://ermagazin.com/post_tag-sitemap1.xml  2016-02-11 08:57
http://ermagazin.com/post_tag-sitemap2.xml  2016-02-11 08:57

1 Answer 1


When Googlebot is crawling such a large number of bad URLs, it is almost always because your site is misconfigured and you are linking to the URLs incorrectly somewhere.

In your case it is the "show all articles" link. For example on this page I see the following in the HTML source code:

<a href="http://ermagazin.com/zakopao-zivu-djevojku-8-mjeseci-zbog-vjerovanja-da-ce-to-donijeti-bogatstvo-tanzanija-u-soku" class="more-articles-button">show all articles</a>

It appears that when I click on that button in a browser, I don't get to the 404 page. You must have some JavaScript that intercepts the click and causes browsers to to something else. However, Googlebot scans the HTML source code and finds that link. When it tries to follow it, it gets a 404 version of each and every article on your site.

You need to fix that link, and look for others like it.

Another thing that you can do is redirect requests for the URLs without .html to the correct versions. Since you are using WordPress, you might want to use a WorpPress 404 plugin that allows you to monitor and redirect 404 errors. I used to use one called "True Google 404" that ran the words in not found URLs through site search and automatically redirect to the proper page. Sadly that plugin appears not to be available anymore. I did a quick search but I didn't find any plugins that allow redirects based on patterns from WordPress 404s.

  • I am little confused right now, I had Smart 404 but i uninstalled it because, how to than find in dashboard broken link to correct it if all broken returns to 301? :D Also, isn't it said to avoid redirections, I think that slows website load speed right? Feb 11, 2016 at 18:40
  • 301 redirects are needed to get Google to the right place. When properly implemented they should not slow down the site. Feb 11, 2016 at 18:56
  • I added yesterday WorpPress 404 plugin and so far no errors, than you, I will ask you one more question although is not the right section to ask, but i must, i have in my dashboard h1, h2, h3 when writing article, but when check for heading, it shows h1- 0 h2- Media, science,,, so I have no h1 and h2 are my site categories, any idea please? Feb 12, 2016 at 21:13
  • Not sure. You could probably ask it as another question if you need others to look at it. Feb 12, 2016 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.