0

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site.

I cleaned the websites in question and updated the robots.txt.

the robots.txt file look like this:

User-agent: *
Disallow: /
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

which means the first line is Disallow: / yet all major crawlers are hitting(including google, bing, apple) non existing pages in the root directory and hitting 404 not found

I have updated the robots.txt file around 12 hours earlier.

Am I missing something here?

  • 1
    12 hours is not enough time to give search engines to find and honor a new robots.txt file. They may not request a view the new one for days. – John Conde Oct 6 '18 at 13:23
  • Hmm. Thanks. I was also thinking that this might be the only reason. – Joshi Oct 6 '18 at 13:34
  • Hi John Conde - I have another question like there is a non-existant link which If I access through browser - I am getting 404 article not found, but in access log for the crawler it is representing as "GET /czc2MTk1WHY3bzEwNTdY HTTP/1.1" 301 610 - any reason for the same. – Joshi Oct 6 '18 at 13:40
  • Is there a real reason why you need to get rid of the crawler 404 request, like a bandwidth concern? If the links stay 404 for long enough, the crawlers will stop crawling them and the situation will sort itself. – Maximillian Laumeister Oct 6 '18 at 17:07
  • yes it is a bandwidth concern for example there are not just 10-20 links but in runs into thousands. so I would like to know, say I live the 404, will the same links be hit again by the same crawler? if not then I can leave it as it is. – Joshi Oct 6 '18 at 18:39
2

As @Henry has already pointed out, you have a serious problem with your robots.txt file that needs fixing.

However...

yes it is a bandwidth concern for example there are not just 10-20 links but in runs into thousands.

This really should not be a "bandwidth" concern. If it is then you likely already have other hosting concerns that need fixing. Yes, search engines will continue requesting these non-existent URLs for a while. Return a 410 Gone to get the search engines to drop the requests sooner. To minimise the impact on your server you should ensure that such requests are not routed through your framework and return a minimum text-only response (or no response body at all).

Exactly how you implement this will depend on the format of the invalid URLs, how your site is implemented and whether you have server access.

1

There's an issue with your robots.txt file that, instead of fixing the 404 problem, will exacerbate it. The directive:

User-agent: *
Disallow: /

ensures that all search engines are prohibited from crawling your entire website. (In fact, you could have excluded everything after these two lines, because they already state that nothing should be crawled.)

The problem is, back when your websites were crawled, all of these nonexistent pages were indexed. In order for search engines to take those pages out of the index faster, they should be able to crawl your site, see what is actually on it, and then they can proceed to exclude the 404 pages. That won't happen if you're blocking them, and it may take forever to get those 404's out of the index. It's also hurting your rankings.

In order to allow the search engines the best access to your site, exclude Disallow: / from your files. Also, make sure that your content, images, pages, categories, and CSS and JavaScript files can be crawled. After you make those updates, resubmit your sites to be crawled by Google and Bing in their respective webmaster tools accounts.

Separately, and as others have alluded to in the comments, it's normal for most websites to have some amount of 404's, though it's a good idea to try to fix this when possible. Also, any changes to what's been indexed won't happen right away; it may take days, weeks, or even a month or two to get things to where you want them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.