Major Crawlers not honoring robots.txt?

Question

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site.

I cleaned the websites in question and updated the robots.txt.

the robots.txt file look like this:

User-agent: *
Disallow: /
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

which means the first line is Disallow: / yet all major crawlers are hitting(including google, bing, apple) non existing pages in the root directory and hitting 404 not found

I have updated the robots.txt file around 12 hours earlier.

Am I missing something here?

Answer 1

As @Henry has already pointed out, you have a serious problem with your robots.txt file that needs fixing.

However...

yes it is a bandwidth concern for example there are not just 10-20 links but in runs into thousands.

This really should not be a "bandwidth" concern. If it is then you likely already have other hosting concerns that need fixing. Yes, search engines will continue requesting these non-existent URLs for a while. Return a 410 Gone to get the search engines to drop the requests sooner. To minimise the impact on your server you should ensure that such requests are not routed through your framework and return a minimum text-only response (or no response body at all).

Exactly how you implement this will depend on the format of the invalid URLs, how your site is implemented and whether you have server access.

Answer 2

There's an issue with your robots.txt file that, instead of fixing the 404 problem, will exacerbate it. The directive:

User-agent: *
Disallow: /

ensures that all search engines are prohibited from crawling your entire website. (In fact, you could have excluded everything after these two lines, because they already state that nothing should be crawled.)

The problem is, back when your websites were crawled, all of these nonexistent pages were indexed. In order for search engines to take those pages out of the index faster, they should be able to crawl your site, see what is actually on it, and then they can proceed to exclude the 404 pages. That won't happen if you're blocking them, and it may take forever to get those 404's out of the index. It's also hurting your rankings.

In order to allow the search engines the best access to your site, exclude Disallow: / from your files. Also, make sure that your content, images, pages, categories, and CSS and JavaScript files can be crawled. After you make those updates, resubmit your sites to be crawled by Google and Bing in their respective webmaster tools accounts.

Separately, and as others have alluded to in the comments, it's normal for most websites to have some amount of 404's, though it's a good idea to try to fix this when possible. Also, any changes to what's been indexed won't happen right away; it may take days, weeks, or even a month or two to get things to where you want them.