Searches for "more information about text formats"

Question

I have a site using Drupal as CMS, and got "infected" badly with spammer bots.

Moreover, recently I signed up for Google's Webmaster Tools, and I see that one of the top search queries that result in my site is "more information about text formats" And I know that this block of text appears in drupal sites.

So I'm guessing that this is the bots' way to find spammable drupal sites. So is there a precaution for this? I already made robots.txt to Disallow: /drupal-subfolder/ but apparently this does not affect malicious bots.

Answer 1

You can prepare a list of bots which you don't want to allow on your website & mention them in robots.txt like

User-agent: <bot_identifier>
Disallow: /

Here is a exhaustive bad bots list.

But still many bad bots will not respect it & crawl your site. For that you need to install application like Fail2ban to restrict the crawlers which are crawling at a higher rate.