Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a site using Drupal as CMS, and got "infected" badly with spammer bots.

Moreover, recently I signed up for Google's Webmaster Tools, and I see that one of the top search queries that result in my site is "more information about text formats" And I know that this block of text appears in drupal sites.

So I'm guessing that this is the bots' way to find spammable drupal sites. So is there a precaution for this? I already made robots.txt to Disallow: /drupal-subfolder/ but apparently this does not affect malicious bots.

share|improve this question

migrated from stackoverflow.com Jul 15 '13 at 13:02

This question came from our site for professional and enthusiast programmers.

You can prepare a list of bots which you don't want to allow on your website & mention them in robots.txt like

User-agent: <bot_identifier>
Disallow: /

Here is a exhaustive bad bots list.

But still many bad bots will not respect it & crawl your site. For that you need to install application like Fail2ban to restrict the crawlers which are crawling at a higher rate.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.