I have this line:

Crawl-delay: 50

in the robots.txt file at the root of my website.

But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.

I don't want to disable the user agent but to make it to respect the delay value (50 in my situation).

How to make "crawl-delay" works for all "user-agent" in robots.txt? I also read somewhere that this can be done in .htaccess, is this correct?

  • What web server are you using?
    – user27849
    Commented May 19, 2013 at 19:29
  • 2
    Not all search engines observe Crawl-Delay entries. Yandex has a webmaster tool that allows you to set site scanning restrictions if you don't want to block traffic outright and lose Yandex as a search engine. Commented May 19, 2013 at 19:41
  • 1
    Google doesn't follow Crawl-delay commands, either. You can use Google Webmaster Tools to set crawl rate. Commented May 19, 2013 at 21:35
  • Just wondering if I can manage the "Crawl-Delay" in .htaccess file?
    – chefnelone
    Commented May 20, 2013 at 6:38
  • 1
    @FiascoLabs: there are at least 7 user agents whose don't observe Crawl-Delay value. I don't want to register, login, and understand their tool to get this. There must be another way to make the Crawl-Delay works. I just reformulate my question to see if this can be done in htaccess file.
    – chefnelone
    Commented May 20, 2013 at 7:00

1 Answer 1


I know you previously asked to block the bots and now you just want to slow them down.

The only way to force them to slow down is not to serve them content if they are going too fast. The problem with not serving content is it will surely have negative consequences to your search engine rankings.

In case you decide to block some of the less important bots, you can block a bot's user agent using Apache's configuration like this:

SetEnvIf User-Agent BadBot GoAway=1
Order allow,deny
Allow from all
Deny from env=GoAway

Just replace BadBot with Yandex or whatever the User Agent of the bot you want to block is.

When Apache sees the user agent it will pretend like there is no content available.

Keep in mind the User Agent headers are very easy to change so if Yandex changes theirs to something else, this hack won't work.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.