Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have this line:

Crawl-delay: 50

in the robots.txt file at the root of my website.

But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.

I don't want to disable the user agent but to make it to respect the delay value (50 in my situation).

How to make "crawl-delay" works for all "user-agent" in robots.txt? I also read somewhere that this can be done in .htaccess, is this correct?

share|improve this question
    
What web server are you using? –  Tom Dworzanski May 19 '13 at 19:29
2  
Not all search engines observe Crawl-Delay entries. Yandex has a webmaster tool that allows you to set site scanning restrictions if you don't want to block traffic outright and lose Yandex as a search engine. –  Fiasco Labs May 19 '13 at 19:41
1  
Google doesn't follow Crawl-delay commands, either. You can use Google Webmaster Tools to set crawl rate. –  Arttu Raittila May 19 '13 at 21:35
    
Just wondering if I can manage the "Crawl-Delay" in .htaccess file? –  chefnelone May 20 '13 at 6:38
1  
@FiascoLabs: there are at least 7 user agents whose don't observe Crawl-Delay value. I don't want to register, login, and understand their tool to get this. There must be another way to make the Crawl-Delay works. I just reformulate my question to see if this can be done in htaccess file. –  chefnelone May 20 '13 at 7:00
show 5 more comments

1 Answer

I know you previously asked to block the bots and now you just want to slow them down.

The only way to force them to slow down is not to serve them content if they are going too fast. The problem with not serving content is it will surely have negative consequences to your search engine rankings.

In case you decide to block some of the less important bots, you can block a bot's user agent using Apache's configuration like this:

SetEnvIf User-Agent BadBot GoAway=1
Order allow,deny
Allow from all
Deny from env=GoAway

Just replace BadBot with Yandex or whatever the User Agent of the bot you want to block is.

When Apache sees the user agent it will pretend like there is no content available.

Keep in mind the User Agent headers are very easy to change so if Yandex changes theirs to something else, this hack won't work.

share|improve this answer
    
thanks, I'll this this approach. –  chefnelone May 21 '13 at 5:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.