Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I was wondering if it's possible to let bots match two groups of User-Agent: blocks.

I have this block of robots.txt

User-Agent: AhrefsBot   # https://ahrefs.com/robot/
Crawl-Delay: 5

User-agent: *
Disallow: /error.php
Disallow: /cron.php
... and many more rules ...

Now, I wonder if AhrefsBot will now respect the crawl-delay and also respect the rules in user-agent:*? If not, then I'd have to duplicate everything under user-agent:* which would be quite unhandy.

share|improve this question
up vote 1 down vote accepted

Once an User-Agent is mentioned generally it only reads the rules mentioned below it. Most possibly AhrefsBot will not respect rules in user-agent:* and you need to duplicate the rules. Here is example from twitter and facebook robots.txt to support my opinion. They have duplicated rules for all bot specifically mentioned. https://twitter.com/robots.txt https://www.facebook.com/robots.txt

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.