Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I was wondering if it's possible to let bots match two groups of User-Agent: blocks.

I have this block of robots.txt

User-Agent: AhrefsBot   # https://ahrefs.com/robot/
Crawl-Delay: 5

User-agent: *
Disallow: /error.php
Disallow: /cron.php
... and many more rules ...

Now, I wonder if AhrefsBot will now respect the crawl-delay and also respect the rules in user-agent:*? If not, then I'd have to duplicate everything under user-agent:* which would be quite unhandy.

share|improve this question
add comment

1 Answer 1

up vote 1 down vote accepted

Once an User-Agent is mentioned generally it only reads the rules mentioned below it. Most possibly AhrefsBot will not respect rules in user-agent:* and you need to duplicate the rules. Here is example from twitter and facebook robots.txt to support my opinion. They have duplicated rules for all bot specifically mentioned. https://twitter.com/robots.txt https://www.facebook.com/robots.txt

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.