2

I was wondering if it's possible to let bots match two groups of User-Agent: blocks.

I have this block of robots.txt

User-Agent: AhrefsBot   # https://ahrefs.com/robot/
Crawl-Delay: 5

User-agent: *
Disallow: /error.php
Disallow: /cron.php
... and many more rules ...

Now, I wonder if AhrefsBot will now respect the crawl-delay and also respect the rules in user-agent:*? If not, then I'd have to duplicate everything under user-agent:* which would be quite unhandy.

1

Once an User-Agent is mentioned generally it only reads the rules mentioned below it. Most possibly AhrefsBot will not respect rules in user-agent:* and you need to duplicate the rules. Here is example from twitter and facebook robots.txt to support my opinion. They have duplicated rules for all bot specifically mentioned. https://twitter.com/robots.txt https://www.facebook.com/robots.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.