I was wondering if it's possible to let bots match two groups of
I have this block of robots.txt
User-Agent: AhrefsBot # https://ahrefs.com/robot/ Crawl-Delay: 5 User-agent: * Disallow: /error.php Disallow: /cron.php ... and many more rules ...
Now, I wonder if AhrefsBot will now respect the crawl-delay and also respect the rules in
If not, then I'd have to duplicate everything under
user-agent:* which would be quite unhandy.