I'm looking at a corner case with robots.txt, and am curious if you have any guidance.
Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file.
I've pointed to the Google Resource as our standard for how it should look, but we're running into a question regarding duplicate groups for the same user agent.
Basically, what happens if you have a robots.txt file structured like this:
User-agent: * Disallow: *.asd Disallow: *.exe User-agent: * Disallow: /app/ Disallow: /api/
Would all crawlers interpret this as:
User-agent: * Disallow: *.asd Disallow: *.exe Disallow: /app/ Disallow: /api/
Or would they pick one over the other? Google says: "Only one group of records is valid for a particular crawler.", so I'd interpret this as a crawler is going to pick either group, but not both...
But have no direct experience with duplicate groups/specifying the same user agent more than once in robots.txt.