Timeline for Do the order of "Disallow" and "Sitemap" lines in robots.txt matter?
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 16, 2019 at 14:00 | comment | added | Henry Visotski | Thank you for the notes and the spec link, @MrWhite -- makes sense. I modified my answer to include a reference to your updates. | |
| Sep 16, 2019 at 13:53 | history | edited | Henry Visotski | CC BY-SA 4.0 |
added 76 characters in body
|
| Sep 16, 2019 at 12:18 | comment | added | MrWhite |
However, all mainstream search engine bots (Googlebot, Bingbot, YandexBot, ...) match according to the longest (most specific) path (which is in accordance with the recently proposed spec - "Robots Exclusion Protocol"), regardless of the order of the rules. So, for the mainstream search bots, it doesn't actually matter where you put the Allow and Disallow directives.
|
|
| Sep 16, 2019 at 12:15 | comment | added | MrWhite |
"What does matter is where you put Allow vs Disallow ... the best practice is to put Allow first" - For maximum compatibility; yes. Although strictly speaking it should be the most specific rule first, regardless of whether it is Allow or Disallow - although Allow directives are expected to be more specific, so they would naturally go first.
|
|
| Sep 8, 2019 at 0:10 | vote | accept | CommunityBot | ||
| Sep 7, 2019 at 7:01 | history | answered | Henry Visotski | CC BY-SA 4.0 |