Timeline for Robots.txt allow directory root while disallowing all deeper URLs
Current License: CC BY-SA 4.0
4 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 2, 2019 at 19:59 | comment | added | Stephen Ostermiller♦ |
Worth pointing out that the rule set only works with user agents that support both the Allow directive and regular expressions. Googlebot will obey that properly, and probably Bingbot. Not many other user agents will though.
|
|
| Apr 2, 2019 at 19:00 | comment | added | Maximillian Laumeister♦ | The "$" matches the end of a URL for anyone curious. More info: developers.google.com/search/reference/robots_txt | |
| Apr 2, 2019 at 19:00 | history | edited | Maximillian Laumeister♦ | CC BY-SA 4.0 |
Formatting
|
| Apr 2, 2019 at 18:51 | history | answered | Yaseen | CC BY-SA 4.0 |