2

I had to make an update to an update to a robots.txt to disallow access to sub directory. There was already a disallow statement but i added another one, is the updated robots.txt file below correct ?

The previous robots.txt :

User-Agent: *
Disallow: 
Sitemap: http://example.com/sitemap.xml

The updated robots.txt (note the addition of the 3rd line) :

User-Agent: *
Disallow: 
Disallow: /terms/*
Sitemap: http://example.com/sitemap.xml
3

You can, but the second statement renders the first obsolete. Essentially, in human-readable terms, you're saying:

All bots
Disallow nothing - so crawl anything you want
But Disallow the "terms" directory - don't crawl that
Sitemap here

Above, you can take out the second line, to tell the bots to not crawl the "terms" directory.

Thus, this would work:

User-Agent: *
Disallow: /terms/
Sitemap: http://example.com/sitemap.xml
| improve this answer | |
  • 1
    I know that Googlebot uses the "longest line that matches" as a rule for which line takes precedence. However, I suspect that some bots would get confused. – Stephen Ostermiller May 7 '18 at 15:44
  • 1
    You don't need a blank disallow like: "Disallow: " - So I agree with this comment – Michael d May 8 '18 at 9:24
  • 1
    The * in /terms/* should be removed. It’s superfluous for consumers like Google, and harmful for consumers that follow the original robots.txt spec (where * has no special meaning). – unor May 8 '18 at 16:44
  • @unor Great catch. While I wouldn't be too concerned with the original spec - as specs are meant to evolve, for example the Allow directive was not in the original spec - I agree that the wildcard is superfluous. I updated the answer accordingly. Thanks for pointing this out! – Henry Visotski May 8 '18 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.