I had to make an update to an update to a robots.txt to disallow access to sub directory. There was already a disallow statement but i added another one, is the updated robots.txt file below correct ?

The previous robots.txt :

User-Agent: *
Sitemap: http://example.com/sitemap.xml

The updated robots.txt (note the addition of the 3rd line) :

User-Agent: *
Disallow: /terms/*
Sitemap: http://example.com/sitemap.xml

1 Answer 1


You can, but the second statement renders the first obsolete. Essentially, in human-readable terms, you're saying:

All bots
Disallow nothing - so crawl anything you want
But Disallow the "terms" directory - don't crawl that
Sitemap here

Above, you can take out the second line, to tell the bots to not crawl the "terms" directory.

Thus, this would work:

User-Agent: *
Disallow: /terms/
Sitemap: http://example.com/sitemap.xml
  • 1
    I know that Googlebot uses the "longest line that matches" as a rule for which line takes precedence. However, I suspect that some bots would get confused. Commented May 7, 2018 at 15:44
  • 1
    You don't need a blank disallow like: "Disallow: " - So I agree with this comment
    – Michael d
    Commented May 8, 2018 at 9:24
  • 1
    The * in /terms/* should be removed. It’s superfluous for consumers like Google, and harmful for consumers that follow the original robots.txt spec (where * has no special meaning).
    – unor
    Commented May 8, 2018 at 16:44
  • @unor Great catch. While I wouldn't be too concerned with the original spec - as specs are meant to evolve, for example the Allow directive was not in the original spec - I agree that the wildcard is superfluous. I updated the answer accordingly. Thanks for pointing this out! Commented May 8, 2018 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.