I had to make an update to an update to a robots.txt to disallow access to sub directory. There was already a disallow statement but i added another one, is the updated robots.txt file below correct ?
The previous robots.txt :
User-Agent: *
Disallow:
Sitemap: http://example.com/sitemap.xml
The updated robots.txt (note the addition of the 3rd line) :
User-Agent: *
Disallow:
Disallow: /terms/*
Sitemap: http://example.com/sitemap.xml
You can, but the second statement renders the first obsolete. Essentially, in human-readable terms, you're saying:
All bots
Disallow nothing - so crawl anything you want
But Disallow the "terms" directory - don't crawl that
Sitemap here
Above, you can take out the second line, to tell the bots to not crawl the "terms" directory.
Thus, this would work:
User-Agent: *
Disallow: /terms/
Sitemap: http://example.com/sitemap.xml
* in /terms/* should be removed. It’s superfluous for consumers like Google, and harmful for consumers that follow the original robots.txt spec (where * has no special meaning).