Within Webmaster Tools I have supplied my test content:

User-agent: *
Disallow: /tag/

When I specify the URL to test against, for example:


It gives me this result:

"Allowed: Detected as a directory; specific files may have different restrictions"

As far as I know I have set this up correctly, shouldn't Google be saying that the /tag/ directory is "disallowed"?

  • Have you set up the exact same host in GWT? (e.g. with resp. without www.?)
    – unor
    Commented Nov 9, 2013 at 11:39

2 Answers 2


If you've just updated your robots.txt file then Google may not have re-read it yet. Give it some time to get re-crawled. Do make sure you have a newline between User-agent: * and Disallow:/admin/.

You could also ask Google to permanently remove specific directories using the URL Removal Tool. From Google's help page:

  1. On the Webmaster Tools home page, click the site you want.
  2. On the left-hand menu, click Optimization and then click Remove URLs.
  3. Click New removal request.
  4. Type the URL of the directory you want removed from search results and then click Continue. (The URL is case-sensitive)
  5. Click Remove directory.
  6. Click Submit Request.

Also take a look at when not to use the URL Removal Tool.

User-agent: *
Disallow: /admin/ 
Disallow: /tag/

Try this, it must work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.