Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Within Webmaster Tools I have supplied my test content:

User-agent: *
Disallow:/admin/
Disallow: /tag/

When I specify the URL to test against, for example:

http://www.site.com/tag/

It gives me this result:

"Allowed: Detected as a directory; specific files may have different restrictions"

As far as I know I have set this up correctly, shouldn't Google be saying that the /tag/ directory is "disallowed"?

share|improve this question
    
Have you set up the exact same host in GWT? (e.g. with resp. without www.?) –  unor Nov 9 '13 at 11:39
add comment

2 Answers

If you've just updated your robots.txt file then Google may not have re-read it yet. Give it some time to get re-crawled. Do make sure you have a newline between User-agent: * and Disallow:/admin/.

You could also ask Google to permanently remove specific directories using the URL Removal Tool. From Google's help page:

  1. On the Webmaster Tools home page, click the site you want.
  2. On the left-hand menu, click Optimization and then click Remove URLs.
  3. Click New removal request.
  4. Type the URL of the directory you want removed from search results and then click Continue. (The URL is case-sensitive)
  5. Click Remove directory.
  6. Click Submit Request.

Also take a look at when not to use the URL Removal Tool.

share|improve this answer
add comment
User-agent: *
Disallow: /admin/ 
Disallow: /tag/

Try this, it must work.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.