Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

It may be a silly question but I need to clarify my doubt because it's related to robots.txt.

I need to prevent some label path of my website using robots.txt file.

User-agent: *
Disallow: /directory/wp-admin/path/label

If I give like this, would it affect all the paths separately?

Which means, does Googlebot consider directory, wp-admin, path and label separately?

share|improve this question
Hello John, you ask many questions about robots.txt these days, don't hesitate to read pages of this website to understand general functionalities of robots.txt. – Zistoloen Jul 10 '13 at 10:55
up vote 3 down vote accepted

The path specified in the Disallow: field is simply a URL prefix. So, any URL that starts with this prefix will be blocked.

Disallow: /directory/wp-admin/path/label

From your example, this will therefore block all of the following URLs:


But will not block:


Googlebot does not see the separate directories that make up the path. It is simply one value, one URL prefix.

More information on the Google Developers website: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

share|improve this answer

Robots.txt treats each path directive individually...

So, for example:-

Disallow: /directory/wp-admin/path/label

This would disallow crawling of the directory label but not everything preceeding this directory nor following it.

Disallow: /directory/wp-admin/path/label/*

This would disallow the crawling of everything within the directory label , including sub directories and their contents and the directory itself.

share|improve this answer
This is not correct, Disallow: /directory/wp-admin/path/label will block everything "following it". It will block everything that .../label/* blocks and more. The non-standard wildcard * at the end of the value is superfluous. – w3dk Jul 10 '13 at 11:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.