Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

It may be a silly question but I need to clarify my doubt because it's related to robots.txt.

I need to prevent some label path of my website using robots.txt file.

User-agent: *
Disallow: /directory/wp-admin/path/label

If I give like this, would it affect all the paths separately?

Which means, does Googlebot consider directory, wp-admin, path and label separately?

share|improve this question
1  
Hello John, you ask many questions about robots.txt these days, don't hesitate to read pages of this website to understand general functionalities of robots.txt. –  Zistoloen Jul 10 '13 at 10:55
add comment

2 Answers

up vote 3 down vote accepted

The path specified in the Disallow: field is simply a URL prefix. So, any URL that starts with this prefix will be blocked.

Disallow: /directory/wp-admin/path/label

From your example, this will therefore block all of the following URLs:

/directory/wp-admin/path/label
/directory/wp-admin/path/labelfoo
/directory/wp-admin/path/label/
/directory/wp-admin/path/label/bar.html

But will not block:

/directory/wp-admin/path/foo
/directory/wp-admin/path/
/directory/wp-admin/hello.html
:

Googlebot does not see the separate directories that make up the path. It is simply one value, one URL prefix.

More information on the Google Developers website: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

share|improve this answer
add comment

Robots.txt treats each path directive individually...

So, for example:-

User-agent:*
Disallow: /directory/wp-admin/path/label

This would disallow crawling of the directory label but not everything preceeding this directory nor following it.

User-agent:*
Disallow: /directory/wp-admin/path/label/*

This would disallow the crawling of everything within the directory label , including sub directories and their contents and the directory itself.

share|improve this answer
    
This is not correct, Disallow: /directory/wp-admin/path/label will block everything "following it". It will block everything that .../label/* blocks and more. The non-standard wildcard * at the end of the value is superfluous. –  w3d Jul 10 '13 at 11:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.