Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section.

User-agent: Mediapartners-Google
Disallow: /search
Allow: /

I know that Disallow will prevent Googlebot from indexing a webpage, but I don't understand the usage of Disallow: /search.

What is the exact meaning of Disallow: /search?

share|improve this question
add comment

3 Answers

up vote 2 down vote accepted

In the Disallow field you specify the beginning of URL paths of URLs that should be blocked.

So if you have Disallow: /, it blocks everything, as every URL path starts with /.

If you have Disallow: /a, it blocks all URLs whose paths begin with /a. That could be /a.html, /a/b/c/hello, or /about.

In the same sense, if you have Disallow: /search, it blocks all URLs whose paths begin with the string /search. So it would block the following URLs, for example (if the robots.txt is in http://example.com/):

  • http://example.com/search
  • http://example.com/search.html
  • http://example.com/searchengine
  • http://example.com/search/
  • http://example.com/search/index.html

While the following URLs would still be allowed:

  • http://example.com/foo/search
  • http://example.com/sea

Note that robots.txt doesn’t know/bother if the string matches a directory, a file or nothing at all. It only looks at the characters in the URL.

share|improve this answer
add comment

It tells AdSense not to crawl anything files in the /search directory or below (i.e. any subdirectories of /search).

share|improve this answer
    
John, I think the search/ restriction is not for all user-agents but only for Mediapartners-Google user-agent (AdSense bot). –  Zistoloen Jul 9 '13 at 13:08
    
Small detail that makes quite a difference. Thanks for pointing that out. –  John Conde Jul 9 '13 at 13:10
1  
+1 after editing ;-). –  Zistoloen Jul 9 '13 at 13:13
1  
John code mentioned about /search directory. What does it means? my question is about this only. –  Sathiya Kumar Jul 9 '13 at 13:15
2  
John answered your question, it means bots (AdSense bot only here) are not allowed to access resources under /search directory. For example, AdSense bot doesn't have the right to access on this type of URL: www.example.com/search/ or www.example.com/search/file.html. –  Zistoloen Jul 9 '13 at 13:35
add comment

Since the OP indicated in his comments that he was only interested in the "/search directory", my answer below is in regards to disallowing just a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory:

Disallow: /search

According to the following Google Webmaster Tools help doc below, directory names should be proceeded and followed by a forward slash /, as also specified in the other following reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.