Skip to main content
3 of 5
added 155 characters in body
dan
  • 15.2k
  • 11
  • 45
  • 52

Since the OP indicated in comments he was only interested in the search directory, my answer below is in regards to disallowing a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory, conventionally considered to be a file because it lacks a trailing forward slash:

Disallow: /search

In robots.txt, directory names should be proceeded and followed by a forward slash /, as specified in all of these reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /
dan
  • 15.2k
  • 11
  • 45
  • 52