Skip to main content
deleted 82 characters in body
Source Link
dan
  • 15.2k
  • 11
  • 45
  • 52

Since the OP indicated in his comments that he was only interested in the "/search directory", my answer below is in regards to disallowing just a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory, conventionally considered to be a file because it lacks a trailing forward slash:

Disallow: /search

According to the following Google Webmaster Tools help doc below, directory names should be proceeded and followed by a forward slash /, as also specified in the other following reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /

Since the OP indicated in his comments that he was only interested in the "/search directory", my answer below is in regards to disallowing just a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory, conventionally considered to be a file because it lacks a trailing forward slash:

Disallow: /search

According to the following Google Webmaster Tools help doc below, directory names should be proceeded and followed by a forward slash /, as also specified in the other following reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /

Since the OP indicated in his comments that he was only interested in the "/search directory", my answer below is in regards to disallowing just a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory:

Disallow: /search

According to the following Google Webmaster Tools help doc below, directory names should be proceeded and followed by a forward slash /, as also specified in the other following reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /
Clarifications.
Source Link
dan
  • 15.2k
  • 11
  • 45
  • 52

Since the OP indicated in his comments that he was only interested in the search"/search directory", my answer below is in regards to disallowing just a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory, conventionally considered to be a file because it lacks a trailing forward slash:

Disallow: /search

In robots.txtAccording to the following Google Webmaster Tools help doc below, directory names should be proceeded and followed by a forward slash /, as also specified in all of thesethe other following reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /

Since the OP indicated in comments he was only interested in the search directory, my answer below is in regards to disallowing a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory, conventionally considered to be a file because it lacks a trailing forward slash:

Disallow: /search

In robots.txt, directory names should be proceeded and followed by a forward slash /, as specified in all of these reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /

Since the OP indicated in his comments that he was only interested in the "/search directory", my answer below is in regards to disallowing just a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory, conventionally considered to be a file because it lacks a trailing forward slash:

Disallow: /search

According to the following Google Webmaster Tools help doc below, directory names should be proceeded and followed by a forward slash /, as also specified in the other following reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /
added 155 characters in body
Source Link
dan
  • 15.2k
  • 11
  • 45
  • 52

Since the OP indicated in comments he was only interested in the search directory, my answer below is in regards to disallowing a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory, conventionally considered to be a file because it lacks a trailing forward slash:

Disallow: /search

In robots.txt, directory names should be proceeded and followed by a forward slash /, as specified in all of these reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /

The following is a directive for robots not to crawl something named "search" located in the root directory, conventionally considered to be a file because it lacks a trailing forward slash:

Disallow: /search

In robots.txt, directory names should be proceeded and followed by a forward slash /, as specified in all of these reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /

Since the OP indicated in comments he was only interested in the search directory, my answer below is in regards to disallowing a "search" directory:

The following is a directive for robots not to crawl something named "search" located in the root directory, conventionally considered to be a file because it lacks a trailing forward slash:

Disallow: /search

In robots.txt, directory names should be proceeded and followed by a forward slash /, as specified in all of these reference sources:

Google Webmaster Tools - Block or remove pages using a robots.txt file

To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/

Robotstxt.org - What to put in it

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Wikipedia - Robots exclusion standard

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

So according to Google (as copied above), the following would disallow bots with the user-agent Mediapartners-Google from crawling the "search" directory located in the the root directory, but allow all other directories to be crawled:

User-agent: Mediapartners-Google
Disallow: /search/
Allow: /
Post Undeleted by dan
Post Deleted by dan
Formatting.
Source Link
dan
  • 15.2k
  • 11
  • 45
  • 52
Loading
Source Link
dan
  • 15.2k
  • 11
  • 45
  • 52
Loading