1

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ?

Disallow: *?s=

3 Answers 3

3
Disallow: *?s=

Bots following the original robots.txt specification would not be allowed to crawl URLs like these:

  • http://example.com/*?s=
  • http://example.com/*?s=foo
  • http://example.com/*?s=/

So they interpret *, ? and = literally (i.e., these characters have to appear at the beginning of the URL path).

But many bots use (their own) extensions to the robots.txt specification, where some characters are reserved, i.e., they get a specific meaning.

Google, for example, uses * for pattern matching:

To block any sequence of characters, use an asterisk (*).

That means the Googlebot is not allowed to crawl URLs like these:

  • http://example.com/?s=
  • http://example.com/?s=foo
  • http://example.com/foo?s=
  • http://example.com/foo?s=bar
  • http://example.com/foo/foo/foo?s=bar

Other bots may have other interpretations.

1

If you want to disallow all robots to crawl your site simply use:

User-agent: *
Disallow: /

User-agent: * means that all robots should follow the rule that comes next. And Disallow: / prevents them to crawl any path. You can see more here on robotstxt.org. I think your Disallow: *?s= means that robots are not allowed to crawl URIs with parameters, but I'm not sure about that.

1
  • The current entry would only block URLs that contain '?s=', but not any other parameters without that specific string.
    – Max
    Commented May 6, 2015 at 9:48
0

Usually, this line disallow inner search results from crawling. The best way to prevent site from crawling is to close it with the password (custom authorization).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.