2

I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com,

A) Will the code below do the following for all the crawlers?

User-agent: *

Disallow: /

Allow: /$
Allow: /a/$
Allow: /a/login.php
Allow: /a/login.php?return=/pligg/
  1. Accepting http://stackoverflow.com/ will accept http://stackoverflow.com too?

  2. Accepting http://stackoverflow.com/a/

  3. Accepting http://stackoverflow.com/a/login.php

  4. Accepting http://stackoverflow.com/a/login.php?return=/pligg/

  5. Not accepting any other page on http://stackoverflow.com

B) Which is right: robots.txt or robot.txt?

4

A) Yes for the most important ones (Googlebot...).

  1. Yes

B) The right file name is robots.txt.

Otherwise, the right code to do what you want is the following:

User-agent: *

Disallow: /

Allow: /a/$
Allow: /a/login.php
Allow: /a/login.php?return=/pligg/
3

Your robots.txt is invalid. Line breaks are not allowed in a record. So it should look like:

User-agent: *
Disallow: /
Allow: /$
Allow: /a/$
Allow: /a/login.php
Allow: /a/login.php?return=/pligg/

Will the code below do the following for all the crawlers?

No, your robots.txt won’t work that way for all crawlers.

Allow is not part of the original robots.txt specification. Only some parsers will understand it (and they might have implemented the wildcards differently), all other parsers will ignore the Allow lines.

  • wiil google read Allow ? – Nok Imchen Sep 6 '13 at 21:37
  • 1
    @Nok Imchen: Yes, Google understands Allow:. More information here – Zistoloen Sep 6 '13 at 21:44
  • 1
    Do you have a reference line breaks not being allowed? I was unaware of this myself. – Stephen Ostermiller Nov 28 '13 at 0:20
  • @StephenOstermiller: In the linked spec it says, "The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL)". Later on "record" is defined, by listing lines that may/must exist in a record, which doesn’t contain blank ones (User-agent, Disallow, comments with #, unrecognised headers). So blank lines serve as record (aka. block) boundary exclusively. – unor Nov 28 '13 at 6:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.