Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com,

A) Will the code below do the following for all the crawlers?

User-agent: *

Disallow: /

Allow: /$
Allow: /a/$
Allow: /a/login.php
Allow: /a/login.php?return=/pligg/
  1. Accepting http://stackoverflow.com/ will accept http://stackoverflow.com too?

  2. Accepting http://stackoverflow.com/a/

  3. Accepting http://stackoverflow.com/a/login.php

  4. Accepting http://stackoverflow.com/a/login.php?return=/pligg/

  5. Not accepting any other page on http://stackoverflow.com

B) Which is right: robots.txt or robot.txt?

share|improve this question

2 Answers 2

A) Yes for the most important ones (Googlebot...).

  1. Yes

B) The right file name is robots.txt.

Otherwise, the right code to do what you want is the following:

User-agent: *

Disallow: /

Allow: /a/$
Allow: /a/login.php
Allow: /a/login.php?return=/pligg/
share|improve this answer

Your robots.txt is invalid. Line breaks are not allowed in a record. So it should look like:

User-agent: *
Disallow: /
Allow: /$
Allow: /a/$
Allow: /a/login.php
Allow: /a/login.php?return=/pligg/

Will the code below do the following for all the crawlers?

No, your robots.txt won’t work that way for all crawlers.

Allow is not part of the original robots.txt specification. Only some parsers will understand it (and they might have implemented the wildcards differently), all other parsers will ignore the Allow lines.

share|improve this answer
    
wiil google read Allow ? –  Nok Imchen Sep 6 '13 at 21:37
1  
@Nok Imchen: Yes, Google understands Allow:. More information here –  Zistoloen Sep 6 '13 at 21:44
1  
Do you have a reference line breaks not being allowed? I was unaware of this myself. –  Stephen Ostermiller Nov 28 '13 at 0:20
    
@StephenOstermiller: In the linked spec it says, "The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL)". Later on "record" is defined, by listing lines that may/must exist in a record, which doesn’t contain blank ones (User-agent, Disallow, comments with #, unrecognised headers). So blank lines serve as record (aka. block) boundary exclusively. –  unor Nov 28 '13 at 6:17

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.