Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

When creating a robots.txt, are the following the same?

# Slurp
User-agent: Slurp
Disallow: /


# Slurp User-agent: Slurp Disallow: /

It seems like for someone disallowing a large number of crawlers at / then a single line notation would be easier to read and realize what is being blocked and what not.

Can one do a single line notation?

share|improve this question

migrated from stackoverflow.com Oct 29 '11 at 3:30

This question came from our site for professional and enthusiast programmers.

No. The RFC says:

The format logically consists of a non-empty set or records,
separated by blank lines. The records consist of a set of lines of
the form:

  <Field> ":" <value>

The formal syntax includes:

agentline    = "User-agent:" *space agent  [comment] CRLF
disallowline = "Disallow" ":" *space path [comment] CRLF
allowline    = "Allow" ":" *space rpath [comment] CRLF
extension    = token : *space value [comment] CRLF

If you have a Google account and use Google webmaster tools, you can use their robots.txt checker.

share|improve this answer
+1 but to carry over one FYI bit from Badger's response, the leading # on the second snippet in the question does comment out the entire line, causing it to do nothing. – Su' Oct 29 '11 at 9:38

Second form do nothing, it's just full-line comment

The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains lines of the form ":". The field name is case insensitive.

Comments can be included in file using UNIX bourne shell conventions: the '#' character is used to indicate that preceding space (if any) and the remainder of the line up to the line termination is discarded. Lines containing only a comment are discarded completely, and therefore do not indicate a record boundary.

User-agent: Slurp Disallow: /

works the same way as multiline definition

It seems like for someone disallowing a large number of crawlers at / then a single line notation would be easier to read

It depends. You can use multiline definition and multi-Useragent per URL because

If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record.

of write oneliner for each crawler

share|improve this answer
-1 Quoting a spec is good, but misinterpreting it to get precisely the wrong answer is unhelpful. – Peter Taylor Oct 29 '11 at 8:16

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.