Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Can I combine robots.txt rules on one line, separated by whitespace?

Will this work?

User-agent: *  Disallow: /
share|improve this question
    
Why would you want to do that? Maybe there's a different way to achieve the same thing... –  John Mueller Mar 19 '13 at 20:19
add comment

2 Answers 2

up vote 8 down vote accepted

No. See the Google Robots.txt specifications, specifically the File Formats section.

The expected file format is plain text encoded in UTF-8. The file consists of records (lines) separated by CR, CR/LF or LF.

Only valid records will be considered; all other content will be ignored. For example, if the resulting document is a HTML page, only valid text lines will be taken into account, the rest will be discarded without warning or error.

So line breaks between lines are necessary.

share|improve this answer
add comment

The file MUST consist of one or more records separated by one or more blank lines.

Robots.txt must use this format:

<field>:<optionalspace><value><optionalspace>
<field>:<optionalspace><value><optionalspace>

Which looks like:

User-agent: *
Disallow: /

Or without optional spaces and more line breaks:

User-agent:*

Disallow:/

Useful Links

share|improve this answer
    
Where did you find that empty lines are allowed inside of a record? AFAIR they are only allowed to separate records. –  unor Mar 18 '13 at 13:18
    
On the standard for Robot Exclusion page, it details about the use of blank lines and optional spaces. It's important to note their is no REAL 'official robot standards' or guidelines but this is the very first original proposal and considered the standard. –  bybe Mar 18 '13 at 13:21
    
Yeah, that is the spec I linked to. It says: "The file consists of one or more records separated by one or more blank lines. Each record contains lines of the form […]" — and there blank lines are not listed as allowed line inside of a record. –  unor Mar 19 '13 at 9:02
    
record = field, so field can be separated by a line. –  bybe Mar 19 '13 at 11:28
    
No, a record is a whole block: "The record starts with one or more User-agent lines, followed by one or more Disallow lines […]". A line starts with a field, a record consists of lines. –  unor Mar 21 '13 at 9:03
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.