Can I combine robots.txt rules on one line, separated by whitespace?

Will this work?

User-agent: *  Disallow: /
  • Why would you want to do that? Maybe there's a different way to achieve the same thing... – John Mueller Mar 19 '13 at 20:19

No. See the Google Robots.txt specifications, specifically the File Formats section.

The expected file format is plain text encoded in UTF-8. The file consists of records (lines) separated by CR, CR/LF or LF.

Only valid records will be considered; all other content will be ignored. For example, if the resulting document is a HTML page, only valid text lines will be taken into account, the rest will be discarded without warning or error.

So line breaks between lines are necessary.


The file MUST consist of one or more records separated by one or more blank lines.

Robots.txt must use this format:


Which looks like:

User-agent: *
Disallow: /

Or without optional spaces and more line breaks:



Useful Links

  • Where did you find that empty lines are allowed inside of a record? AFAIR they are only allowed to separate records. – unor Mar 18 '13 at 13:18
  • On the standard for Robot Exclusion page, it details about the use of blank lines and optional spaces. It's important to note their is no REAL 'official robot standards' or guidelines but this is the very first original proposal and considered the standard. – Simon Hayter Mar 18 '13 at 13:21
  • Yeah, that is the spec I linked to. It says: "The file consists of one or more records separated by one or more blank lines. Each record contains lines of the form […]" — and there blank lines are not listed as allowed line inside of a record. – unor Mar 19 '13 at 9:02
  • record = field, so field can be separated by a line. – Simon Hayter Mar 19 '13 at 11:28
  • No, a record is a whole block: "The record starts with one or more User-agent lines, followed by one or more Disallow lines […]". A line starts with a field, a record consists of lines. – unor Mar 21 '13 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.