2

I'm just curious because I can't seem to find any mention of it anywhere. What is the standard (or at the very least typical) response to robots.txt without a user-agent parameter? Does it assume the following rules apply to it just as User-agent: * would or does it ignore them completely?

For example, would a robots.txt containing only Disallow: / block all/most standard crawlers or would it do nothing at all?

1
  • 1
    your question is impossible to answer. They all behave differently. There are rules (robotstxt.org) but no one has to follow them
    – ojf
    Aug 17, 2015 at 21:12

2 Answers 2

2

Bots would simply ignore this.

I'd expect them to look for specific user agents that they need to take note of. So Googlebot for example would look for User-agent: * or User-agent: Googlebot.

So as those two are missing then the'd probably go ahead and crawl the entire site.

We all know that adding user-agents is essential in robots.txt so there's no point trying to second guess them.

1

A Disallow field that does not follow a User-agent field (ie. is not part of a group) is invalid. Bots should ignore this as being invalid. However, the rules that govern robots.txt are not a strict standard, so as with anything "invalid" in this respect, robot behaviour could be unpredicatble.

The Original robots.txt standard (1994) simply states:

The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below. Unrecognised headers are ignored.

In this respect, a Disallow field could be seen as an "unrecognised header". (?)

In the 1997 Internet Draft specification: A Method for Web Robots Control it states, with respect to the User-agent line:

If no such record exists [ie. a specific user-agent match], it should obey the first record with a User-agent line with a "*" value, if present. If no record satisfied either condition, or no records are present at all, access is unlimited.

The Google docs (Robots.txt specifications) state that a disallow field is "only valid as a group-member record" and a user-agent field indicates the "start of group". The Google docs also include a more formal definition which indicates that an entry must start with a "startgroupline" (ie. a user-agent field).

If you test this in the "robots.txt Tester" in Google Search Console (formerly Google Webmaster Tools) then any Disallow field that is not part of a group (ie. does not start with a User-agent field) is simply flagged as an error and is ignored.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.