2

I'm just curious because I can't seem to find any mention of it anywhere. What is the standard (or at the very least typical) response to robots.txt without a user-agent parameter? Does it assume the following rules apply to it just as User-agent: * would or does it ignore them completely?

For example, would a robots.txt containing only Disallow: / block all/most standard crawlers or would it do nothing at all?

1
  • 1
    your question is impossible to answer. They all behave differently. There are rules (robotstxt.org) but no one has to follow them
    – ojf
    Aug 17 '15 at 21:12
2

Bots would simply ignore this.

I'd expect them to look for specific user agents that they need to take note of. So Googlebot for example would look for User-agent: * or User-agent: Googlebot.

So as those two are missing then the'd probably go ahead and crawl the entire site.

We all know that adding user-agents is essential in robots.txt so there's no point trying to second guess them.

1

A Disallow field that does not follow a User-agent field (ie. is not part of a group) is invalid. Bots should ignore this as being invalid. However, the rules that govern robots.txt are not a strict standard, so as with anything "invalid" in this respect, robot behaviour could be unpredicatble.

The Original robots.txt standard (1994) simply states:

The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below. Unrecognised headers are ignored.

In this respect, a Disallow field could be seen as an "unrecognised header". (?)

In the 1997 Internet Draft specification: A Method for Web Robots Control it states, with respect to the User-agent line:

If no such record exists [ie. a specific user-agent match], it should obey the first record with a User-agent line with a "*" value, if present. If no record satisfied either condition, or no records are present at all, access is unlimited.

The Google docs (Robots.txt specifications) state that a disallow field is "only valid as a group-member record" and a user-agent field indicates the "start of group". The Google docs also include a more formal definition which indicates that an entry must start with a "startgroupline" (ie. a user-agent field).

If you test this in the "robots.txt Tester" in Google Search Console (formerly Google Webmaster Tools) then any Disallow field that is not part of a group (ie. does not start with a User-agent field) is simply flagged as an error and is ignored.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy