Is there a difference between an empty robots.txt and no robots.txt at all?

Question

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all.

Do crawlers behave differently in these two cases? So, is it safe to just delete an empty robots.txt?

Answer 1

Do crawlers behave differently in these two cases?

A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling.

You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here.

So, is it safe to just delete an empty robots.txt?

Yes, with the above caveat.

Answer 2

No. There's no difference.

You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.

Answer 3

I don't know when this started, but WordPress auto-generates robots.txt if you don't have one. In my opinion, this is a terrible change. In my case, the auto-generated WordPress robots.txt file is sending robots to my admin page (?!) and this ended up getting indexed in DuckDuckGo, very strange.

WordPress created some kind of "filter" to configure robots.txt behavior, but a) I can't find any code to keep WordPress out of robots.txt entirely, and b) even if I manage to code something, that only affects one theme and one blog at a time.

I'd say this is a flaw in WordPress, assuming we all want an auto-generated robots.txt file.

Creating a "blank" robots.txt forces WordPress to stay out of my business, and sends the appropriate message to robots.

In short, if you see a blank robots.txt used for a WordPress blog (or some other software) this could be the reason: to prevent the auto-generation of something worse than a blank file.