20

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a simple robots.txt file that will prevent the 404 errors from appearing in my log file.

What is a minimum valid robots.txt file that will allow everything on the site to be crawled?

3 Answers 3

24

As indicated here, create a text file named robots.txt in the top-level directory of your web server. You can leave it empty, or add:

User-agent: *
Disallow:

If you want robots to crawl everything. If not, then see the above link for more examples.

5
  • Why add "Disallow:" and not just "Allow: *"? Jan 5, 2014 at 14:18
  • 3
    @Patrik "Allow" is for overriding any previous "Disallow" directives. It is meaningless if there is no "Disallow". Honestly the best solution is a blank file. Jan 5, 2014 at 19:23
  • 2
    Ah, I see. I also agree that a blank file is the best. Jan 5, 2014 at 21:40
  • 2
    @PatrikAlienus Because "Allow" is not in robots.txt specification.
    – user11153
    Mar 6, 2015 at 12:37
  • 1
    @user11153: Huh? What about section "3.2.2 The Allow and Disallow lines" of the 1997 Internet Draft specification A Method for Web Robots Control?
    – David Cary
    Mar 8, 2015 at 12:25
5

The best minimal robots.txt is a completely empty file.

Any other "null" directives such as an empty Disallow or Allow: * are not only useless because they are no-ops, but add unneeded complexity.

If you don't want the file to be completely empty - or you want to make it more human-readable - simply add a comment beginning with the # character, such as # blank file allows all. Crawlers ignore lines starting with #.

-1

I would say this;

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

It will allow Google to crawl everything but will disallow Google to Crawl your aadminn panel. Which is an ideal situation for you.

1
  • 3
    I may be missing something, but I don't think the asker said they are using Wordpress. Jan 27, 2019 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.