1

If I want my website to be crawled, do I need an "empty" disallow?

Is there any difference between

User-agent: *
Disallow:
Disallow: /folder/

and

User-agent: *
Disallow: /folder/

I have seen a robots.txt where the first option is used but I don´t understand why.

  • Your second example is correct providing you only intend to block /folder/. – closetnoc Mar 29 '17 at 18:19
2

An empty disallow matches nothing and is ignored. The second example is what you're looking for.

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

http://www.robotstxt.org/robotstxt.html

| improve this answer | |
  • It looks like he wants to allow all crawling except one specific folder – Stephen Ostermiller Mar 29 '17 at 19:45
  • @Stephen This is true. That is why the second example works. The first two lines "allow all robots complete access" as shown in the quote. – grg Mar 29 '17 at 19:46
  • Got it. I thought you were trying to repeat his second option in the quote. – Stephen Ostermiller Mar 29 '17 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.