1

If I want my website to be crawled, do I need an "empty" disallow?

Is there any difference between

User-agent: *
Disallow:
Disallow: /folder/

and

User-agent: *
Disallow: /folder/

I have seen a robots.txt where the first option is used but I don“t understand why.

  • Your second example is correct providing you only intend to block /folder/. – closetnoc Mar 29 '17 at 18:19
2

An empty disallow matches nothing and is ignored. The second example is what you're looking for.

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

http://www.robotstxt.org/robotstxt.html

  • It looks like he wants to allow all crawling except one specific folder – Stephen Ostermiller Mar 29 '17 at 19:45
  • @Stephen This is true. That is why the second example works. The first two lines "allow all robots complete access" as shown in the quote. – grg Mar 29 '17 at 19:46
  • Got it. I thought you were trying to repeat his second option in the quote. – Stephen Ostermiller Mar 29 '17 at 19:53

Your Answer

By clicking ā€œPost Your Answerā€, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.