1

If I want my website to be crawled, do I need an "empty" disallow?

Is there any difference between

User-agent: *
Disallow:
Disallow: /folder/

and

User-agent: *
Disallow: /folder/

I have seen a robots.txt where the first option is used but I don´t understand why.

1
  • Your second example is correct providing you only intend to block /folder/.
    – closetnoc
    Mar 29, 2017 at 18:19

1 Answer 1

2

An empty disallow matches nothing and is ignored. The second example is what you're looking for.

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

http://www.robotstxt.org/robotstxt.html

3
  • It looks like he wants to allow all crawling except one specific folder Mar 29, 2017 at 19:45
  • @Stephen This is true. That is why the second example works. The first two lines "allow all robots complete access" as shown in the quote.
    – grg
    Mar 29, 2017 at 19:46
  • Got it. I thought you were trying to repeat his second option in the quote. Mar 29, 2017 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.