To allow crawling of all but a specific folder, do I need to include an empty disallow directive in robots.txt?

Question

If I want my website to be crawled, do I need an "empty" disallow?

Is there any difference between

User-agent: *
Disallow:
Disallow: /folder/

and

User-agent: *
Disallow: /folder/

I have seen a robots.txt where the first option is used but I don´t understand why.

Your second example is correct providing you only intend to block /folder/. — closetnoc, Mar 29 '17 at 18:19

grg · Accepted Answer · 2017-03-29 19:44:59Z

2

An empty disallow matches nothing and is ignored. The second example is what you're looking for.

To allow all robots complete access
User-agent: *
Disallow:
(or just create an empty "/robots.txt" file, or don't use one at all)

answered Mar 29 '17 at 18:56

grg

2,6771 gold badge12 silver badges21 bronze badges

It looks like he wants to allow all crawling except one specific folder – Stephen Ostermiller♦ Mar 29 '17 at 19:45
@Stephen This is true. That is why the second example works. The first two lines "allow all robots complete access" as shown in the quote. – grg Mar 29 '17 at 19:46
Got it. I thought you were trying to repeat his second option in the quote. – Stephen Ostermiller♦ Mar 29 '17 at 19:53

add a comment |

1 Answer 1