If I want my website to be crawled, do I need an "empty" disallow?
Is there any difference between
User-agent: *
Disallow:
Disallow: /folder/
and
User-agent: *
Disallow: /folder/
I have seen a robots.txt where the first option is used but I don“t understand why.