5

I want to allow all bots to crawl everything, do I need to create a robots.txt?

If the file is missing will it have a negative influence on some crawlers?

If I need to create it, what should it look like?

1 Answer 1

10

You do not need one as not having one is interpreted as meaning you want to have all of your content crawled. But I recommend using a blank one just to prevent the accumulation of unnecessary 404 errors in your stats.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.