Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I want to allow all bots to crawl everything, do I need to create a robots.txt?

If the file is missing will it have a negative influence on some crawlers?

If I need to create it, what should it look like?

share|improve this question
add comment

1 Answer

up vote 9 down vote accepted

You do not need one as not having one is interpreted as meaning you want to have all of your content crawled. But I recommend using a blank one just to prevent the accumulation of unnecessary 404 errors in your stats.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.