Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to allow all bots to crawl everything, do I need to create a robots.txt?

If the file is missing will it have a negative influence on some crawlers?

If I need to create it, what should it look like?

share|improve this question
up vote 10 down vote accepted

You do not need one as not having one is interpreted as meaning you want to have all of your content crawled. But I recommend using a blank one just to prevent the accumulation of unnecessary 404 errors in your stats.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.