So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following:

# go away
User-agent: *
Disallow: /

Now, I'm pretty competent at reading computer, and as far as I can tell, this says ALL spiders shouldn't look at ANYTHING in the root directory or below.

Am I reading this correctly? Because that just seems insane.

2 Answers 2


Maybe someone didn't want to pay for spider traffic?

Regardless, you are reading it correctly:


Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

 User-agent: *
 Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.


I've put this sort of robots.txt in place when first developing a site because I don't want it to be indexed by Google and others before it's ready.

I've also forgotten to edit that after the site has gone live. *facepalm*

  • The problem with this is that this is a site for a multi-million dollar web design company, with over 35 employees, which has been around for over 15 years! Dec 15, 2011 at 14:34
  • Umm, choke, ROFL Dec 16, 2011 at 3:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.