Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following:

# go away
User-agent: *
Disallow: /

Now, I'm pretty competent at reading computer, and as far as I can tell, this says ALL spiders shouldn't look at ANYTHING in the root directory or below.

Am I reading this correctly? Because that just seems insane.

share|improve this question

migrated from superuser.com Dec 15 '11 at 14:27

This question came from our site for computer enthusiasts and power users.

2 Answers 2

up vote 9 down vote accepted

Maybe someone didn't want to pay for spider traffic?

Regardless, you are reading it correctly:

http://www.robotstxt.org/robotstxt.html

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

 User-agent: *
 Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

share|improve this answer

I've put this sort of robots.txt in place when first developing a site because I don't want it to be indexed by Google and others before it's ready.

I've also forgotten to edit that after the site has gone live. *facepalm*

share|improve this answer
    
The problem with this is that this is a site for a multi-million dollar web design company, with over 35 employees, which has been around for over 15 years! –  Andrew Alexander Dec 15 '11 at 14:34
    
Umm, choke, ROFL –  Fiasco Labs Dec 16 '11 at 3:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.