7

So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following:

# go away
User-agent: *
Disallow: /

Now, I'm pretty competent at reading computer, and as far as I can tell, this says ALL spiders shouldn't look at ANYTHING in the root directory or below.

Am I reading this correctly? Because that just seems insane.

10

Maybe someone didn't want to pay for spider traffic?

Regardless, you are reading it correctly:

http://www.robotstxt.org/robotstxt.html

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

 User-agent: *
 Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

|improve this answer|||||
1

I've put this sort of robots.txt in place when first developing a site because I don't want it to be indexed by Google and others before it's ready.

I've also forgotten to edit that after the site has gone live. *facepalm*

|improve this answer|||||
  • The problem with this is that this is a site for a multi-million dollar web design company, with over 35 employees, which has been around for over 15 years! – Andrew Alexander Dec 15 '11 at 14:34
  • Umm, choke, ROFL – Fiasco Labs Dec 16 '11 at 3:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.