So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following:
# go away
User-agent: *
Disallow: /
Now, I'm pretty competent at reading computer, and as far as I can tell, this says ALL spiders shouldn't look at ANYTHING in the root directory or below.
Am I reading this correctly? Because that just seems insane.
Maybe someone didn't want to pay for spider traffic?
Regardless, you are reading it correctly:
http://www.robotstxt.org/robotstxt.html
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
User-agent: *
Disallow: /
The
"User-agent: *"means this section applies to all robots. The"Disallow: /"tells the robot that it should not visit any pages on the site.
I've put this sort of robots.txt in place when first developing a site because I don't want it to be indexed by Google and others before it's ready.
I've also forgotten to edit that after the site has gone live. *facepalm*
