I'm about to deploy a website for testing and I want to let all the indexing bots to not index my site during this phase.

According to Google Webmasters Help section apparently you only need the following in your robots.txt file:

Disallow: /

But according to http://www.robotstxt.org/faq/prevent.html and other sources you have to write it like this:

User-agent: *
Disallow: /

I'm afraid that the bots still will index my site if I leave the User-agent: * part out.

So which is the proper way to do it?


The original robots.txt specification is http://www.robotstxt.org/orig.html, where it says:

The record starts with one or more User-agent lines […]

So yes, User-agent is required.

| improve this answer | |

As far as I know and as far as I have search for it, there is no robots.txt documentation saying you can drop the user-agent line. I would not rely on this.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.