Restricting crawler activity to certain directories with robots.txt

Question

I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers.

In my robots.txt, I have this:

User-Agent: *
Disallow: /compagnies/
Disallow: /floors/
Disallow: /spaces/
Disallow: /buildings/
Disallow: /users/
Disallow: /

I put this file in /mysite/public. I tested the file with a robots.txt validator and got no errors.

However, Google always returns the result of my site. For testing, I added Disallow: /, but again, Google indexed all pages.

floors, spaces, buildings, etc. are not physical directories. Is this a bug? How can I work around it?

Answer 1

If your controllers are not called directly via a URL (e.g. they are web pages that you can pull up in a browser and output HTML but are files that are included via a server side technology) then Google won't index them or even know about them. Google doesn't see your server side code. They only see the HTML it produces.

Now if the pages that include those controllers are being indexed by Google then you need to block those URLs. Simply adding the directories where an included file resides will have absolutely no effect on search engine crawlers.

For example, if http://www.example.com/test/ uses a controller in the /floors/ directory:

• If you want that page to be indexed, do nothing.

• If you do not want http://www.example.com/test/ to be indexed, then you need to add /test/ to your robots.txt file.

• Having /floors/ in your robots.txt will have no effect on http://www.example.com/test/ being indexed because Google doesn't see the /floors/ directory. They only see the output from http://www.example.com/test/.

FYI, Disallow: / tells search engines not to index your site at all. If you want Google to index your content you'll need to remove that line.