I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers.

In my robots.txt, I have this:

User-Agent: *
Disallow: /compagnies/
Disallow: /floors/
Disallow: /spaces/
Disallow: /buildings/
Disallow: /users/
Disallow: /

I put this file in /mysite/public. I tested the file with a robots.txt validator and got no errors.

However, Google always returns the result of my site. For testing, I added Disallow: /, but again, Google indexed all pages.

floors, spaces, buildings, etc. are not physical directories. Is this a bug? How can I work around it?

  • 1
    Changes to robots.txt aren't always immediate. You can find diagnostic information for this in Google Webmaster tools for your site.
    – jdeseno
    Nov 5, 2011 at 13:32
  • yes i know, but i make test 1 time a day since 5 days.
    – neimad
    Nov 5, 2011 at 14:24
  • While I was editing, I noticed that you're disallowing /compagnies/; did you mean /companies/?
    – Pops
    Nov 18, 2011 at 17:50
  • You said that you put the robots.txt file in /mysite/public. Does that mean that the file is at the root of your web site? That is, if I requested http://www.yoursite.com/robots.txt, would I get the file? robots.txt must be at the root of the site. Nov 18, 2011 at 19:27

1 Answer 1


If your controllers are not called directly via a URL (e.g. they are web pages that you can pull up in a browser and output HTML but are files that are included via a server side technology) then Google won't index them or even know about them. Google doesn't see your server side code. They only see the HTML it produces.

Now if the pages that include those controllers are being indexed by Google then you need to block those URLs. Simply adding the directories where an included file resides will have absolutely no effect on search engine crawlers.

For example, if http://www.example.com/test/ uses a controller in the /floors/ directory:

  • If you want that page to be indexed, do nothing.

  • If you do not want http://www.example.com/test/ to be indexed, then you need to add /test/ to your robots.txt file.

  • Having /floors/ in your robots.txt will have no effect on http://www.example.com/test/ being indexed because Google doesn't see the /floors/ directory. They only see the output from http://www.example.com/test/.

FYI, Disallow: / tells search engines not to index your site at all. If you want Google to index your content you'll need to remove that line.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.