Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers.

In my robots.txt, I have this:

User-Agent: *
Disallow: /compagnies/
Disallow: /floors/
Disallow: /spaces/
Disallow: /buildings/
Disallow: /users/
Disallow: /

I put this file in /mysite/public. I tested the file with a robots.txt validator and got no errors.

However, Google always returns the result of my site. For testing, I added Disallow: /, but again, Google indexed all pages.

floors, spaces, buildings, etc. are not physical directories. Is this a bug? How can I work around it?

share|improve this question
1  
Changes to robots.txt aren't always immediate. You can find diagnostic information for this in Google Webmaster tools for your site. –  jdeseno Nov 5 '11 at 13:32
    
yes i know, but i make test 1 time a day since 5 days. –  neimad Nov 5 '11 at 14:24
    
While I was editing, I noticed that you're disallowing /compagnies/; did you mean /companies/? –  Pops Nov 18 '11 at 17:50
    
You said that you put the robots.txt file in /mysite/public. Does that mean that the file is at the root of your web site? That is, if I requested http://www.yoursite.com/robots.txt, would I get the file? robots.txt must be at the root of the site. –  Jim Mischel Nov 18 '11 at 19:27
add comment

migrated from stackoverflow.com Nov 19 '11 at 3:53

This question came from our site for professional and enthusiast programmers.

1 Answer

If your controllers are not called directly via a URL (e.g. they are web pages that you can pull up in a browser and output HTML but are files that are included via a server side technology) then Google won't index them or even know about them. Google doesn't see your server side code. They only see the HTML it produces.

Now if the pages that include those controllers are being indexed by Google then you need to block those URLs. Simply adding the directories where an included file resides will have absolutely no effect on search engine crawlers.

For example, if http://www.example.com/test/ uses a controller in the /floors/ directory:

  • If you want that page to be indexed, do nothing.

  • If you do not want http://www.example.com/test/ to be indexed, then you need to add /test/ to your robots.txt file.

  • Having /floors/ in your robots.txt will have no effect on http://www.example.com/test/ being indexed because Google doesn't see the /floors/ directory. They only see the output from http://www.example.com/test/.

FYI, Disallow: / tells search engines not to index your site at all. If you want Google to index your content you'll need to remove that line.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.