Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I put a robots.txt in my server
This is the code that I putted in

User-agent: *
Disallow: /

I just want my entire site won't be searchable in google.
I also put in the NO INDEX, NO FOLLOW on the header of my site, Here's the code:


I'm wondering why is it that when I try to search the my website in google it is still showing up. what has happened here?
Should this take effect immedietly? or what?

Any ideas would be greatly appreciated as well as rewarded! Thanks! :)

share|improve this question

migrated from stackoverflow.com Dec 9 '11 at 1:32

This question came from our site for professional and enthusiast programmers.

It's possible that search engines have yet to revisit your site since you added the robots.txt file. They won't be aware of the new instructions until they come back to check on your site. – rdlowrey Dec 9 '11 at 1:29
propagation sir. It doesn't happen immediately. – Kai Qing Dec 9 '11 at 1:29
How would I let search engine visit my site? or how would they revisit it again? Thanks! :) – Elson Solano Dec 9 '11 at 1:30

It never takes effect immediately. Google has to go to your site and look at the updated robots.txt file, which can take hours or days.

I suggest you make an account on Google Webmaster Tools. This will give you insight into what Google knows about your site.

I believe other large search engines offer similar functionality. It's worth looking at sometime.

share|improve this answer

From my experience if you google your domain it will show up (I had this myself with a robots.txt-disabled website myself). However, the content will not be indexed.

If your website has already been in Google's SERPs, it will take a while (probably a few weeks to a couple of months) until your content disapears.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.