Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In failing to check my robots.txt on a community site, it had:

User-agent: *
Disallow: /

I only clued into this because of a message in my AdSense account saying that five URL's couldn't be crawled. Only five, out of about 600 that are currently indexed.

Even though I mistakenly had Disallow: /, why were these pages still indexed and getting traffic?

share|improve this question
Related, possibly duplicate: Why do Google search results include pages disallowed in robots.txt? (Disclosure: Asked and answered by me.) – Ilmari Karonen Jul 24 '13 at 20:27
up vote 7 down vote accepted

Robots.txt just tells search engines not to crawl your pages. It does not tell them to not index your pages. So if your pages have links from other websites the search engines will know they exist. And because off-page factors affect rank, sometimes greatly, your pages can rank well for long-tail search terms without ever being crawled.

To actually prevent search engines from actually indexing your pages you will need to use the x-robots-tag.

share|improve this answer

Using robots.txt is the first step for remove your website from Google search results. You should use "Remove URL Tool" from your Google Webmaster Tools account to tell Google remove your pages or completely your website from its search results.

share|improve this answer

You just indicated to search engines to not crawl your webpages, not to desindex them. That's why, URLs are still indexed and you get traffic from them.

share|improve this answer

Robots.txt doesn't stop indexing, it only stop crawling. So your pages could still be indexed and traffic referred with a disallow : / directive. Though as Google wouldnt be able to crawl your site then you could expect for traffic to diminish in time.

share|improve this answer

Add <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">in your website's index page to stop indexing.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.