In failing to check my robots.txt on a community site, it had:
User-agent: *
Disallow: /
I only clued into this because of a message in my AdSense account saying that five URL's couldn't be crawled. Only five, out of about 600 that are currently indexed.
Even though I mistakenly had Disallow: /, why were these pages still indexed and getting traffic?
Robots.txt just tells search engines not to crawl your pages. It does not tell them to not index your pages. So if your pages have links from other websites the search engines will know they exist. And because off-page factors affect rank, sometimes greatly, your pages can rank well for long-tail search terms without ever being crawled.
To actually prevent search engines from actually indexing your pages you will need to use the x-robots-tag.
Using robots.txt is the first step for remove your website from Google search results. You should use "Remove URL Tool" from your Google Webmaster Tools account to tell Google remove your pages or completely your website from its search results.
You just indicated to search engines to not crawl your webpages, not to desindex them. That's why, URLs are still indexed and you get traffic from them.
Robots.txt doesn't stop indexing, it only stop crawling. So your pages could still be indexed and traffic referred with a disallow : / directive. Though as Google wouldnt be able to crawl your site then you could expect for traffic to diminish in time.
Add <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">in your website's index page to stop indexing.
