I disabled indexing WP tags,wp-contents,etc in robot.txt but it google still indexing!

Question

This is totally killing me.

I have a wordpress website about flash games and this is the link if you want to check x1q.com (it is not english)

Well the problem is that I've disabled indexing tags and wp-directories and cetra but what I see is that google still indexing tags and I have a hell lots of tags instead of indexing posts.

I've already submitted the sitemap.xml but these tags keep indexing why? shouldn't google robots understand that I don't want to index them????

Sitemap: http://x1q.com/sitemap.xml

User-agent: *

Disallow: /wp-content/ Disallow: /wp-icludes/ Disallow: /cgi-bin Disallow: /cgi-bin/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /archives/ Disallow: /tag/* Disallow: /tag/ Disallow: /wp-* Disallow: /login/ Disallow: /trackback Disallow: /.js$ Disallow: /.inc$ Disallow: /.css$ Disallow: /.php$ Disallow: /?* Disallow: /? Disallow: /20

User-agent: All Allow: /

User-agent: Googlebot-Image Disallow: /

User-agent: ia_archiver Disallow: /

Answer 1

Keep in mind that the robots.txt controls crawling, not indexing (docs). It's possible that a URL is indexed even if it has never been crawled before. In general, that doesn't matter -- it's not like the URL is going to show up in search results if the rest of your site is indexed normally. If you do want to be certain that these URLs do not show up in search results, you can either:

• Allow crawling (remove the disallow) and serve a "noindex" robots meta tag with the pages
• Use the URL removal tools in Google Webmaster Tools to have those URLs removed from Google's index

FWIW I would also reconsider disallowing CSS/JavaScript, as this can be used to generate preview images for your pages. Also, don't use robots.txt as a means of canonicalization (the ".php" and "?" in your robots.txt file), if we can't crawl it, we can't recognize that you have a better version.

Answer 2

2 things:

1. When you say Google is still index those pages, do you mean that they're still on Google or can you see Googlebot in your website logs?
2. It might be that Google hasn't "registered" your robots.txt or your new sitemap and is still using the old one. I would definitely expect this of the sitemap, but not the robots.txt file.