Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

This is totally killing me.

I have a wordpress website about flash games and this is the link if you want to check x1q.com (it is not english)

Well the problem is that I've disabled indexing tags and wp-directories and cetra but what I see is that google still indexing tags and I have a hell lots of tags instead of indexing posts.

I've already submitted the sitemap.xml but these tags keep indexing why? shouldn't google robots understand that I don't want to index them????

Sitemap: http://x1q.com/sitemap.xml

User-agent: *

Disallow: /wp-content/ Disallow: /wp-icludes/ Disallow: /cgi-bin Disallow: /cgi-bin/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /archives/ Disallow: /tag/* Disallow: /tag/ Disallow: /wp-* Disallow: /login/ Disallow: /trackback Disallow: /.js$ Disallow: /.inc$ Disallow: /.css$ Disallow: /.php$ Disallow: /?* Disallow: /? Disallow: /20

User-agent: All Allow: /

User-agent: Googlebot-Image Disallow: /

User-agent: ia_archiver Disallow: /

share|improve this question

2 Answers 2

Keep in mind that the robots.txt controls crawling, not indexing (docs). It's possible that a URL is indexed even if it has never been crawled before. In general, that doesn't matter -- it's not like the URL is going to show up in search results if the rest of your site is indexed normally. If you do want to be certain that these URLs do not show up in search results, you can either:

  • Allow crawling (remove the disallow) and serve a "noindex" robots meta tag with the pages
  • Use the URL removal tools in Google Webmaster Tools to have those URLs removed from Google's index

FWIW I would also reconsider disallowing CSS/JavaScript, as this can be used to generate preview images for your pages. Also, don't use robots.txt as a means of canonicalization (the ".php" and "?" in your robots.txt file), if we can't crawl it, we can't recognize that you have a better version.

share|improve this answer
    
correct me if I'm wrong but using noindex, follow should also help in ranking the post pages as well assuming the tags pages provide valuable links in terms of internal architecture, that would not be counted if crawling was disallowed in the robots.txt. –  Joshak Dec 23 '10 at 12:44

2 things:

  1. When you say Google is still index those pages, do you mean that they're still on Google or can you see Googlebot in your website logs?
  2. It might be that Google hasn't "registered" your robots.txt or your new sitemap and is still using the old one. I would definitely expect this of the sitemap, but not the robots.txt file.
share|improve this answer
    
Thanks Piers. I've created the robot.txt after google started indexing tags. My indexed pages increasing google and I think they are tags (can't tell because they are a lot) but I have cloud tag in the footer of my website is that a problem? because I heard that putting cloud tags on the website keeps google indexing tags. and by the way the sitemap.txt is new U made it after I created robot.txt –  Mohammad Dec 23 '10 at 9:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.