Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

With the new update of Google algorithm called Penguin, I think my site was being penalized due to webspam. But of course I don't create post which seems to be spam to Google. It is just I think how Google index my site.

I found out that Google index the URL of my site like:

http://www.example.com/comment/reply/3866/26556

So there are so many comment/reply URL index by Google. I have already added:

Disallow: /comment/reply/ Disallow: /?q=comment/reply/

but still Google still index this URL.

Any idea how to prevent Google from indexing comments?

share|improve this question
    
It would be easier to answer this if you posted a link to your site; otherwise it's hard to say more than "there's something wrong with your robots.txt that you need to fix". –  Ilmari Karonen May 2 '12 at 12:36
add comment

4 Answers

You haven't mentioned how long ago you added those Disallow rules. The effect isn't instantaneous, requiring at the very least a wait until you're spidered again, and even then might take a bit longer for them to actually get removed from the index/results.

If you use Webmaster Tools, are they showing up in your "Crawler access" screen(under Site Configuration)? That'll at least give you an idea when the robots.txt file was last grabbed.

share|improve this answer
    
+1: This is a good point, and something I missed in my answer. If you didn't have those URLs disallowed before, just adding disallow rules for them in robots.txt doesn't make Google drop them from their index immediately. (In fact, they may never drop them, at least not if they keep seeing links to them on other pages.) However, disallowing the /comment/reply directory does mean that you can submit a directory removal request for it through Google's Webmaster Tools. –  Ilmari Karonen May 7 '12 at 11:36
add comment

Have you made sure that your robots.txt syntax is correct? If you've signed up for Google's Webmaster Tools, you can use their robots.txt testing tool to see how Googlebot interprets it, but there are also several third-party robots.txt syntax checkers on the web.

You can also add robots meta tags to your reply pages to stop search engines from indexing them. One reason to do this, even if you have the pages disallowed in robots.txt, is that not all bots necessarily understand the fancier robots.txt syntax extensions such as * wildcards, or at least may not understand them the same way.

share|improve this answer
add comment

You can use google webmaster tool: site-configuration section -> site links in order to demote links on your website. You can also use robots.txt as suggested by Ilmari Karonen as well as configure .htaccess (or httpd.conf) to preform 301 redirect

share|improve this answer
add comment

Using disallow in your robots file will not stop Google from indexing those links or pages. That only tells Google do not crawl them.

If those pages are linked to from other pages on your domain they still will index the pages.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.