Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself.

I've not engaged in any shady link-building (or indeed any link-building at all!) and the site is technically fine (fast pages, no malware, fully responsive). So my first guess is that I'm being penalized for duplicate content. Although there's a huge amount of rich content on the site, there's also a lot of algorithmically-generated pages - for example, one for each town in the UK. (This isn't done for SEO, but I guess maybe Google thinks it is.)

So I need to stop Googlebot finding, and objecting to, this content.

I would rather do it via a meta tag (<meta name="googlebot" content="noindex" />) on the relevant pages than with using robots.txt.

So my question are:

  • Is a meta tag a workable alternative to using robots.txt for hiding "problematic" content?
  • Is there anything else I should be doing?
  • 1
    It's very unlike that Google has punished you for content that is accessibly by various ways, for example the mega popular WordPress by default doesn't use canonical links and as you can expect snippets of blog entries across lots of pages when using tags and categories. Also you shouldn't need to deindex pages, simply use canonical links on the pages with the content. Commented Jun 23, 2014 at 10:01
  • What is a non SEO reason for generating duplicate content for each town in the UK?
    – JamesRyan
    Commented Jun 23, 2014 at 11:08
  • I'd noindex the auto-generated city pages until you have some real valuable content on.
    – user29671
    Commented Jun 23, 2014 at 11:23
  • @JamesRyan: two reasons really - one is that it's a placeholder for future content, whether by me (e.g. when I write news stories that relate to that town) or user-generated (and there's plenty of pointers towards that). The other is that there is some content, but it's map-based rather than textual - which means it's JavaScript and pretty much unindexable. Commented Jun 23, 2014 at 16:57

1 Answer 1


If you think that you have been penalized for duplicate or low quality content do NOT block indexing from robots.txt. Googlebot has already indexed these pages and it must have access to them so it can remove them.

What to do

  1. Add a 301 redirection to all duplicate pages, so you have only 1 page for each topic
  2. Add the noindex meta tag in all pages that you think that are low quality auto generated, like the tag pages, search results, and any other feed pages, so that Googlebot will remove these pages from the index and pass the juice to the internal links.


It is perfectly fine to have auto generated content/feed pages, but it depends on the amount. If your website has 100 unique pages and 10000 tag pages this is a disaster. If you have 10000 unique pages and 50 tag pages this is excellent.

  • Not sure Google has ever specified needing to crawl 2-3 times to act on noindex tag. "When we see the noindex meta tag on a page, Google will completely drop the page from our search results," support.google.com/webmasters/answer/93710?hl=en
    – user29671
    Commented Jun 23, 2014 at 11:43
  • From my experience when Googlebot receives a lot of 404, 410, noindex on a website, it does not takes immediately action, to prevent human mistakes or hackers actions. If you are sure that Googlebot takes immediate action, please comment back so I have my answer updated.
    – krokola
    Commented Jun 23, 2014 at 12:12
  • Never heard of google not removing noindex in case of hackers or whatnot, I have heard that 404's might be delayed in removal because they can be transient and not long lasting/errors. I do believe google remove noindex straight away because that is what I have found and it matches what Google state themselves.
    – user29671
    Commented Jun 23, 2014 at 12:17
  • "If the content is currently in our index, we will remove it after the next time we crawl it. " support.google.com/webmasters/answer/93710?hl=en
    – user29671
    Commented Jun 23, 2014 at 12:18
  • This is a really helpful answer - thanks very much. I'll start on adding the noindex tags and hope it makes a difference. Cheers! Commented Jun 23, 2014 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.