Hiding duplicate content - should I use meta tags or robots.txt?

Question

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself.

I've not engaged in any shady link-building (or indeed any link-building at all!) and the site is technically fine (fast pages, no malware, fully responsive). So my first guess is that I'm being penalized for duplicate content. Although there's a huge amount of rich content on the site, there's also a lot of algorithmically-generated pages - for example, one for each town in the UK. (This isn't done for SEO, but I guess maybe Google thinks it is.)

So I need to stop Googlebot finding, and objecting to, this content.

I would rather do it via a meta tag (<meta name="googlebot" content="noindex" />) on the relevant pages than with using robots.txt.

So my question are:

• Is a meta tag a workable alternative to using robots.txt for hiding "problematic" content?
• Is there anything else I should be doing?

Answer 1

If you think that you have been penalized for duplicate or low quality content do NOT block indexing from robots.txt. Googlebot has already indexed these pages and it must have access to them so it can remove them.

What to do

1. Add a 301 redirection to all duplicate pages, so you have only 1 page for each topic
2. Add the noindex meta tag in all pages that you think that are low quality auto generated, like the tag pages, search results, and any other feed pages, so that Googlebot will remove these pages from the index and pass the juice to the internal links.

Hint

It is perfectly fine to have auto generated content/feed pages, but it depends on the amount. If your website has 100 unique pages and 10000 tag pages this is a disaster. If you have 10000 unique pages and 50 tag pages this is excellent.