2

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself.

I've not engaged in any shady link-building (or indeed any link-building at all!) and the site is technically fine (fast pages, no malware, fully responsive). So my first guess is that I'm being penalized for duplicate content. Although there's a huge amount of rich content on the site, there's also a lot of algorithmically-generated pages - for example, one for each town in the UK. (This isn't done for SEO, but I guess maybe Google thinks it is.)

So I need to stop Googlebot finding, and objecting to, this content.

I would rather do it via a meta tag (<meta name="googlebot" content="noindex" />) on the relevant pages than with using robots.txt.

So my question are:

  • Is a meta tag a workable alternative to using robots.txt for hiding "problematic" content?
  • Is there anything else I should be doing?
  • 1
    It's very unlike that Google has punished you for content that is accessibly by various ways, for example the mega popular WordPress by default doesn't use canonical links and as you can expect snippets of blog entries across lots of pages when using tags and categories. Also you shouldn't need to deindex pages, simply use canonical links on the pages with the content. – Simon Hayter Jun 23 '14 at 10:01
  • What is a non SEO reason for generating duplicate content for each town in the UK? – JamesRyan Jun 23 '14 at 11:08
  • I'd noindex the auto-generated city pages until you have some real valuable content on. – user29671 Jun 23 '14 at 11:23
  • @JamesRyan: two reasons really - one is that it's a placeholder for future content, whether by me (e.g. when I write news stories that relate to that town) or user-generated (and there's plenty of pointers towards that). The other is that there is some content, but it's map-based rather than textual - which means it's JavaScript and pretty much unindexable. – Richard Fairhurst Jun 23 '14 at 16:57
3

If you think that you have been penalized for duplicate or low quality content do NOT block indexing from robots.txt. Googlebot has already indexed these pages and it must have access to them so it can remove them.

What to do

  1. Add a 301 redirection to all duplicate pages, so you have only 1 page for each topic
  2. Add the noindex meta tag in all pages that you think that are low quality auto generated, like the tag pages, search results, and any other feed pages, so that Googlebot will remove these pages from the index and pass the juice to the internal links.

Hint

It is perfectly fine to have auto generated content/feed pages, but it depends on the amount. If your website has 100 unique pages and 10000 tag pages this is a disaster. If you have 10000 unique pages and 50 tag pages this is excellent.

  • Not sure Google has ever specified needing to crawl 2-3 times to act on noindex tag. "When we see the noindex meta tag on a page, Google will completely drop the page from our search results," support.google.com/webmasters/answer/93710?hl=en – user29671 Jun 23 '14 at 11:43
  • From my experience when Googlebot receives a lot of 404, 410, noindex on a website, it does not takes immediately action, to prevent human mistakes or hackers actions. If you are sure that Googlebot takes immediate action, please comment back so I have my answer updated. – krokola Jun 23 '14 at 12:12
  • Never heard of google not removing noindex in case of hackers or whatnot, I have heard that 404's might be delayed in removal because they can be transient and not long lasting/errors. I do believe google remove noindex straight away because that is what I have found and it matches what Google state themselves. – user29671 Jun 23 '14 at 12:17
  • "If the content is currently in our index, we will remove it after the next time we crawl it. " support.google.com/webmasters/answer/93710?hl=en – user29671 Jun 23 '14 at 12:18
  • This is a really helpful answer - thanks very much. I'll start on adding the noindex tags and hope it makes a difference. Cheers! – Richard Fairhurst Jun 23 '14 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.