Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled?

User-agent: *
Disallow: /blog/tag
share|improve this question
    
No dumb robots, use a canonical tag. It establishes where CMS multiaccess mechanisms and the URLs they create point to as the original so Google knows it isn't duplication. They already though of this and provide the mechanism. Don't use an axe when you need a finely honed chisel. The sculpture will come out better looking. –  Fiasco Labs Apr 21 '13 at 15:49
add comment

2 Answers 2

2009 - Google, Bing, and Yahoo announce Support for Canonical Links

There is no need to use robots.txt to tackle the duplicate content problem, in February 2009, Google, Yahoo and Microsoft (for their Bing search engine) announced support for the canonical link element which aims to tackle the problem of duplicate content.

Rel="canonical" To The Rescue

So rather than blocking Google and other search engines indexing the pages into results you should opt to use Canonicalization by using <link rel="canonical" href="http://www.yourdomain.com/path-here" /> This will tell Google and the other search engines which are the master pages and those that are duplicate and meaning that you will receive no slaps from Google.

SEO Plugins Make Canonicalization Easy

There are many plugins for various content management systems that automate this process for you, for example and assuming that you are using WordPress you can install Yoast SEO which will ensure that all pages use canonical and pages such as Tags will never harm your actual pages. I've used many CMS engines so if you using Drupal, Joomla, Blogengine or any other let me know and I'll recommend some alternatives.

References:

I recommend that you read more on canonical to give you a clearer indication what it can do for you:

share|improve this answer
    
Don't block content, use the Canonical Tag to point multiple access pages to the original page. All CMS systems allow for multiple ways of finding the original page. Google, et.al. realized this and gave you a tool to tell them which page you consider to be cannon for the original URL to access it. Google then looks at that tag, no matter by what URL it finds it and does not then consider the multiple ways of contacting that page to be duplicates. –  Fiasco Labs Apr 21 '13 at 15:47
add comment

I usually just add a noindex meta tag to tag pages. But your method in robots.txt should work as well, the only downside being that you'll need to manually add each tag, unless they're in a sub-folder of their own ie /blog/tags/tag in which case you could simply add Disallow: /blog/tags/*.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.