Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled?

User-agent: *
Disallow: /blog/tag
  • No dumb robots, use a canonical tag. It establishes where CMS multiaccess mechanisms and the URLs they create point to as the original so Google knows it isn't duplication. They already though of this and provide the mechanism. Don't use an axe when you need a finely honed chisel. The sculpture will come out better looking. Apr 21, 2013 at 15:49

2 Answers 2


2009 - Google, Bing, and Yahoo announce Support for Canonical Links

There is no need to use robots.txt to tackle the duplicate content problem, in February 2009, Google, Yahoo and Microsoft (for their Bing search engine) announced support for the canonical link element which aims to tackle the problem of duplicate content.

Rel="canonical" To The Rescue

So rather than blocking Google and other search engines indexing the pages into results you should opt to use Canonicalization by using <link rel="canonical" href="http://www.yourdomain.com/path-here" /> This will tell Google and the other search engines which are the master pages and those that are duplicate and meaning that you will receive no slaps from Google.

SEO Plugins Make Canonicalization Easy

There are many plugins for various content management systems that automate this process for you, for example and assuming that you are using WordPress you can install Yoast SEO which will ensure that all pages use canonical and pages such as Tags will never harm your actual pages. I've used many CMS engines so if you using Drupal, Joomla, Blogengine or any other let me know and I'll recommend some alternatives.


I recommend that you read more on canonical to give you a clearer indication what it can do for you:

  • Don't block content, use the Canonical Tag to point multiple access pages to the original page. All CMS systems allow for multiple ways of finding the original page. Google, et.al. realized this and gave you a tool to tell them which page you consider to be cannon for the original URL to access it. Google then looks at that tag, no matter by what URL it finds it and does not then consider the multiple ways of contacting that page to be duplicates. Apr 21, 2013 at 15:47

I usually just add a noindex meta tag to tag pages. But your method in robots.txt should work as well, the only downside being that you'll need to manually add each tag, unless they're in a sub-folder of their own ie /blog/tags/tag in which case you could simply add Disallow: /blog/tags/*.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.