i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content.

For example i have a page for article itemA belonging to category catA/subcatA, with URL:


this is the URL that i want to be indexed from google.

this article is also visible via tagging in various other places in the web site. The URLs produced via tagging is like:


this URL i want NOT to be indexed from google. However i want to have indexed all tag listings:


so how can i achieve this? dissalow URLs of including a specific string with a '/' at the end?

/tagA1/ itemA - dissalow

/tagA1 - allow


You should not use robots.txt to block duplicate content.

The first step is to stop linking to 'bad' URLs. Each article should have one, canonical URL. So for example the URL /tagA1/itemA should not exist. On your tag page that lists the articles, they should link to the preferred URL of /catA/subcatA/itemA.

If for some reason that is not possible, or you have links pointing to the 'bad' URLs from elsewhere, there are two possible solutions:

  1. 301 redirect the 'bad' URL to the 'good' one. This could be done via htaccess, especially if there are clear patterns for the redirects. This is the preferred solution.
  2. Use the "rel=canonical" tag. Details in Google help files

A different approach ,

If you are using any CMS(Wordpress,Joomla etc) every CMS have separate page for listing tags and tags/with item.

So you can simple use canonical urls or nofollow,noindex option with meta tags.

You already mention you have thousands of dynamic urls so its better to use meta tags with nofollow,noindex based on your requirement in each pages.

Hope its make sense.

  • Yeap i am using Joomla and i am thinking of setting index,nofollow to my tag listing pages. Since i only need not to follow and index their individual items and not the actual listing. I was not sure though about crawlers behavior regarding the 'nofollow' metatag. Does it actually prevents them from indexing the outgoing link URLs?
    – thanili
    Nov 11 '13 at 9:30
  • or does it make sense to produce my tag listings with:<a href="" rel="nofollow"> manner? Is is going to prevent google form indexing these duplicate URLs?
    – thanili
    Nov 11 '13 at 9:35
  • if you are using Joomla then you can simple manage its from article options settings.In you list page you have set like index,nofollow and tags/details page you can noindex/nofollow. those have different layout in Joomla.
    – Jobin
    Nov 11 '13 at 9:35
  • sure that will work. but you have keep in mind on this case the direct to taga1/item1 should be noindex.
    – Jobin
    Nov 11 '13 at 9:39
  • this is what i want to achieve. to have indexed URLs: /catA/itemA, /tagA1 and NOT to have indexed only URLs of the form tagA1/itemA
    – thanili
    Nov 11 '13 at 9:41
User-agent: Google
Disallow: /tagA1/
allow: /tagA1

If you use this, It will disallow all the pages comes after tagA1/ and not tagA

Learn more information about robots.txt from http://www.robotstxt.org/robotstxt.html

  • i know ... but this is not the case for me, since these kind of urls: /tagA1/itemA etc are thousands! i need to find a generic pattern (something like a regular expression for each tag) to use!
    – thanili
    Nov 11 '13 at 9:07
  • You need to disallow all the pages that comes via tagA1 and don't need to disallow tagA1. right? Nov 11 '13 at 9:20
  • yeap! how can i achieve this?
    – thanili
    Nov 11 '13 at 9:21
  • I have edited my answer. Please find it. Nov 11 '13 at 10:11
  • thanks for responding Sathiya! i will give it a try if i will not find a way to fix my code for canonical URLs
    – thanili
    Nov 11 '13 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.