Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content.

For example i have a page for article itemA belonging to category catA/subcatA, with URL:


this is the URL that i want to be indexed from google.

this article is also visible via tagging in various other places in the web site. The URLs produced via tagging is like:


this URL i want NOT to be indexed from google. However i want to have indexed all tag listings:


so how can i achieve this? dissalow URLs of including a specific string with a '/' at the end?

/tagA1/ itemA - dissalow

/tagA1 - allow

share|improve this question
You asked the same question on SO. – unor Nov 12 '13 at 13:37
up vote 1 down vote accepted

A different approach ,

If you are using any CMS(Wordpress,Joomla etc) every CMS have separate page for listing tags and tags/with item.

So you can simple use canonical urls or nofollow,noindex option with meta tags.

You already mention you have thousands of dynamic urls so its better to use meta tags with nofollow,noindex based on your requirement in each pages.

Hope its make sense.

share|improve this answer
Yeap i am using Joomla and i am thinking of setting index,nofollow to my tag listing pages. Since i only need not to follow and index their individual items and not the actual listing. I was not sure though about crawlers behavior regarding the 'nofollow' metatag. Does it actually prevents them from indexing the outgoing link URLs? – thanili Nov 11 '13 at 9:30
or does it make sense to produce my tag listings with:<a href="" rel="nofollow"> manner? Is is going to prevent google form indexing these duplicate URLs? – thanili Nov 11 '13 at 9:35
if you are using Joomla then you can simple manage its from article options settings.In you list page you have set like index,nofollow and tags/details page you can noindex/nofollow. those have different layout in Joomla. – Jobin Jose Nov 11 '13 at 9:35
sure that will work. but you have keep in mind on this case the direct to taga1/item1 should be noindex. – Jobin Jose Nov 11 '13 at 9:39
this is what i want to achieve. to have indexed URLs: /catA/itemA, /tagA1 and NOT to have indexed only URLs of the form tagA1/itemA – thanili Nov 11 '13 at 9:41

You should not use robots.txt to block duplicate content.

The first step is to stop linking to 'bad' URLs. Each article should have one, canonical URL. So for example the URL /tagA1/itemA should not exist. On your tag page that lists the articles, they should link to the preferred URL of /catA/subcatA/itemA.

If for some reason that is not possible, or you have links pointing to the 'bad' URLs from elsewhere, there are two possible solutions:

  1. 301 redirect the 'bad' URL to the 'good' one. This could be done via htaccess, especially if there are clear patterns for the redirects. This is the preferred solution.
  2. Use the "rel=canonical" tag. Details in Google help files
share|improve this answer
User-agent: Google
Disallow: /tagA1/
allow: /tagA1

If you use this, It will disallow all the pages comes after tagA1/ and not tagA

Learn more information about robots.txt from http://www.robotstxt.org/robotstxt.html

share|improve this answer
i know ... but this is not the case for me, since these kind of urls: /tagA1/itemA etc are thousands! i need to find a generic pattern (something like a regular expression for each tag) to use! – thanili Nov 11 '13 at 9:07
You need to disallow all the pages that comes via tagA1 and don't need to disallow tagA1. right? – Sathiya Kumar Nov 11 '13 at 9:20
yeap! how can i achieve this? – thanili Nov 11 '13 at 9:21
I have edited my answer. Please find it. – Sathiya Kumar Nov 11 '13 at 10:11
thanks for responding Sathiya! i will give it a try if i will not find a way to fix my code for canonical URLs – thanili Nov 11 '13 at 14:19

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.