I have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content.

For example i have a page for article itemA belonging to category catA/subcatA, with URL:


this is the URL that i want to be indexed from google.

this article is also visible via tagging in various other places in the web site. The URLs produced via tagging is like:


this URL i want NOT to be indexed from google. However i want to have indexed all tag listings:


so how can i achieve this? dissalow URLs of including a specific string with a '/' at the end?

/tagA1/ itemA - dissalow

/tagA1 - allow


I would take advantage of Google's regular expression support. You want to allow /tagA1 and /tagA1/, but not /tagA11/XXX. So:

User-agent: *
Allow: /tagA1$
Allow: /tagA1/$
Disallow: /tagA1

The $ denotes end of string. So /tagA1 will be allowed, but tagA1/foo will be blocked.

Note that Google and Bing support the $. Others might not. Note also that the order of Allow and Disallow here is important. Many crawlers read the robots.txt sequentially and apply the "first matching rule". If you put the Disallow first, those crawlers would never index the pages you allow.


Hi you can use 2 methods to do this

Method one , use this meta tag "<meta name="robots" content="noindex, nofollow" />" and place it in the header area.

Method two is ,

User-agent: *
Disallow: /tagA1/itemA
  • URLs are produced dynamically via tagging so the actual HTML page for itemA is the same for all URLs. So where am i going to put the noindex,nofollow meta tag? For sure not in itemA ... Also i want the URL: /tagA1 to be indexed so i can not put a noindex meta tag there also. I am not sure about how nofollow metatag works since i think it is for not counting outbound link from a specific page.
    – thanili
    Nov 11 '13 at 9:15
  • also method 2 does not work for me since there are thousands of URLs i should block this way. I need something more generic. Maybee a regular expression?
    – thanili
    Nov 11 '13 at 9:18
  • take a look here: seochat.com/c/a/search-engine-optimization-help/…
    – thanili
    Nov 11 '13 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy