Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a URL like this:


I had a problem and many duplicated pages have been created like this:


And so on, those pages have the same content as the main one, so what I did to fix it was redirecting all the pages with more than two times php keyword in the URL to the main URL.

But I have did it late, so Google has to redirect maybe more than 20.000 pages that have been already crawled.

So I want to setup a Disallow in robots.txt to block it for spending resources on those urls.

So my question is, what pattern should I use to disallow pages with more than two times the keyword php in the URL?

Will, Disallow: /*php*php* work as expected? I am asking this because I don't want to accidentally block good URLs.

share|improve this question
Why don't you just use either 1) 301 redirect to a correct page (using mod_rewrite and .htaccess or in your php code) or 2) add canonical URL entry in <head> section of your HTML -- googlewebmastercentral.blogspot.co.uk/2009/02/… ? – LazyOne Jun 14 '12 at 10:05
Thanks! I have finally ended up adding a canonical to the good url. – Alexandru Trandafir Catalin2 Jun 19 '12 at 9:57
up vote 3 down vote accepted

Googlebot does support wildcards in robots.txt. They announced this in their blog. http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html

Other browsers don't actually support wildcards, so that syntax is not universal.

However, putting urls into robots.txt does not prevent googlebot from indexing them. Your solution of the canonical tag sounds like a much better idea to get them out of the index. 301 redirects would also work.

share|improve this answer

Simply you can use:

Disallow: /freelance-jobs-new-york-php-php*/

see this google page


share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.