Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a URL like this:

www.example.com/freelance-jobs-new-york

I had a problem and many duplicated pages have been created like this:

www.example.com/freelance-jobs-new-york-php-php
www.example.com/freelance-jobs-new-york-php-php-php
www.example.com/freelance-jobs-new-york-php-php-php-php

And so on, those pages have the same content as the main one, so what I did to fix it was redirecting all the pages with more than two times php keyword in the URL to the main URL.

But I have did it late, so Google has to redirect maybe more than 20.000 pages that have been already crawled.

So I want to setup a Disallow in robots.txt to block it for spending resources on those urls.

So my question is, what pattern should I use to disallow pages with more than two times the keyword php in the URL?

Will, Disallow: /*php*php* work as expected? I am asking this because I don't want to accidentally block good URLs.

share|improve this question
4  
Why don't you just use either 1) 301 redirect to a correct page (using mod_rewrite and .htaccess or in your php code) or 2) add canonical URL entry in <head> section of your HTML -- googlewebmastercentral.blogspot.co.uk/2009/02/… ? –  LazyOne Jun 14 '12 at 10:05
    
Thanks! I have finally ended up adding a canonical to the good url. –  Alexandru Trandafir Catalin2 Jun 19 '12 at 9:57
add comment

1 Answer 1

up vote 1 down vote accepted

Googlebot does support wildcards in robots.txt. They announced this in their blog. http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html

Other browsers don't actually support wildcards, so that syntax is not universal.

However, putting urls into robots.txt does not prevent googlebot from indexing them. Your solution of the canonical tag sounds like a much better idea to get them out of the index. 301 redirects would also work.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.