Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

It's my understanding that URLs are of the format example.com/something/somethingelse.

It's my understanding that parameters follow the URL with a question mark example.com?l=fr_FR

My CMS website has language translations that use parameters. The example above is for the French language version of my site.

I would like to block all non english translations from Google Index using robots.txt.

In the blocked URLs tool in GWT I tried to test this:

# robots.txt generated at http://www.mcanerin.com
User-agent: *
Disallow: 
Disallow: /cgi-bin/
Disallow: ?l=

Against the following URL, which appeared as one of the duplicate page titles in HTML improvements.

https://example.com/reports/view/884?l=eu

This is my first time fiddling with this tool in GWT so I'm not sure if I'm using it right.

The test results for Googlebot says

Allowed by line 3: Disallow:

I wanted the code to prevent Google indexing any URLs that contain the following string

?l=

Here are some examples of URLs I'd like to block from the index. These URLs generate duplicate titles according to GWT. /reports/view/884?l=km /reports/view/884?l=ne_NP /reports/view/884?l=te /index.php/page/index/12?l=fr_FR&l=hy_AM /index.php/page/index/12?l=ht_HT&l=bn_BD /index.php/page/index/12?l=hu_HU&l=hy_AM

Can I tell the robots to exclude URLs with tags that contain

?l=

share|improve this question

1 Answer 1

up vote 4 down vote accepted

You can block URLs that contain ?l= from being indexed by search engine robots by using the following robots.txt directive:

Disallow: /*?l=

The / indicates the root directory, and * is a wildcard for anything up to ?l=, followed by anything after it.

share|improve this answer
1  
Dan, you're the best. Thank you very much again, the test in GWT showed that this did work. –  Doug Firr Sep 28 '13 at 22:20
    
No problem at all :-) I added a little bit of info just now to explain what it means. –  dan Sep 28 '13 at 22:20
1  
Plus 1 for a great answer. –  user2434 Oct 3 '13 at 10:14
    
Thanks Simon :-) –  dan Oct 3 '13 at 14:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.