Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

When I enter a URL from my site into Google search, I get back that URL but with an added query string in the results. For example when I search for https://example.com/blog/blog/2013/02, the search results show it with parameters as https://example.com/blog/blog/2013/02?limit=200.

I have disallowed the parameters in the robots.txt file as Disallow: /*?. Now the Google search result shows the message as

A description for this result is not available because of this site's robots.txt – learn more.

How can I avoid having this added query string on the URL?

share|improve this question
1  
Is this query string used by your site/page at all? –  w3d May 14 at 13:05
2  
As well, if the parameter is not causing an actual problem, then I would do what I can to correct it, but not block it. You can rewrite the request to remove the parameter. But blocking it is a bad idea. In effect, you blocked all accesses using parameters thus blocking Google entirely. So I am not surprised by the message in the SERPs. –  closetnoc May 14 at 16:17
1  
No the query string is not used in my site @w3d –  alamelu May 15 at 9:49
    
@closetnoc - If we rewrite the request url as without querystring, will not display above said message in google search? –  alamelu May 15 at 9:55
    
I added an answer. Try this and let me know. My plans blew up and so I should be here most of the day. –  closetnoc May 15 at 14:35

3 Answers 3

Robots.txt will only prevent bots from crawling the Disallowed URLs, not from indexing them. If the Disallowed URLs are linked to externally, or internally from a page that isn't Disallowed, they'll appear in the index with the snippet text you've quoted.

If you want to exclude them from the index entirely, the best option is probably the canonical link element:

<head> <link rel="canonical" href="http://www.example.com"> </head>

In the example you give, the page https://example.com/blog/blog/2013/02?limit=200 would contain the following:

<head> <link rel="canonical" href="https://example.com/blog/blog/2013/02"> </head>

That's assuming HTTPS is your preferred protocol. If it isn't, you should normalise that via 301 redirect.

The advantage of this approach is that you don't have to configure search engine Webmaster Tools.

Using Webmaster Tools

An alternative is to use URL Parameter Filters in Google and Bing Webmaster Tools. In Google, you'll find it under Crawl > URL Parameter Filters.

Typically, that page will already be populated with parameters the crawler has discovered, though you can specify them manually too.

Assuming ?limit=200 is controlling how many items are shown on a page, you'd configure it as follows in Google WMT:

Select "Yes: Changes, reorders or narrows page content"

Select "Narrows"

Select "No URLs"

share|improve this answer
    
It would seem (from comments) that ?limit=200 is not actually used by the site after all, so this probably affects the rules for URL parameters you'd want to set in GWT. –  w3d May 17 at 10:30
    
Also, don't combine a rel=canonical with the robots.txt - Google won't see the rel=canonical in those cases. –  John Mueller May 21 at 8:44
    
@JohnMueller Good point. Edited. –  GDav May 21 at 8:59

Okay. First get rid of the Disallow: /*? in the robots.txt file. This is causing the message from Google. Google is saying that it has no access to your site at all.

In your .htaccess file, try this:

RewriteCond %{REQUEST_URI} ^(*.)\?limit=\d+$ [NC]
RewriteRule .* https://example.com/%1 [R=301,L]

I have not tested this, but I am sure the regex (regular expression) is correct. I at least tested that under a different scenario here. Try this and make several requests to your site using a variety of ?limit=200 added to the end of the request and see if there is a redirect to a URL without it.

I still say that the parameter should have no real effect and should cause no harm. It should be okay to just leave it.

share|improve this answer
    
I think $1 should be %1 in the RewriteRule substitution, to refer to the RewriteCond parenthesised subpattern (as opposed to the RewriteRule pattern). (+1) –  w3d May 17 at 10:27
    
@w3d I will check. I do weird things with .htaccess so that when I try and do something within the normal range, I have to readjust my thinking. I do regexes often in code, but as far as .htaccess, they seem to fit certain patterns so I really do not get to exercise the .htaccess side of my brain. Now, would that be the right or left? Or is it top or bottom? –  closetnoc May 17 at 20:44
1  
@w3d Got it! You are right- of course I did not doubt you- I just wanted to do more research to help clarify my mind. Believe me- that is a real task! Thanks for the help! –  closetnoc May 18 at 2:12

Not sure where the query param is coming from, but there is a way to strip it off in Google Analytics. See https://support.google.com/analytics/answer/1010249?hl=en, topic 'Exclude URL Query Parameter'

share|improve this answer
1  
Doesn't this relate to GA reports only? There are similar options relating to query string parameters in Google Webmaster Tools which relates specifically to Google Search. –  w3d May 14 at 21:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.