7

When I enter a URL from my site into Google search, I get back that URL but with an added query string in the results. For example when I search for https://example.com/blog/blog/2013/02, the search results show it with parameters as https://example.com/blog/blog/2013/02?limit=200.

I have disallowed the parameters in the robots.txt file as Disallow: /*?. Now the Google search result shows the message as

A description for this result is not available because of this site's robots.txt – learn more.

How can I avoid having this added query string on the URL?

5
  • 1
    Is this query string used by your site/page at all?
    – MrWhite
    May 14, 2014 at 13:05
  • 2
    As well, if the parameter is not causing an actual problem, then I would do what I can to correct it, but not block it. You can rewrite the request to remove the parameter. But blocking it is a bad idea. In effect, you blocked all accesses using parameters thus blocking Google entirely. So I am not surprised by the message in the SERPs.
    – closetnoc
    May 14, 2014 at 16:17
  • 1
    No the query string is not used in my site @w3d
    – alamelu
    May 15, 2014 at 9:49
  • @closetnoc - If we rewrite the request url as without querystring, will not display above said message in google search?
    – alamelu
    May 15, 2014 at 9:55
  • I added an answer. Try this and let me know. My plans blew up and so I should be here most of the day.
    – closetnoc
    May 15, 2014 at 14:35

3 Answers 3

6

Robots.txt will only prevent bots from crawling the Disallowed URLs, not from indexing them. If the Disallowed URLs are linked to externally, or internally from a page that isn't Disallowed, they'll appear in the index with the snippet text you've quoted.

If you want to exclude them from the index entirely, the best option is probably the canonical link element:

<head> <link rel="canonical" href="http://www.example.com"> </head>

In the example you give, the page https://example.com/blog/blog/2013/02?limit=200 would contain the following:

<head> <link rel="canonical" href="https://example.com/blog/blog/2013/02"> </head>

That's assuming HTTPS is your preferred protocol. If it isn't, you should normalise that via 301 redirect.

The advantage of this approach is that you don't have to configure search engine Webmaster Tools.

Using Webmaster Tools

An alternative is to use URL Parameter Filters in Google and Bing Webmaster Tools. In Google, you'll find it under Crawl > URL Parameter Filters.

Typically, that page will already be populated with parameters the crawler has discovered, though you can specify them manually too.

Assuming ?limit=200 is controlling how many items are shown on a page, you'd configure it as follows in Google WMT:

Select "Yes: Changes, reorders or narrows page content"

Select "Narrows"

Select "No URLs"

3
  • It would seem (from comments) that ?limit=200 is not actually used by the site after all, so this probably affects the rules for URL parameters you'd want to set in GWT.
    – MrWhite
    May 17, 2014 at 10:30
  • Also, don't combine a rel=canonical with the robots.txt - Google won't see the rel=canonical in those cases. May 21, 2014 at 8:44
  • @JohnMueller Good point. Edited.
    – GDVS
    May 21, 2014 at 8:59
3

Okay. First get rid of the Disallow: /*? in the robots.txt file. This is causing the message from Google. Google is saying that it has no access to your site at all.

In your .htaccess file, try this:

RewriteCond %{REQUEST_URI} ^(*.)\?limit=\d+$ [NC]
RewriteRule .* https://example.com/%1 [R=301,L]

I have not tested this, but I am sure the regex (regular expression) is correct. I at least tested that under a different scenario here. Try this and make several requests to your site using a variety of ?limit=200 added to the end of the request and see if there is a redirect to a URL without it.

I still say that the parameter should have no real effect and should cause no harm. It should be okay to just leave it.

3
  • I think $1 should be %1 in the RewriteRule substitution, to refer to the RewriteCond parenthesised subpattern (as opposed to the RewriteRule pattern). (+1)
    – MrWhite
    May 17, 2014 at 10:27
  • @w3d I will check. I do weird things with .htaccess so that when I try and do something within the normal range, I have to readjust my thinking. I do regexes often in code, but as far as .htaccess, they seem to fit certain patterns so I really do not get to exercise the .htaccess side of my brain. Now, would that be the right or left? Or is it top or bottom?
    – closetnoc
    May 17, 2014 at 20:44
  • 1
    @w3d Got it! You are right- of course I did not doubt you- I just wanted to do more research to help clarify my mind. Believe me- that is a real task! Thanks for the help!
    – closetnoc
    May 18, 2014 at 2:12
0

Not sure where the query param is coming from, but there is a way to strip it off in Google Analytics. See https://support.google.com/analytics/answer/1010249?hl=en, topic 'Exclude URL Query Parameter'

1
  • 1
    Doesn't this relate to GA reports only? There are similar options relating to query string parameters in Google Webmaster Tools which relates specifically to Google Search.
    – MrWhite
    May 14, 2014 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.