4

I found a similar question with a great answer for removing a single URL parameter. However, what if I have 20+ URL parameters that I don't want indexed?

Also, in the example solution below, it is assuming that you want to specify a parameter range (ex: ?id=0 to ?id=9)... In my situation I would simply want to not index anything with the ?id parameter at all, regardless of what follows in the url string. Let's also say that I would not want to index the ?start and ?Page parameters either... Can someone help me out with a revised version of the following code?

For NOINDEX:

<IfModule mod_rewrite.c>
RewriteCond %{QUERY_STRING} ^id=([0-9]*)$
RewriteRule .* - [E=NOINDEX_HEADER:1] 
</IfModule>

<IfModule mod_headers.c>
Header set X-Robots-Tag "noindex, follow" env=NOINDEX_HEADER
</IfModule>

For CANONICAL:

<IfModule mod_rewrite.c>
RewriteCond %{QUERY_STRING} ^id=([0-9]*)$
RewriteRule .* - [E=CANONICAL_HEADER:1]
</IfModule>

<IfModule mod_headers.c>
Header set Link '%{HTTP_HOST}%{REQUEST_URI}e; rel="canonical"' env=CANONICAL_HEADER
</IfModule>

Thank you @Evgeniy and @JohnMueller for the above code.
Reference: How to correctly remove parameters from the Google index?

Improve this question
0

2 Answers 2

Reset to default
4

You only really need to set the rel="canonical" header. This should be sufficient in ensuring only the canonical URL (ie. the one with no URL params) appears in the SERPs. Setting a noindex robots meta tag for such URLs would seem to be overkill (and a tad risky) IMO.

Presumably you are unable to set a rel="canonical" meta tag in the HTML itself?

...what if I have 20+ URL parameters that I don't want indexed?

Is it safe to say that you don't want any URL with any URL params (ie. any query string) indexed? In which case you can simply change your RewriteCond directive to read:

RewriteCond %{QUERY_STRING} .

That is, there is a query string of any length.

If, however, you want to exclude 20 specific URL params then you are going to have to name every one of them. For example:

RewriteCond %{QUERY_STRING} (?:^|&)(id|start|page|another)=

The (?:^|&) is a non-capturing group to ensure we only match these specific param names and not something like sid or lastpage, etc. (if they could possibly be URL param names).

Header set Link '%{HTTP_HOST}%{REQUEST_URI}e; rel="canonical"' env=CANONICAL_HEADER

This is invalid. You are missing the scheme (eg. http), e symbol after the %{HTTP_HOST} variable (this would result in a 500 error) and angled brackets (<..>) around the URL. This should be of the form:

Header set Link '<http://%{HTTP_HOST}e%{REQUEST_URI}e>; rel="canonical"'

Reference: (Google's support doc regarding canonical URLs)
https://support.google.com/webmasters/answer/139066?hl=en

UPDATE: However, the %{REQUEST_URI}e environment variable, when used in this context, includes the query string - which really defeats the object of this excercise. This whole block should be rewritten as:

RewriteCond %{QUERY_STRING} .
RewriteRule (.*) - [E=CANONICAL_URI:$1]
Header set Link '<http://%{HTTP_HOST}e/%{CANONICAL_URI}e>; rel="canonical"' env=CANONICAL_URI

Instead of using the REQUEST_URI variable, we capture the URL-path only (which excludes the query string) using the RewriteRule directive and store this in the CANONICAL_URI variable. This is then used in the Header directive instead.

There is also no need for the <IfModule> containers here. It either works or it breaks, these directives are not intended to be optional (are they?).

Improve this answer
6
  • Updated answer with respect to your Header set Link directive.
    – MrWhite
    Sep 6, 2016 at 23:49
  • Ok, excellent information. And yes, you are correct that I really do not want to index URL's with ANY parameters. So, just to be sure, this code alone, added to my htaccess, should add a "noindex" to URL's with any parameters? code RewriteCond %{QUERY_STRING} . RewriteRule .* - [E=NOINDEX_HEADER:1]'
    – Bryan Earl
    Sep 6, 2016 at 23:58
  • Yes, when used in combination with your first code block (ie. Header set X-Robots-Tag ...) it will add a noindex X-Robots-Tag HTTP response header on URLs that contain any parameters (you should check the network traffic to confirm this, don't just take my word for it, as there could be something else that overrides this). However, as mentioned above, the rel="canonical" response header by itself would ordinarily be preferable IMO.
    – MrWhite
    Sep 7, 2016 at 0:54
  • Apologies, but can you elaborate on the canonical header a bit more? You say "response header by itself" would ordinarily be preferred... Does that mean that adding the Header set Link '<http://%{HTTP_HOST}e%{REQUEST_URI}e>; rel="canonical"' code (and not the rewrite) would typically take care of it? What exactly is that code doing? For instance, if google came across this URL: [domain.com/classifieds/archery/all,0?start=8] would it automatically tell it that the canonical is [domain.com/classifieds/archery]? Essentially removing the final slash and everything past it?
    – Bryan Earl
    Sep 7, 2016 at 1:02
  • "(and not the rewrite)" - You would need the mod_rewrite directives that immediately precede it, those set the CANONICAL_HEADER environment variable that the following Header directive then checks. The response header is not set unless the environment variable is also set. Actually, there is an error with that Header directive, since the REQUEST_URI variable in this context contains the query string! I'll update my answer with a working example.
    – MrWhite
    Sep 7, 2016 at 1:47
1

In addition to redirects and meta tags, it is possible to prevent Google from indexing specific URL parameters by configuring them in Google Search Console: https://support.google.com/webmasters/answer/6080550?hl=en

You can set parameters to "Doesn't effect page content." This is for tracking parameters that don't actually change the page. This setting causes allows Googlebot to crawl them but then index the version without them.

You can also set parameters to "Changes page content" and then "Crawl no URLs". This causes Googlebot to not crawl those URLs with the parameters on them at all. They would then mostly fall out of the index. You could use this for pagination and sorting parameters that cause duplicate content compared to the page without any parameters.

Improve this answer
2
  • I have set all my URL parameters in the search console, but for some reason Google keeps indexing a lot of URL's with parameters.
    – Bryan Earl
    Sep 7, 2016 at 0:05
  • Just like adding something to robots.txt, it can take Google a while for the "No crawling" rules. It can take weeks for the majority to drop out of the index, then Google may still index any that have external links. Also, if you have instructed Google not to crawl them, then Google will never see any redirects or canonical tags you put in place. If you do implement those solutions, be sure to remove the no crawl rules you have set in search console.
    – Stephen Ostermiller ♦
    Sep 7, 2016 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.