6

Within the GWT console under URL Parameters there are two settings that I'm not clear on. What is the difference between setting a URL parameter to:

  1. No: Doesn't affect page content

and

  1. Yes:Changes, reorders... and then picking No Urls under which URLs google bot should crawl?

I am trying to remove all URLs with certain parameters from the index and not sure which setting to choose. I've already submitted a sitemap with the new URLs and I will set up a 301 redirect to the new URLs also, but I think some of our old parameters are causing issues with crawling efficiency and duplicate content so I want to try and clean that up.

2

to remove all URLs with certain parameters from the index and not sure which setting to choose.

Search console doesn't allow to remove parametrized URLs from index. It gives only some possibilities to degrade their importance on crawling.

Setting urls with certain parameters to noindex could be done by rewrite rule and setting x-robots-tag, like:

<IfModule mod_rewrite.c>
RewriteCond %{QUERY_STRING} ^id=([0-9]*)$
RewriteRule .* - [E=NOINDEX_HEADER:1] 
</IfModule>

<IfModule mod_headers.c>
Header set X-Robots-Tag "noindex, follow" env=NOINDEX_HEADER
</IfModule>

In this example you set to noindex all urls with parameters like ?id=0 to ?id=n, where n is integer.

Update: According to John Mueller's advice, there is an example rule to set canonical to certain parametrized urls:

<IfModule mod_rewrite.c>
RewriteCond %{QUERY_STRING} ^id=([0-9]*)$
RewriteRule .* - [E=CANONICAL_HEADER:1]
</IfModule>

<IfModule mod_headers.c>
Header set Link '%{HTTP_HOST}%{REQUEST_URI}e; rel="canonical"' env=CANONICAL_HEADER
</IfModule>
  • If you just want them dropped from the index, the noindex like that is the right solution. If these are duplicates, you probably just want to put a rel=canonical on them and let it sort itself out like that. You don't need a noindex in that case. – John Mueller Nov 10 '15 at 20:48
  • 1
    @JohnMueller Thank you for meaningful advice - i've updated the post – Evgeniy Nov 11 '15 at 8:00
  • We are using Tomcat so we don't have a .htaccess file. In our case the parameters are causing some major issues with the index. I ran a script to count the actual number of urls that should be indexed for our site and it's around 25k ulrs. However the console says we have around 125k urls indexed! Some of the parameters say over 200k 'urls monitored'. What I've decided to do is 404 all the old urls w/ parameters and submit a sitemap with the new clean urls. I know that 404 doesn't transfer page rank etc but it was going to be really complicated to try and figure out where to 301 to. – Joe M Nov 13 '15 at 18:59
  • The Header set Link ... directive as given above is syntactically invalid and there are other issues with the REQUEST_URI variable in this context. This answer appears to have been referenced in another question in which I've provided an explanation and working example in the second part of my answer to that question. – MrWhite Sep 7 '16 at 13:49
  • Hello. Recent facing same problem. I use Filtering System in WP website. Sudden my website bandwidth got extremely high. Did lot of work(change server, cloudflare). After more analysis i found that in webmaster tools URL parameters where crawled over 24 Lac for each Filter parameter. That is why index over 2,500000. So edit the parameter by this (Yes:Changes, reorders.../sorts, No urls than Save). But the index status not change yet and i want to delete and no index the parameters. So if i use the code given up in .htaccess file will it be okay to solve the problem. Thank You – Mourin Nov 26 '17 at 11:00
0
  1. No: Doesn't affect page content

Googlebot is likely to crawl 1 URL that contains this parameter. From the help message that displays (highlighting my own):

Select this option if this parameter can be set to any value without changing the page content. For example, select this option if the parameter is a session ID. If many URLs differ only in this parameter, Googlebot will crawl one representative URL.


  1. Yes:Changes, reorders... and then picking No Urls under which URLs google bot should crawl?

Google won't crawl any URLs that contain this parameter. From the tooltip:

No URLs: Googlebot won't crawl any URLs containing this parameter. This is useful if your site uses many parameters to filter content.


I am trying to remove all URLs with certain parameters from the index and not sure which setting to choose. I've already submitted a sitemap with the new URLs and I will set up a 301 redirect to the new URLs also

The above URL parameter options block crawling. However, if you have an alternative URL to redirect to (as you suggest) then you are probably better off implementing the 301 redirect instead - this will result in the URLs being "updated" in the SERPs. If you block crawling then the URLs will probably simply drop out of the index over time.

However, it depends on the URL parameter and how they have been indexed. If you simply have a lot of duplicate content as a result of this URL parameter being indexed and these pages aren't being linked to then blocking crawling might be the better option.

  • Back in July and August I set a number of the parameters to 'No: Doesn't affect page content' but the number of URLs Monitored is still very large. Would you expect this number to drop or do I have to 301/404 those urls to get the number of urls monitored to decrease? – Joe M Nov 9 '15 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.