0

When adding parameters to Webmaster Tools for URLs that I don't want indexed, should I add the parameters with actual brackets, example:

technology[above_average]

or should add in the URL encoding characters which are showing in the SERPs URL like so:

technology%5Babove_average%5D

Thoughts?

1

Googlebot expects URL parameter names to be escaped when adding them via Google Webmaster Tools.

I ran an experiment on my site where I created a page with two different URL parameters: unesc% and escap%. I created page for the experiment on my site. On it uses the parameters with four values each

I then added escap%25 and unesc% to Google Webmaster Tools as "Effect: None, Crawl: Representative URL" and sat back to watch what would happen.

The first thing I noticed was that Googlebot actually came to my site and crawled all the parameters, despite the webmaster tools setting. This is either because:

  • There is a bug in Googlebot where it doesn't respect the settings
  • They designed Gogolebot not to respect the settings
  • It takes some time for Googlebot to become aware of the settings. (I put the settings in before Googlebot had a chance to crawl, but it started crawling within an hour).

After Googlebot crawled, webmaster tools didn't report that Googlebot was monitoring any matching URLs. I figured that it would take some time before GWT started showing what Googlebot was actually doing so I waited. A couple weeks later I now get this:

enter image description here

I would still suggest adding both to your site just as a precaution (I'm only linking to the escaped version of the URL, I'm not user if it would be different if I linked to the un-escaped version).


Here is the testing that I had done earlier:

I tested it by adding both, just to see how the tool would react.

Google webmaster tools, url parameters with encoding

I thought that it might give me an error for one of the two, or possibly canonicalize them to the same thing. I could actually add both of them (as you can see from the screenshot).

That leads me to guess that the un-escaped one is the one that would match while Googlebot is crawling. But, that is just a guess. Since it allows you to add both, you might as well just go ahead and add both to be safe.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.