Disallowing URL parameter in robots.txt

Question

I have important URLs on my site to which I’d like to append temporary parameters. For example:

example.com/tool/input

will have the parameter

example.com/tool/input?foo

input in the URL is dynamic.

Although example.com/tool and example.com/tool/input are important URLs on the site, I do not want Google to crawl or index the URLs with the parameters.

So what I think I want to do is to add a robots meta tag to the pages with parameters, AND add the following robots.txt directive:

User-agent: *
Disallow: /*?foo

Would this be the best way to treat the URLs, or is there something I'm missing that might be better?

Answer 1

Adding a meta tag and disallowing in robots.txt if not a good approach. You should only do one or the other. If you disallow then Google won't be able to see the status when it crawls. Of the two, the meta tag is a much stronger signal than robots.txt. Google sometimes indexes pages that are disallowed. However if Google can crawl and see the tag it won't index the page.

There is a better option than either of those. You should add a canonical tag instead. The canonical tag lets Google know about the duplication and which one is your preferred URL. Using the canonical tag can take advantage of inbound links to the parameter that noindex tag will waste.

Answer 2

I would say (global)

*?*foo*

or (only /tool/input)

/tool/input?*foo*

please note: parameters are exchangeable and have no sorting!