0

I have important URLs on my site to which I’d like to append temporary parameters. For example:

example.com/tool/input

will have the parameter

example.com/tool/input?foo

input in the URL is dynamic.

Although example.com/tool and example.com/tool/input are important URLs on the site, I do not want Google to crawl or index the URLs with the parameters.

So what I think I want to do is to add a robots meta tag to the pages with parameters, AND add the following robots.txt directive:

User-agent: *
Disallow: /*?foo

Would this be the best way to treat the URLs, or is there something I'm missing that might be better?

  • Note that I fixed two errors in your robots.txt: It’s User-agent, not Useragent, and empty lines aren’t allowed within a record. – unor Mar 26 at 9:19
  • 1) Is the content the same, or does the parameter change the content? 2) Would it be sufficient to disallow indexing, or don’t you want search engines to even see the content (i.e., crawl)? – unor Mar 26 at 9:20
  • Thanks @unor. I don't want search engines to see the content. The parameter will create alot of new pages and I don't want it to affect the crawl budget. The parameter does change the content (most will change, some will be the same). – pixel1234 Mar 26 at 15:50
1

Adding a meta tag and disallowing in robots.txt if not a good approach. You should only do one or the other. If you disallow then Google won't be able to see the status when it crawls. Of the two, the meta tag is a much stronger signal than robots.txt. Google sometimes indexes pages that are disallowed. However if Google can crawl and see the tag it won't index the page.

There is a better option than either of those. You should add a canonical tag instead. The canonical tag lets Google know about the duplication and which one is your preferred URL. Using the canonical tag can take advantage of inbound links to the parameter that noindex tag will waste.

  • Thanks @stephen. What if the parameters are creating millions of temporary new pages? Should I be concerned about crawl budget if I don't exclude in robots? And would a canonical tag work even if the content isn't the same? – pixel1234 Mar 26 at 21:42
  • The canonical tag only works if the content is nearly identical. What are these "temporary parameters?" Are they session ids or something that change for every visitor? If so they will make your site uncrawlable and ruin your SEO no matter how you try to handle them. – Stephen Ostermiller Mar 26 at 22:10
1

I would say (global)

*?*foo*

or (only /tool/input)

/tool/input?*foo*

please note: parameters are exchangeable and have no sorting!

  • Thanks Marius! I'm curious why you suggested removing the first slash. Seems like "/" is recommended here: developers.google.com/search/reference/robots_txt – pixel1234 Mar 26 at 18:41
  • You´re right! Mistake! - fixed it – Marius Mar 27 at 7:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.