7

I am collecting a large number of URLs. I am not responsible for the websites in question, and I want to remove tracking parameters that do not affect the content of the website. With the tracking parameters, it's impossible to identify two URLs that should be considered equal.

For example, if I have the following links:

  1. http://example.com/blog/post1?utm_xyz=1234
  2. http://example.com/blog/post1?utm_xyz=5678
  3. http://example.net/viewblog?post_id=2&utm_xyz=9999

I want to convert to the equivalent canonical-type URLs:

  1. http://example.com/blog/post1
  2. http://example.com/blog/post1
  3. http://example.net/viewblog?post_id=2

The first two are for the same content, but have different tracking parameters. The last example illustrates why I can't just remove all query parameters.

The most common of these are the utm_ ones, but I have also found:

  • Piwik: pk_campaign and pk_kwd
  • WebTrends: WT.nav, WT.mc_id
  • unknown, maybe Apple: campaign_id
  • Wikimedia: wprov
  • HootSuite: hootPostID

Is there a well-known list of these query parameters that I can safely remove?

(I am using the canonical URLs where they are supplied in the HTML metadata, but I want to use this approach when none is supplied.)

5
  • What do you mean by "safely remove"? In what context? Removing these query parameters will obviously break tracking, so why do you want to remove them? Commented Oct 9, 2017 at 10:31
  • 1
    I'm collecting URLs (I'm doing something similar to running a search engine crawler). I want to remove the tracking parameters from these URLs to create a canonical URL. By 'safely' I mean not remove parameters that will serve different content. For example post_id in a blog post URL is important.
    – Joe
    Commented Oct 9, 2017 at 10:39
  • That makes sense. I know Googlebot auto-ignores the UTM tracking parameters. I'd assume it probably does so for some of the other tracking params as well. I've never seen a published list of these parameters from any search engine crawler though. It would certainly be nice to create such a list. Commented Oct 9, 2017 at 11:43
  • Agreed. I'm glad I didn't miss something obvious!
    – Joe
    Commented Oct 9, 2017 at 11:57
  • This is a useful question. Wikipedia, for example, forbids tracking parameters in URLs. Commented Mar 21, 2019 at 20:11

2 Answers 2

3

Part of my RewriteCond used to deduplicate URL for a more efficient caching: utm_(?:source|medium|campaign|term|content)|gclid|fbclid|msclkid|emci|emdi|ceid|sourceid|hootPostID|__s

0
1

I guess your intention is to clean the scraped URLs.

You can refer to articles on best practices of using UTM. Commonly used keywords for utm_medium are based on the naming conventions used in Google Analytics such as, social, referral, email.

At the end of the day there is no good way if you’re doing this based on a fixed list of keywords. Because the parameters can be anything.

You will have a better chance of sanitising your results by using regex to detect and remove any UTM parameters.

For a URL like https://example.com?utm_source=facebook&utm_medium=social&utm_campaign=book-launch-2014 you need to search and replace the parameters with nothing:

  • utm_source
  • utm_medium
  • utm_campaign
  • utm_term
  • utm_content
1
  • The question was, is there a set of well-known tracking parameters besides utm_*
    – Hakanai
    Commented Nov 16, 2023 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.