6

I am collecting a large number of URLs. I am not responsible for the websites in question, and I want to remove tracking parameters that do not affect the content of the website. With the tracking parameters, it's impossible to identify two URLs that should be considered equal.

For example, if I have the following links:

  1. http://example.com/blog/post1?utm_xyz=1234
  2. http://example.com/blog/post1?utm_xyz=5678
  3. http://example.net/viewblog?post_id=2&utm_xyz=9999

I want to convert to the equivalent canonical-type URLs:

  1. http://example.com/blog/post1
  2. http://example.com/blog/post1
  3. http://example.net/viewblog?post_id=2

The first two are for the same content, but have different tracking parameters. The last example illustrates why I can't just remove all query parameters.

The most common of these are the utm_ ones, but I have also found:

  • Piwik: pk_campaign and pk_kwd
  • WebTrends: WT.nav, WT.mc_id
  • unknown, maybe Apple: campaign_id
  • Wikimedia: wprov
  • HootSuite: hootPostID

Is there a well-known list of these query parameters that I can safely remove?

(I am using the canonical URLs where they are supplied in the HTML metadata, but I want to use this approach when none is supplied.)

Improve this question
5
  • What do you mean by "safely remove"? In what context? Removing these query parameters will obviously break tracking, so why do you want to remove them? – Stephen Ostermiller♦ Oct 9 '17 at 10:31
  • 1
    I'm collecting URLs (I'm doing something similar to running a search engine crawler). I want to remove the tracking parameters from these URLs to create a canonical URL. By 'safely' I mean not remove parameters that will serve different content. For example post_id in a blog post URL is important. – Joe Oct 9 '17 at 10:39
  • That makes sense. I know Googlebot auto-ignores the UTM tracking parameters. I'd assume it probably does so for some of the other tracking params as well. I've never seen a published list of these parameters from any search engine crawler though. It would certainly be nice to create such a list. – Stephen Ostermiller♦ Oct 9 '17 at 11:43
  • Agreed. I'm glad I didn't miss something obvious! – Joe Oct 9 '17 at 11:57
  • This is a useful question. Wikipedia, for example, forbids tracking parameters in URLs. – sondra.kinsey Mar 21 '19 at 20:11
1

I guess your intention is to clean the scraped URLs.

You can refer to articles on best practices of using UTM. Commonly used keywords for utm_medium are based on the naming conventions used in Google Analytics such as, social, referral, email.

At the end of the day there is no good way if you’re doing this based on a fixed list of keywords. Because the parameters can be anything.

You will have a better chance of sanitising your results by using regex to detect and remove any UTM parameters.

For a URL like https://example.com?utm_source=facebook&utm_medium=social&utm_campaign=book-launch-2014 you need to search and replace the parameters with nothing:

  • utm_source
  • utm_medium
  • utm_campaign
  • utm_term
  • utm_content
Improve this answer
1

Part of my RewriteCond used to deduplicate URL for a more efficient caching: utm_(?:source|medium|campaign|term|content)|gclid|fbclid|msclkid|emci|emdi|ceid|sourceid|hootPostID|__s

Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.