2

I am optimising the crawling experience of the website as a large portion of the site isn't being crawled.

A large portion of the 'crawl budget' is being used on crawling URLs with utm parameters for Google Shopping.

If I block these parameters from being crawled in the robots.txt file, will this have an impact on the Google Shopping ads?

I'm struggling to find confirmation of the Googlebot Google Merchant uses. I'm thinking that if it uses Adsbot - Google then I will block the parameters just for regular Googbot but allow for Adsbot.

My question;

  • Does Google Merchant use Adsbot?
  • If not, will blocking these pages from Googlebot negatively impact the shopping campaign?
  • If it will, are there other alternatives to stop Googlebot wasting time on these pages?
1

2 Answers 2

2

No, because they are different bots: the shopping googlebot and the web googlebot. Block parameters from the web googlebot, but let open for shopping googlebot. Shopping is the adsbot.

2

A bit late to the party, but it's better to address this before somebody else gets the wrong decisions.

UTM parameters are querystring variable you can add to a URL that should be used in external links.

Already if they are going to be used in internal links, this is a big mistake for GA Universal, as each internal link using a utm_* field triggers a new session start. So unless you want your numbers being inflated without reason, you should be refrain from doing so. This behaviour is changing with GA 4 ... but until then it is a big NO.

What happens when someone clicks on a URL with UTM parameters?

When the URL is hit, and the server renders a page, those variables are later sent back to your Google Analytics which will make them available to your for segmenting the data.

UTM parameters should not be used for anything else; meaning that no logic should be appended to alter the behaviour of the site.

Crawling wise, what does that mean?

The above comes in tandem with the fact that regardless of the crawler, regardless of the variable, the task of a crawler is to get the returned HTML content and do parse it according to its own programmed tasks.

  • Googlebot will come and pick the content to index and later rank pages to return them according to the user demand
  • AdsBot will execute the page to check the conformity of the page to be used for create listing, understand what marketers are interested the most, adjust the biddings criteria and so on.
  • Other crawler ... I well don't know

When a crawler executes a page, it does have no knowledge of what a given variable does. A UTM variable is like a test=123 ... it does something that should be only meaningful for the site. So crawlers certainly will not try to understand whether a utm_source=twitter or utm_campaing=san_valentine is useful/important/meaningful ... also because the content of those variable could be - to an extent - encrypted and made significative only for the site owner alone.

E.g. I could create a utm_source=xsh1232 that in an internal mapping of mine correspond to a given site, but you will never understand as an external user/crawler what it means.

Long story short

Don't overthink too much. There is no need to add utm_ into your robots.txt unless some colleagues of your/or your consulted firm has not created internal links using them. Should that be the case, a disallow: *utm_ in the generic rules is more than enough to avoid the crawler following them for nothing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.