1

I am optimising the crawling experience of the website as a large portion of the site isn't being crawled.

A large portion of the 'crawl budget' is being used on crawling URLs with utm parameters for Google Shopping.

If I block these parameters from being crawled in the robots.txt file, will this have an impact on the Google Shopping ads?

I'm struggling to find confirmation of the Googlebot Google Merchant uses. I'm thinking that if it uses Adsbot - Google then I will block the parameters just for regular Googbot but allow for Adsbot.

My question;

  • Does Google Merchant use Adsbot?
  • If not, will blocking these pages from Googlebot negatively impact the shopping campaign?
  • If it will, are there other alternatives to stop Googlebot wasting time on these pages?
1

No, because they are different bots: the shopping googlebot and the web googlebot. Block parameters from the web googlebot, but let open for shopping googlebot. Shopping is the adsbot.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.