My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different.
I'm running a price comparison website. I have ads directing users to my home page, and other ads directing users to a product page.
While browsing my server's access logs, I noticed that Adsbot was loading these landing pages as expected, but also links present in this page, even though these are never used as landing pages.
Like many price comparison sites, when a user clicks on a link to go check a merchant's offer, this takes him to a "Redirecting..." page that contains an analytics tag and and Adwords conversion tracker for my statistics, then redirects the user to the actual merchant page.
The problem is, AdsBot is following these links from my product pages.
I can't understand why it's doing this, as it's only supposed to assess the quality of my landing pages. I can definitely understand why it's loading assets (scripts, stylesheets, images) from the page to measure page speed, but why would it follow the damn links on the page afterwards? This is killing my statistics with "fake" clicks.
My robots.txt contains exclusions for all bots for these pages:
User-agent: * Disallow: /product/*/go/
As we know, AdsBot ignores the blanket exclusion and must be excluded specifically:
In order to avoid decreasing Quality Scores and Ad Rank and increasing CPCs for advertisers who don't intend to restrict AdWords visits to their pages, the system will ignore blanket exclusions and wildcards (for example, User-agent: *) in robots.txt files.
I could specifically forbid AdsBot from accessing these pages:
User-agent: AdsBot-Google Disallow: /product/*/go/
But I'm very nervous about taking this step, as this same page gives a warning about blocking AdsBot in general:
Important: Blocking the review of your webpages can have serious effects: Ads not showing [...], Ads disapproved [...], Site suspended [...]
... in addition to the decreased QS and increased CPC mentioned above.
Even though I'm not blocking AdsBot from legitimately visiting my landing pages, could it penalize me for disallowing visits to links on my website that my landing pages contain?
It came to my mind that I'm also using dynamic search ads. This thread suggests that AdsBot data could be used in place of Googlebot data to index pages for paid search results. Could this be the reason why AdsBot is crawling more than my landing pages?
This would still make me wonder why, in this specific case, AdsBot would not follow the blanket exclusions (why would I want paid traffic on pages I want to be excluded from organic search?) but would make me more confident in disallowing access to AdsBot to parts of my website, if it's not there to assess the quality of my website, but rather to discover content to index for dynamic search.