1

I have an error checker on my password reset page which tells me everytime someone comes to reset their password without a valid token.

I have noticed within the last couple of days that the AdsBot-Google has been spamming this page quite regularly (since the sites launch) with invalid, yet with a valid format, tokens (don't ask me where it got them).

I have checked the validity of the user agent by pinging the originating server (66.249.90.34) and checkig its SSL cert (since all Google bots now use SSL).

Some sites say that the Ads bot scans an ad's landing page, however, my password reset page is defintely not used as a landing page for any of my ads.

I read about this on a Google search but, as usual, every site contradicts the other.

Should I block the bot from this page if it is not needed?

Is it good practice to block the google ads bot on certain pages or is there another reason this is happening?

Improve this question
3
  • 1
    If you are serving ads on your password rest page, I would remove them. As well, I would block access to this page specifically in your robots.txt file. This should solve the problem.
    – closetnoc
    Dec 23, 2014 at 14:39
  • Are you running Google Analytics on the page by any chance? Do you promote the site using AdWords?
    – Nick
    Jun 2, 2016 at 13:13
  • @Nick yes and yes
    – Sammaye
    Jun 2, 2016 at 17:07

2 Answers 2

Reset to default
2

The AdsBot crawler with user agent AdsBot-Google (+http://www.google.com/adsbot.html), is part of AdWords not AdSense. It is supposedly used for checking landing page content to determine quality score.

This bot appears to also visit pages that are not ad landing pages, including pages with no ads on them, no ads to them, and no links to them. Based on the query string parameters it tries to use, I suspect that it may be getting its URL's from Analytics, although I have no idea why Google is using its AdBot to crawl pages that are logged in Analytics but aren't related to advertising in any way.

If you aren't running any ads on the page or to the page, you might try protecting the page with a conditional 410 Gone response code based on user agent if the page is of a temporary nature and only works with a token. Once the token is used, Gone would be an appropriate response since the page no longer exists.

Improve this answer
2
  • I tied to get an answer out of Google themselves on this one and they never came back with anything. I am concerned that if I conditionally 403 the bot it might have an effect on the way Google perceives me
    – Sammaye
    Jun 2, 2016 at 21:58
  • I've updated the answer to suggest a 410 Gone code for use on temporary pages that require a token. That shouldn't be viewed by Google as a faulty website or as trying to hide anything. It just identifies that the requested resource is no longer available.
    – Nick
    Jun 2, 2016 at 22:03
1

Mediapartners-Google (the AdSense bot) will crawl any page where you serve their ads. It is likely crawling these pages because you have ads on them.

When you say "with invalid, yet with a valid format, token" could they be expired tokens? That would make sense. The user uses these tokens (which invalidates them) and then later the bot comes to check the page.

You may certainly add rules for Mediapartners-Google to your robots.txt file. Google asks that you allow it to crawl your whole site so that it can serve ads anywhere. If it isn't allowed to crawl somewhere, then it just won't show ads there.

I can't find a reference to it now, but I believe that AdSense doesn't allow you to serve ads on pages like your "login" page or "password reset" page. Only pages with content that is relevent to your site may serve ads. If it is showing ads triggering these crawls, then you may be in violation of the AdSense terms of service.

Improve this answer
6
  • I actually have no ads displayed on the site, the ads are externally showing on other peoples sites
    – Sammaye
    Jan 4, 2015 at 21:16
  • Also the google ads bot does not seem to comply to no index rules that the normal google bot does so you cannot easily block it without hacking your servers policy directly which sounds wrong
    – Sammaye
    Jan 4, 2015 at 21:17
  • Google's documentation indicates that the ads bot (Mediapartners-Google) does indeed follow robots.txt: support.google.com/webmasters/answer/1061943?hl=en
    – Stephen Ostermiller ♦
    Jan 4, 2015 at 22:03
  • Ah thanks, gotta find out why it isn't then, I have tested the robots in GWT and it blocks the URL but I am still get the ads bot going there
    – Sammaye
    Jan 4, 2015 at 22:40
  • 1
    I can confirm that AdSense has nothing to do with this spider appearing. I came here after getting tons of error reports on an internal page with no links to it, no ads on it (we are not even in AdSense). Goggle was making requests to this page which does have a token URL param, but using invalid tokens. The page isn't live yet and can only be accessed directly, so I don't know how it found it.
    – Nick
    Jun 2, 2016 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.