1

I have blocked the some of the URLs from robots.txt. I think there is No way to access the blocked URLs from my website. But I have seen in my logs, whatever I blocked the URLs from the robots.txt are crawled by Google bot. And also observed "Google bot" triggering the first request followed by "Mediapartners-Google". But I don't have the clarity that whether Googlebot crawling from any other leak or from "Mediapartners-Google".

And I want to know the relationship and differences between "Mediapartners-Google" and "Google bot".

  • 2
    If it is blocked in robots.txt then Google should not be crawling it. For how long have the resources been blocked? Have you confirmed in GSC that the robots.txt syntax is correct? Have you confirmed that these are real Googlebot requests (by performing a reverse lookup on the IP address)? – DocRoot Aug 29 '17 at 14:08
  • Yes I have done the reverse lookup, they are from google and there is no issue with robots.txt I have tested on robots.txt tester – nagababu thota Aug 29 '17 at 14:15
3

You say that there is No way to access the locked URL's from your website? I'm guessing you mean that you have no links pointing to those URL's from your own website. So that's what I will assume.

Blocking URL's or part of your website in your robots.txt is actually no guarantee that they will not be crawled. There are circumstances where the disallow directive in the robots.txt-file will be ignored. Usually it's because you have external links (backlinks) pointing to the URL's you wish to hide. If you are trying to prevent the URL to be indexed by Google you should use the X-Robots-Tag HTTP header. If do you that you must remove the disallow directive in your robots.txt-file, otherwise there could be a conflict. You can read about this in Google Search Console help section: https://support.google.com/webmasters/answer/7424835?hl=en#h12

Google bot is the "normal" bot from Google that will crawl your website and fetch data used for the index. Mediapartners-Google is a bot from Google AdSense or Google Mobile Adsense that will crawl your URL's so they understand what content is on your page(s) and display relevant Ads to your visitors. You can read about this on: http://www.botreports.com/user-agent/mediapartners-google.shtml

  • Interesting fact: All of Googles bots are the same bot. Or at least that was the way it was a couple of years ago. They are just identified differently and have different post fetch processes. Essentially, it is just a matter of queuing requests from various search services using a single but flexible fetch queuing process. – closetnoc Aug 29 '17 at 19:15
  • Sometimes Google will make requests without appearing to take the robots.txt into account. Google will fetch the robots.txt every time it visits in a period greater than 24 hours. If, for some reason, Google cannot fetch the robots.txt file, it will use the last one it did fetch. I have seen this continue for weeks. Google does try and avoid restricted pages. However, as you already know, it is impossible to do a perfect job when fetching pages. – closetnoc Aug 29 '17 at 19:24
  • @closetnoc Makes sense, good point. – Kristian Svensson Aug 29 '17 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.