I'm getting over 100 requests per day from user agent google-xrawler. The requests are all for the same page. Some of the IP addresses are known to be Google IPs, although they don't have valid reverse DNS records.

What is this user agent used for?

  • He is right - Nothing about it on the entire Internet – Goyllo Apr 18 '17 at 18:11
  • Already found and read that thread. Nothing in it says anything about what this user-agent is or does, other than suggesting it might be Google translate, but no one is going to translate the same page, only one page, and 100 times per day every day. – Nick Apr 18 '17 at 19:01
  • "I'm getting over 100 requests" - requests or page views? – DocRoot Apr 18 '17 at 21:52
  • @DocRoot: They are entries in the Nginx access log. They are usually in pairs of HEAD/GET requests. – Nick Apr 19 '17 at 1:24
  • Is it send request only to your feed page? – Goyllo Apr 19 '17 at 5:07

Google requests product feed using the user agent "google-xrawler"


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.