4

I'm getting over 100 requests per day from user agent google-xrawler. The requests are all for the same page. Some of the IP addresses are known to be Google IPs, although they don't have valid reverse DNS records.

What is this user agent used for?

5
  • He is right - Nothing about it on the entire Internet
    – Goyllo
    Apr 18, 2017 at 18:11
  • Already found and read that thread. Nothing in it says anything about what this user-agent is or does, other than suggesting it might be Google translate, but no one is going to translate the same page, only one page, and 100 times per day every day.
    – Nick
    Apr 18, 2017 at 19:01
  • "I'm getting over 100 requests" - requests or page views?
    – DocRoot
    Apr 18, 2017 at 21:52
  • @DocRoot: They are entries in the Nginx access log. They are usually in pairs of HEAD/GET requests.
    – Nick
    Apr 19, 2017 at 1:24
  • Is it send request only to your feed page?
    – Goyllo
    Apr 19, 2017 at 5:07

1 Answer 1

2

Google requests product feed using the user agent "google-xrawler"

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.