4

I'm getting over 100 requests per day from user agent google-xrawler. The requests are all for the same page. Some of the IP addresses are known to be Google IPs, although they don't have valid reverse DNS records.

What is this user agent used for?

Improve this question
5
  • He is right - Nothing about it on the entire Internet
    – Goyllo
    Commented Apr 18, 2017 at 18:11
  • Already found and read that thread. Nothing in it says anything about what this user-agent is or does, other than suggesting it might be Google translate, but no one is going to translate the same page, only one page, and 100 times per day every day.
    – Nick
    Commented Apr 18, 2017 at 19:01
  • "I'm getting over 100 requests" - requests or page views?
    – DocRoot
    Commented Apr 18, 2017 at 21:52
  • @DocRoot: They are entries in the Nginx access log. They are usually in pairs of HEAD/GET requests.
    – Nick
    Commented Apr 19, 2017 at 1:24
  • Is it send request only to your feed page?
    – Goyllo
    Commented Apr 19, 2017 at 5:07

1 Answer 1

Reset to default
2

Google requests product feed using the user agent "google-xrawler"

Improve this answer
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.