I keep getting calls to my server where there is random Javascript appended on the end of lots of the calls, e.g.:


The user agent is always this:


I have jQuery, Modernizr and other JS and originally thought that some browser was messing up it's JS calls, however this particular IP address hasn't requested any images so I'm wondering if it is some kind of attack.

Is this a common occurence?

  • The fact that they haven't requested any images means that whoever is behind it has gone to great lengths to modify their browser to prevent this. How far apart are the requests? – Christopher Aug 17 '12 at 10:51
  • Its anywhere between a couple of seconds and a minute between them. – cjk Aug 17 '12 at 12:43

This is almost certainly a robot.

It is a common trick for robots to use simple heuristics to look for URLs within JavaScript strings. It's fairly simple for absolute URLs, but for relative URLs it can be hit and miss, in this case clearly more of a miss.

The fact that the robot is not crawling with a user agent that identifies him may mean that it has some less than honest intentions. If you can block the IP address, I would do so if for no other reason than the misleading user-agent, it is bad etiquette for robots to have user-agents that provide no contact info.

In cases where you see this pattern and the user agent provided contact info, I would contact the operator(s). If it is a legit crawl, they will accommodate you.

I do not believe this is an attack, though. Most likely this is a text scraper (therefor no images). Possibly scraping content to populate link farms.

  • Agreed, considering that the calls are coming in at a few seconds each. Although I'd have to disagree on one point - the modified user agent. See the Wikipedia page on User Agents (User Agent Spoofing): en.wikipedia.org/wiki/User_agent#User_agent_spoofing – Christopher Aug 17 '12 at 14:06
  • @Christopher It is good etiquette, if your a crawl operator, to design your user-agent in a manner that allows site admins to contact you if there is a problem. Typically, this takes the form of "Mozilla/5.0 (compatible; CRAWLERNAME/VERSION +OPERATOR_CONTACT_URL)", e.g. Mozilla/5.0 (compatible; heritrix/3.1 +example.com). This gets around most of the browser sniffing while still being polite to the site admins. I speak from experience having operated web crawlers for the last 8 years. Also, this is staight out of the Internet Archive's playbook. One of the biggest crawlers around. – Kris Aug 17 '12 at 15:15
  • I apologize, when I read your answer it seemed like you were making a general statement - not specified to the crawler scope. – Christopher Aug 17 '12 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.