I keep getting calls to my server where there is random Javascript appended on the end of lots of the calls, e.g.:


The user agent is always this:


I have jQuery, Modernizr and other JS and originally thought that some browser was messing up it's JS calls, however this particular IP address hasn't requested any images so I'm wondering if it is some kind of attack.

Is this a common occurence?

  • The fact that they haven't requested any images means that whoever is behind it has gone to great lengths to modify their browser to prevent this. How far apart are the requests? Aug 17, 2012 at 10:51
  • Its anywhere between a couple of seconds and a minute between them.
    – cjk
    Aug 17, 2012 at 12:43

1 Answer 1


This is almost certainly a robot.

It is a common trick for robots to use simple heuristics to look for URLs within JavaScript strings. It's fairly simple for absolute URLs, but for relative URLs it can be hit and miss, in this case clearly more of a miss.

The fact that the robot is not crawling with a user agent that identifies him may mean that it has some less than honest intentions. If you can block the IP address, I would do so if for no other reason than the misleading user-agent, it is bad etiquette for robots to have user-agents that provide no contact info.

In cases where you see this pattern and the user agent provided contact info, I would contact the operator(s). If it is a legit crawl, they will accommodate you.

I do not believe this is an attack, though. Most likely this is a text scraper (therefor no images). Possibly scraping content to populate link farms.

  • Agreed, considering that the calls are coming in at a few seconds each. Although I'd have to disagree on one point - the modified user agent. See the Wikipedia page on User Agents (User Agent Spoofing): en.wikipedia.org/wiki/User_agent#User_agent_spoofing Aug 17, 2012 at 14:06
  • @Christopher It is good etiquette, if your a crawl operator, to design your user-agent in a manner that allows site admins to contact you if there is a problem. Typically, this takes the form of "Mozilla/5.0 (compatible; CRAWLERNAME/VERSION +OPERATOR_CONTACT_URL)", e.g. Mozilla/5.0 (compatible; heritrix/3.1 +example.com). This gets around most of the browser sniffing while still being polite to the site admins. I speak from experience having operated web crawlers for the last 8 years. Also, this is staight out of the Internet Archive's playbook. One of the biggest crawlers around.
    – Kris
    Aug 17, 2012 at 15:15
  • I apologize, when I read your answer it seemed like you were making a general statement - not specified to the crawler scope. Aug 17, 2012 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.