Is there a Java-based web browser that doesn't follow hyperlinks properly?

Question

Looking through my logs, I get this error a lot:

A potentially dangerous Request.Path value was detected from the client (:).

The error is usually due to someone navigating to a URL like this:

http://octopusdeploy.com/blog/https:/tfspreview.com

You'll notice that what should have been a link to https://tfs... has become https:/tfs..., which the web browser is obviously intepreting as being a relative path instead of an absolute URL.

However, if you do a view-source on the original page (http://octopusdeploy.com/blog) you'll find links to https://tfs..., but none to https:/tfs... The client seems to be getting the links wrong.

Looking through the request details I notice that this isn't coming from any ordinary web browser, but from a Java client:

Platform: Unknown 
Browser: Unknown (Version 0.0)
User Agent: Java/1.6.0_04

Is there a known, badly written Java web browser, maybe running on a smart phone somewhere, that just doesn't know how to follow hyperlinks? Have you seen similar errors?

Secondly, if I were to block all requests from user agents identifying as Java bots (as per this post), am I likely to break any known important bots or devices?

Answer 1

A user agent of Java/[version] is the default UA when making any web requests through the Java programming language.

In other words, someone has written a spider-like program in Java, which is clearly parsing links incorrectly.

As Dave suggested in the comments, the requests could be coming from another site - check the referring URL in your weblogs.

In terms of blocking: no, there are no major bots that use the generic Java UA. All well-behaved bots use a proper UA. You could block the UA but the developer could just change it to use a real browser's UA like Chrome's or Firefox's. If it continues to cause problems, an IP block may be a sensible option.