Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

almost every day something (it should be a bot) visits all of my pages on my website (amongst other things it doesn't correctly handle spaces in urls) with Java/1.7.0_07 (or different java versions) as referrer.

Should I worry? Should I block by useragent anything that contains "Java"?

Thank you

share|improve this question
up vote 2 down vote accepted

It seems to be Java crawler. Anyone could have written it or used existing one. I would say you don't need to worry as www is kind of "read only" so no crawl will harm your page. However this is possible way for stealing content or analyse it for an undesired purpose.

It depends on you, will you block it :)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.