I have created a program that searches through every player of a particular online game, visits their information webpage and extracts pieces of information about them (ie. their stats).

The problem is that there are several millions of players. By my initial calculations, it may take over 10 days to complete, and will use up over 30gbs of data traffic. This is less than ideal when you have a 40gb monthly allowance and you want to run the program weekly.

My question is this. How can I run my program quickly and cheaply. For instance, is it possible to buy some webspace with a webhosting company and run my java program somehow from there? I have seen some webhosting for around $2 per month, which seems pretty reasonable.

Or is it a webserver I would be after. Although they seem rather expensive. I am only doing this for my own interest and wouldn't want to spent more than a few dollars.


  • 1
    Perhaps you should ask yourself whether you can achieve your aims using a random sampling of a mere 100k players. – Peter Taylor Aug 16 '11 at 20:15

Also you have to consider that you will also eat up 30GB x 4 of the websites data traffic per month, depending on the website this can be a huge problem for the operator and they will probably detect the bandwidth usage spike as an attack on their website.


Yes, you can purchase the use of a remote computer that would allow you to use that computer's access to the Internet to collect your data.

Here are some difficulties you will still face even if you go this route:

  1. You will need to set up, configure and manage a remote computer, probably from a SSH command line interface.

  2. The info you collected is still stored on a remote computer, you will face the same bandwidth issues you faced before in attempting to move that data to your local computer.

  3. You will need to pay for the bandwidth your new server will use to access the data you are after.

It is probably much easier and more cost effective to purchase additional bandwidth for your local computer which is already configured

  • Hey. Thanks for the reply. I have experience in using computers remotely (putty, winSCP) etc I don't think that will be a problem. The second issue is also not much of a problem as I can use the data remotely. I'm concerned that my internet connection is not fast enough. I can't be waiting 10 days for the data. – Roger Aug 16 '11 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.