Running a program that extracts data from websites

Question

I have created a program that searches through every player of a particular online game, visits their information webpage and extracts pieces of information about them (ie. their stats).

The problem is that there are several millions of players. By my initial calculations, it may take over 10 days to complete, and will use up over 30gbs of data traffic. This is less than ideal when you have a 40gb monthly allowance and you want to run the program weekly.

My question is this. How can I run my program quickly and cheaply. For instance, is it possible to buy some webspace with a webhosting company and run my java program somehow from there? I have seen some webhosting for around $2 per month, which seems pretty reasonable.

Or is it a webserver I would be after. Although they seem rather expensive. I am only doing this for my own interest and wouldn't want to spent more than a few dollars.

Thanks

Answer 1

Also you have to consider that you will also eat up 30GB x 4 of the websites data traffic per month, depending on the website this can be a huge problem for the operator and they will probably detect the bandwidth usage spike as an attack on their website.

Answer 2

Yes, you can purchase the use of a remote computer that would allow you to use that computer's access to the Internet to collect your data.

Here are some difficulties you will still face even if you go this route:

1. You will need to set up, configure and manage a remote computer, probably from a SSH command line interface.

2. The info you collected is still stored on a remote computer, you will face the same bandwidth issues you faced before in attempting to move that data to your local computer.

3. You will need to pay for the bandwidth your new server will use to access the data you are after.

It is probably much easier and more cost effective to purchase additional bandwidth for your local computer which is already configured