Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have created a program that searches through every player of a particular online game, visits their information webpage and extracts pieces of information about them (ie. their stats).

The problem is that there are several millions of players. By my initial calculations, it may take over 10 days to complete, and will use up over 30gbs of data traffic. This is less than ideal when you have a 40gb monthly allowance and you want to run the program weekly.

My question is this. How can I run my program quickly and cheaply. For instance, is it possible to buy some webspace with a webhosting company and run my java program somehow from there? I have seen some webhosting for around $2 per month, which seems pretty reasonable.

Or is it a webserver I would be after. Although they seem rather expensive. I am only doing this for my own interest and wouldn't want to spent more than a few dollars.

Thanks

share|improve this question
    
Perhaps you should ask yourself whether you can achieve your aims using a random sampling of a mere 100k players. –  Peter Taylor Aug 16 '11 at 20:15
add comment

2 Answers

up vote 3 down vote accepted

Also you have to consider that you will also eat up 30GB x 4 of the websites data traffic per month, depending on the website this can be a huge problem for the operator and they will probably detect the bandwidth usage spike as an attack on their website.

share|improve this answer
add comment

Yes, you can purchase the use of a remote computer that would allow you to use that computer's access to the Internet to collect your data.

Here are some difficulties you will still face even if you go this route:

  1. You will need to set up, configure and manage a remote computer, probably from a SSH command line interface.

  2. The info you collected is still stored on a remote computer, you will face the same bandwidth issues you faced before in attempting to move that data to your local computer.

  3. You will need to pay for the bandwidth your new server will use to access the data you are after.

It is probably much easier and more cost effective to purchase additional bandwidth for your local computer which is already configured

share|improve this answer
    
Hey. Thanks for the reply. I have experience in using computers remotely (putty, winSCP) etc I don't think that will be a problem. The second issue is also not much of a problem as I can use the data remotely. I'm concerned that my internet connection is not fast enough. I can't be waiting 10 days for the data. –  Roger Aug 16 '11 at 17:31
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.