Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm looking for a VPS or dedicated that can have a web crawler running all day long. It would gather data and then add it to some websites.

These websites currently have around 1k unique visits/day, and use 35gb of diskspace.

My budget for this is 20-60 dollars/month

I found a good offer in 'myhosting', but it is too much better than the average (which makes me mistrust), and I couldn't find opinions about it. Also they didn't answer the email I sent to them asking about the Mbps (they offer 600gb transfer/month)

Any thoughts?

share|improve this question

2 Answers 2

What language is your crawler written in? Have you considered using Google Application Engine? (GAE) They are free to a certain point, support Java and Python.

http://code.google.com/appengine/

share|improve this answer
    
PHP.. I don't know what that is, I'll check it out, brb. Update: I don't want to share my code with anyone, so I rather have it in my own –  HappyWebMaster Nov 16 '10 at 17:28
    
PHP won't work with GAE, so that's out. Just to point out, your code would still be your own (assuming you don't think Google will steal your code regardless). Since you're in the PHP realm, you can consider Dreamhost, they have very reasonable offerings, with unlimited disk and network.. (be wary of "unlimited" though, there are caveats). dreamhost.com/vps-pricing.html midrange is $50/month. –  Mutant Nov 16 '10 at 18:10
    
You can consider Rackspace as well. They are fantastic, but pricey. rackspacecloud.com/cloud_hosting_products/servers/pricing –  Mutant Nov 16 '10 at 18:14

You'll need a dedicated server for this - you shouldn't have any problems with most DS hosts. A few are listed here but there are plenty out there so just shop around.

The plus side of having your own server is you can host your other web sites on there, assuming your crawler is not a resource hog. A URL retrieval every 2-5 seconds would not slow down the server for the other sites.

Alternatively, why not run it from your own computer? Web pages are usually only a few dozen kB, or smaller with gzip, so it probably wouldn't use masses of your bandwidth.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.