I'm looking for a VPS or dedicated that can have a web crawler running all day long. It would gather data and then add it to some websites.

These websites currently have around 1k unique visits/day, and use 35gb of diskspace.

My budget for this is 20-60 dollars/month

I found a good offer in 'myhosting', but it is too much better than the average (which makes me mistrust), and I couldn't find opinions about it. Also they didn't answer the email I sent to them asking about the Mbps (they offer 600gb transfer/month)

Any thoughts?


What language is your crawler written in? Have you considered using Google Application Engine? (GAE) They are free to a certain point, support Java and Python.


  • PHP.. I don't know what that is, I'll check it out, brb. Update: I don't want to share my code with anyone, so I rather have it in my own – HappyWebMaster Nov 16 '10 at 17:28
  • PHP won't work with GAE, so that's out. Just to point out, your code would still be your own (assuming you don't think Google will steal your code regardless). Since you're in the PHP realm, you can consider Dreamhost, they have very reasonable offerings, with unlimited disk and network.. (be wary of "unlimited" though, there are caveats). dreamhost.com/vps-pricing.html midrange is $50/month. – BDM Nov 16 '10 at 18:10
  • You can consider Rackspace as well. They are fantastic, but pricey. rackspacecloud.com/cloud_hosting_products/servers/pricing – BDM Nov 16 '10 at 18:14

You'll need a dedicated server for this - you shouldn't have any problems with most DS hosts. A few are listed here but there are plenty out there so just shop around.

The plus side of having your own server is you can host your other web sites on there, assuming your crawler is not a resource hog. A URL retrieval every 2-5 seconds would not slow down the server for the other sites.

Alternatively, why not run it from your own computer? Web pages are usually only a few dozen kB, or smaller with gzip, so it probably wouldn't use masses of your bandwidth.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy