Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a list of about one million URLs that I would like fetched and would rather not run this process on my own web server. I am wondering if there are any commercial solutions to this - i.e. any services that will take my list of URLs and return the HTML for each (running it behind proxies, etc., if necessary).

share|improve this question

A Linux virtual private server (some priced at less than $10/mo) can be used to complete this task with minimal configuration - see the wget man page for instructions if you choose to go this route.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.