Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a list of about one million URLs that I would like fetched and would rather not run this process on my own web server. I am wondering if there are any commercial solutions to this - i.e. any services that will take my list of URLs and return the HTML for each (running it behind proxies, etc., if necessary).

share|improve this question

1 Answer 1

A Linux virtual private server (some priced at less than $10/mo) can be used to complete this task with minimal configuration - see the wget man page for instructions if you choose to go this route.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.