I have a list of about one million URLs that I would like fetched and would rather not run this process on my own web server. I am wondering if there are any commercial solutions to this - i.e. any services that will take my list of URLs and return the HTML for each (running it behind proxies, etc., if necessary).
Tell me more
×
Webmasters Stack Exchange is a question and answer site for
pro webmasters. It's 100% free, no registration required.
|
|
||||
|
|
|
A Linux virtual private server (some priced at less than $10/mo) can be used to complete this task with minimal configuration - see the wget man page for instructions if you choose to go this route. |
|||
|
|
