Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I would like to operate a website crawler/scraper from my web server (doing so ethically of course).

What I am wondering is whether this uses my website bandwidth up? If so how badly and is it possible to minimise bandwidth usage? I am currently assigned 5000GB per month website bandwidth.

share|improve this question
    
Note that I am the one doing the crawling through a PHP web crawler –  Amy Neville Jul 21 '13 at 15:28
    
i.e. download rather than upload... –  Amy Neville Jul 21 '13 at 15:28

1 Answer 1

up vote 1 down vote accepted

Anything that requests a resource (a.k.a. web page, image file, stylesheet, etc) from your server uses bandwidth. So crawling other sites from your server will count against your bandwidth limit.

share|improve this answer
    
What about if I am the one crawling other websites? Does that use up my bandwidth as well? –  Amy Neville Jul 21 '13 at 15:27
    
Yes. I misread your question and updated my answer accordingly. –  John Conde Jul 21 '13 at 15:29
    
Ok then, looks like I'll have to find out how much additional bandwidth costs then!! –  Amy Neville Jul 21 '13 at 15:31
    
Things to consider, if you're going to request pages from other sites more then once is to record the last modified date of the page. Then when you want to crawl it again you can get that info without downloading the page and if it hasn't changed you don't have to go and get it again saving you bandwidth. –  John Conde Jul 21 '13 at 15:32
    
Also, attempt to retrieve the page compressed and then uncompress it on the server. That will save bandwidth, too. –  John Conde Jul 21 '13 at 15:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.