2

I would like to operate a website crawler/scraper from my web server (doing so ethically of course).

What I am wondering is whether this uses my website bandwidth up? If so how badly and is it possible to minimise bandwidth usage? I am currently assigned 5000GB per month website bandwidth.

2
  • Note that I am the one doing the crawling through a PHP web crawler Jul 21, 2013 at 15:28
  • i.e. download rather than upload... Jul 21, 2013 at 15:28

1 Answer 1

2

Anything that requests a resource (a.k.a. web page, image file, stylesheet, etc) from your server uses bandwidth. So crawling other sites from your server will count against your bandwidth limit.

5
  • What about if I am the one crawling other websites? Does that use up my bandwidth as well? Jul 21, 2013 at 15:27
  • Yes. I misread your question and updated my answer accordingly.
    – John Conde
    Jul 21, 2013 at 15:29
  • Ok then, looks like I'll have to find out how much additional bandwidth costs then!! Jul 21, 2013 at 15:31
  • Things to consider, if you're going to request pages from other sites more then once is to record the last modified date of the page. Then when you want to crawl it again you can get that info without downloading the page and if it hasn't changed you don't have to go and get it again saving you bandwidth.
    – John Conde
    Jul 21, 2013 at 15:32
  • Also, attempt to retrieve the page compressed and then uncompress it on the server. That will save bandwidth, too.
    – John Conde
    Jul 21, 2013 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.