2

I would like to operate a website crawler/scraper from my web server (doing so ethically of course).

What I am wondering is whether this uses my website bandwidth up? If so how badly and is it possible to minimise bandwidth usage? I am currently assigned 5000GB per month website bandwidth.

Improve this question
2
  • Note that I am the one doing the crawling through a PHP web crawler
    – Amy Neville
    Jul 21, 2013 at 15:28
  • i.e. download rather than upload...
    – Amy Neville
    Jul 21, 2013 at 15:28

1 Answer 1

Reset to default
2

Anything that requests a resource (a.k.a. web page, image file, stylesheet, etc) from your server uses bandwidth. So crawling other sites from your server will count against your bandwidth limit.

Improve this answer
5
  • What about if I am the one crawling other websites? Does that use up my bandwidth as well?
    – Amy Neville
    Jul 21, 2013 at 15:27
  • Yes. I misread your question and updated my answer accordingly.
    – John Conde ♦
    Jul 21, 2013 at 15:29
  • Ok then, looks like I'll have to find out how much additional bandwidth costs then!!
    – Amy Neville
    Jul 21, 2013 at 15:31
  • Things to consider, if you're going to request pages from other sites more then once is to record the last modified date of the page. Then when you want to crawl it again you can get that info without downloading the page and if it hasn't changed you don't have to go and get it again saving you bandwidth.
    – John Conde ♦
    Jul 21, 2013 at 15:32
  • Also, attempt to retrieve the page compressed and then uncompress it on the server. That will save bandwidth, too.
    – John Conde ♦
    Jul 21, 2013 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.