32

I have a TB of compressed media files that I need to get to a remote web server with a bit of urgency. (from my office)

Our upload rate is only 500KB/s, which would suggest ~23 days to upload.

The new hosting company does not accept drives-by-mail.

Is there a good place to go in major metro areas to borrow a fatter pipe?

Improve this question
4
  • 10
    I know this isn't an answer, or even particularly helpful, but if your new hosting company is not willing to work with you on this you need to find a new new hosting company. Not accepting a drive from a customer (for a fee of course) is pretty sucky.
    – voretaq7
    Mar 15, 2012 at 22:45
  • Check my answer it's a data storage company in Tulsa
    – Anagio
    Mar 15, 2012 at 22:53
  • It's got good answers here, but shouldn't this have been transfered to ServerFault?
    – Mark Hurd
    Mar 16, 2012 at 2:53
  • 1
    Perhaps, but I view serverfault as a place to discuss the configuration of hardware and software, whereas this is the usage of whatever webmasters have at their disposal to achieve a more composite task
    – Brian Webster
    Mar 16, 2012 at 3:36

10 Answers 10

Reset to default
16

I would contact some of your local data-centers (rack-hosting, managed service providers, etc) and see about renting a vps or virtual machine for a day or two. Just be sure they allow you to bring in a disk for the initial transfer and they are aware of your bandwidth needs.

You might also just ask if they would simply help you with the upload.. They often tend to be opportunistic, inventive, and might be willing to accommodate you for the chance at getting some of your business.

Improve this answer
3
  • 3
    There's also a remote data-center option: Amazon AWS! Ship them hard drives, and they charge (as of this writing) $80 per device, plus $2.50 per data-loading-hour, plus whatever it costs to store the data as long as it's in AWS plus data transfer out. Possibly not the most attractive option, but just throwing it out there.
    – user14059
    Mar 16, 2012 at 1:10
  • This is great to know. We use AWS quite a lot! Do you know where I can read about this service? It's hard to find on google (at least the key words I'm using)
    – Brian Webster
    Mar 16, 2012 at 1:45
  • 1
    aws.amazon.com/importexport
    – Rich
    Mar 16, 2012 at 8:07
11

You could if you have friends or employees willing to lend you a hand with their home connection, set up a torrent for the large file and set up your employees/friends as seeds. Of course you'd have to cart and copy the file to everyone. Even then, you could split the file into .rars and have "teams" send portions (taking care to make sure they don't download the files they don't have). Crowd sourcing ftw!

Improve this answer
1
  • 3
    This would be particularly effective if you have friends who are in college. Not only do they have access to nice high speed university connections, but it wouldn't be hard for them to ask a couple of friends in different dorms to help with the upload.
    – Lèse majesté
    Mar 16, 2012 at 7:36
7

Find a regional ISP that's locally owned and bring a nice fat external drive that can hold all this. Walk into their office with $300 in cash, credentials for your server, and a 12-pack of beer. Pray that they have a gigabit uplink because a maxed-out 100mbit will take you over 24 hours.

That failing, look for local internet internet exchange point and find any AS that's connected to it. Make really good friends with somebody local from NANOG. Odds are you'll need to be creative. Try looking for any co-location near you.

Improve this answer
2
  • 2
    This is the path I would take. Beer tends to lubricate business interactions.
    – Evik James
    Mar 15, 2012 at 23:09
  • 1
    I think this is quite possibly the best path-of-least-resistance solution here.
    – jdd
    Mar 19, 2012 at 8:39
5

If the files are text files or otherwise cmpress well you can zip them up before uploading which could reduce the file size by up to 90%. Then you can decompress them on the server.

You can also try splitting the uploading between multiple locations. Each one then can give you their maximum speed.

Improve this answer
4

Get a temporary hosting company you can mail your drive to, and then transfer from the temporary host to the permanent host at a faster rate.

Improve this answer
2
  • 3
    Or just switch to a web host that does accept drives by mail. Most web hosts with decent customer service will allow this.
    – Lèse majesté
    Mar 15, 2012 at 21:14
  • I assume the current hosting company was selected because of price or some other long-term criteria, and it would be more costly to change on a permanent basis.
    – xpda
    Mar 17, 2012 at 19:11
4

These guys are in Tulsa, which you have in your profile so if you're in the area call them or stop by and let them know your situation.

http://www.datastorageinc.com/

Improve this answer
3

I just want to add to the existing answers something that might improve your transfer once you've got your hard drive connected to a server with good bandwidth.

I needed to transfer large amounts of data several times between two web servers and instead of using common FTP or WGET commands to upload or download the data, I have been using the Linux SCP command. You can find more info here: http://www.go2linux.org/scp-linux-command-line-copy-files-over-ssh

You will need SSH access to both servers and then you can copy one file, or many files recursively.

So, if you only have large files instead of many small ones, I think it will copy fast enough.

At a transfer rate of 11MB/second between servers the transfer would last about 25 hours.

Good luck! ;)

Improve this answer
1
  • 1
    Better yet, use rsync if the target host supports it. (Many do.)
    – Ilmari Karonen
    Mar 15, 2012 at 21:46
3

http://aws.amazon.com/importexport/

Ship your device to AWS and they will take care of the rest.

Improve this answer
3
  • "...and they will take care of the rest"--no, they won't, not unless your destination is AWS. If it's not, then they will only transfer the data to an S3 bucket. And you're the one who has to get it from the S3 bucket to your storage destination, in which case it's the same as xpda's answer.
    – Lèse majesté
    Mar 16, 2012 at 8:56
  • 2
    Oh absolutely, I should have commented as Rich did (Wasn't there by the time I had posted) on InChargeOfIt answer. Since Hamlin11 especially showed an interest in AWS solution and was looking for the link for this service I posted it. A comment over there would have been more appropriate I agree.
    – subiet
    Mar 16, 2012 at 9:05
  • Well, I think if you can edit the answer to tie in to that comment, and note that for non-AWS-destinations, you just need to initiate a server-to-server transfer afterwards using Amazon's very fat pipe, it'd still be a good answer.
    – Lèse majesté
    Mar 16, 2012 at 10:23
1

If the data is in your office on the slow connection you are in a pickle. If the data is on or at your old host you could try a server to server transfer. Using the ISP pipes instead of your tight office pipe.

If you did it in chunks via a wget from ssh it would be a hell of a lot faster than shipping a drive for install. That said, I have never had to move more than 250GB.

Improve this answer
1
  • The urgency portion of the question came from a catastrophic downtime event at the "old host". We didn't have a warm-backup host, but we did have recent backups locally.
    – Brian Webster
    Mar 15, 2012 at 23:15
0

As the other questions suggest, it mostly boils down to find someone around you with a good connection.

One group you could talk to are university or college admins. Just make sure, that the guy you're talking to is authorized to let you pipe some TB through their wire, and I'm sure you can get to an agreement.

Many faculties at universities, e.g. Particle Physics, post petabytes of data around, so this might be another possible option.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.