Mega-Uploads: The Cloud's Unspoken Hurdle 134
First time accepted submitter n7ytd writes "The Register has a piece today about overcoming one of the biggest challenges to migrating to cloud-based storage: how to get all that data onto the service provider's disks. With all of the enterprisey interweb solutions available, the oldest answer is still the right one: ship them your disks. Remember: 'Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.'"
Pro photography is a huge problem (Score:5, Interesting)
Returning from a site with a tethered computer full of 80 MP 16-bit raw files from a day's worth of shooting would break most bandwidth bills if you tried uploading all these images.
Re:Pro photography is a huge problem (Score:5, Interesting)
Realistically, though, if I want to I can just upload them all to home or a cloud storage in batches overnight, the same way I download 10 gigabyte files at home. It's just plain easier to cart em around, though.
Re:Bandwidth of a station wagon (Score:5, Interesting)
Indeed. He also ignored the core reason for having said bandwidth - you have X amount of data to move in Y time (at under Z cost); what's the best way to do so?
As such, a 'packet' on the freeway system is rather expensive, so you don't want to be putting multiple station wagons on the system if you don't have to. Figure the driver costs $20/hour, the vehicle itself $.50/mile(gas, maintenance, insurance, tolls, etc...), and you're looking at 300 miles in 10 hours. For a single packet you're looking at $350 for that single 'packet'. If a single station wagon doesn't do it, perhaps a cargo van would, which doubles the capacity of the packet while only raising the cost $50, to $400. Still not good enough? Upgrade to a 'package van' like UPS/Fedex trucks. Next step would be a Semi.
In any case, I'd say that you could fit 25TB into a motorcycle today - 3 TB drives are fairly common now, and I can fit 10 into my saddlebags easily. Heck, I can get 1.5TB native tapes [wikipedia.org], about the same size as a HD. Padding it's dimensions up, it's 11 x 11 x 3 cm = 363 cm^3, or 2,755 per cubic meter.
A 2008-11 Dodge Grand Caravan Cargo van [allpar.com] - 143.8 cubic feet = 4.07 cubic meters, giving me room for 11k 1.5TB tapes. 16.5k TB, in 10 hours, if I have a single cargo van. Ouch. Disregarding media cost, that's ~$400.
Do this daily, we're looking at 1.5 terrabits per second. Don't know of any connections that fast.
Monthly, we're down to ~50 gigabit (rounding down). I can guarantee that a 50 gigabit connection will cost more than $400.
Annually, it's 'only' 4 gigabit, and I pay more than $100/month for my megabit class connection, which ISN'T utilized 100%, unlike my calc.
You don't normally need to figure out the bandwidth of the freeway because:
1. Generally 1 vehicle 'packet' is sufficient, and due to the high marginal cost per said vehicle, you normally only want to send one.
2. The roads are used for more than data shipment, which would be like trying to figure out how much bandwidth you have available for VOIP by looking at total circuit bandwidth.
Don't need to ship that much? You should be able to ship about 30 of them for $60, second day air. That's 45TB, or about 140 Mbit of 100% saturated traffic for a month. BTW, during my calcs for paying fedex to ship them, I think that weight might actually be enough of an issue to increase gasoline consumption - but I think I've established that even $800 would be cheap if you need to ship that ridiculous of an amount of data.
Aspera and Friends (Score:4, Interesting)
You you always use a UDP solution such as Aspera [asperasoft.com]. Fast transfer speeds, bandwidth management and they have a specific AWS implimentation. [asperasoft.com]
Other options to look at include Smartjog [smartjog.com], whose new Bolt product looks quite interesting, Riverbed's Steelhead [riverbed.com] product, Filecatalyst [filecatalyst.com] and Signiant [signiant.com].
There are many solutions around now to deal with large file transfers for both small and large business. Most of them use UDP instead of TCP/IP, with Checksums to ensure all data is reliable delivered. Even with just 1Mbps upload speeds, something like one of the above named products will be advantageous. I've worked in the media industry for a number of years, and this type of thing is being used in Film and Television all the time. Of course, there are still tapes being shipped around, but in emerging markets, such as Russia for instance, the file transfer really beats a tape being stuck in customs for weeks or months.
Canada (Score:4, Interesting)