Forgot your password?
typodupeerror
Cloud Businesses Data Storage IT

Mega-Uploads: The Cloud's Unspoken Hurdle 134

Posted by samzenpus
from the mountain-of-disks dept.
First time accepted submitter n7ytd writes "The Register has a piece today about overcoming one of the biggest challenges to migrating to cloud-based storage: how to get all that data onto the service provider's disks. With all of the enterprisey interweb solutions available, the oldest answer is still the right one: ship them your disks. Remember: 'Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.'"
This discussion has been archived. No new comments can be posted.

Mega-Uploads: The Cloud's Unspoken Hurdle

Comments Filter:
  • by Anonymous Coward on Monday May 21, 2012 @04:29PM (#40069091)

    Returning from a site with a tethered computer full of 80 MP 16-bit raw files from a day's worth of shooting would break most bandwidth bills if you tried uploading all these images.

  • by Eponymous Hero (2090636) on Monday May 21, 2012 @04:32PM (#40069139)

    Pressed if disks are accepted, the company responded that “All common database products provide a capability to extract to a common file format like .csv.”

    what a professional answer. and by that i mean it didn't answer the question at all.

    • by Bogtha (906264) on Monday May 21, 2012 @05:25PM (#40069803)

      You don't know the exact dialogue between the journalist and the rep. I've been quoted in print in similarly stupid ways when what I said made absolute sense in context to what was asked. "Pressed if disks are accepted" could have been something like the rep telling them about a new CSV import tool they had built, the journalist saying "So if I mailed you a 5TB database on a disk, could you import that?", and the rep replying "Sure, but you'd need to export the data first...".

    • Didn't he? I recognize (because I have to do this too) a bit of the old 'no, you can't do that..or we have a workaround for it..and I will obliquely say that, but not come right out and do so....because you haven't signed a contract yet' (I admit to not having RTFA).
  • by viking099 (70446) on Monday May 21, 2012 @04:35PM (#40069177)

    Remember: 'Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.'"

    Yeah, the bandwidth is great, but the latency SUCKS.

    • by MrP- (45616) <rob@el i t e m r p.net> on Monday May 21, 2012 @04:37PM (#40069217) Homepage

      A simple solution would be to create station wagons with FTL engines.

      • Or at least a JATO unit

      • Add a flux capacitor to the station wagons.

        What is your transmission speed when the transmission is completed in -1 hour?

    • by ffejie (779512)
      I love this thinking. There was a thread about this some time back that I found most enjoyable, despite my shoddy math.

      A dumptruck full of harddrives. [slashdot.org]
    • True, but that's not even the real problem. The idea of creating an "initial seed" is a good one, but it's only dealing with one sign of a larger problem: our internet connections are often not good enough to deal with the volume of data we're pushing, and so cloud storage solutions can only serve certain cases.

      Storing files on a remote system with limited bandwidth is only good when you're dealing with generally small files. A limited number of large files can be fine so long as you're primarily syncing

      • by jedidiah (1196)

        No. The cloud is really irrelevant.

        Either I can't push it out there and thus can't use it.

        Or, I can push it out there but it is irrelevant because I could just host my own stuff to begin with.

        I either can't use it or don't need it.

        • Scenario 3 is that you can currently host but can forsee a time when you cannot.
          There's also when there's a ton of stuff going in and little going out. Contrived example: Survey 1,000,000 people and get back 10 aggregated results.
      • by jbolden (176878)

        Of course, that's not happening anytime soon, especially since the ISPs and media companies that they're partnered with have no interest in giving people decent upload rates.

        I think if you look at most ISP's over time you'll see that upload rates have gotten far more symmetrical than they used to be. I just for example got moved from 15/5 to 25/25 for the same cost.

        • Not me. My upload speed has actually gone down in recent years, and I have no choice in ISPs. Oh, and it's not like I live out in the middle of the country.
  • by WOOFYGOOFY (1334993) on Monday May 21, 2012 @04:35PM (#40069181)
    Second biggest challenge: trusting any of these places have the motivation to keep your data more secure than credit card companies do.
    • by gestalt_n_pepper (991155) on Monday May 21, 2012 @04:43PM (#40069297)

      No, that's third. The second biggest challenge is believing that those fine hosting companies with servers hosted in lower Slobbovia won't have a few entrepreneurial employees who will *actively* be searching your data for all that is monetizable.

      • A special case of my number two perhaps.

        I like your sig.

      • I thought the second biggest challenge is ensuring that the Empire does not raid the hosting company and render all your files inaccessible...
      • by PPH (736903)
        It has already been searched by Customs on its way out of the country (or the NSA if you pushed it out over the 'Net). And if it is of any interest to one of your competitors who happen to be buddies of the DHS, they'll be the ones contracted to do a 'threat analysis' on it.
    • Exit strategy (Score:4, Insightful)

      by scsirob (246572) on Monday May 21, 2012 @05:00PM (#40069507)

      No, the second biggest challenge is to come up with a viable exit strategy. Once you have several TB at this service provider, how will you move it out of there when the next provider has a better deal? That was one of the major big points for having a cloud in the first place, to have the freedom to move your compute requirements to a better, cheaper, faster (pick two) provider.

      Even if you moved it in with a station wagon full of tapes or disks and your provider let you import it, I'm sure your provider will not be so helpful when you need to move it back out.

      Blatant plug: Perhaps Actifio (www.actifio.com) can fix this for you, by replicating your data in, and also back out of production systems in deduped and compressed format.

      • by dj245 (732906)
        Even if you moved it in with a station wagon full of tapes or disks and your provider let you import it, I'm sure your provider will not be so helpful when you need to move it back out.

        Why not? You are paying for their services right? Even a particularly scummy company would be swayed by the request "my pre-production server crashed hard. Can you mail me the disks?". I am not familiar with all these companies, but Crashplan charges you a fee to put your data on a disk and mail it to you. Unless you
    • by AmiMoJo (196126)

      Send them Truecrypt containers, problem solved.

      • Yeah I think that is the way to go... for the extra paranoid, how do you know you can trust Truecrypt?

        http://superuser.com/questions/164162/is-truecrypt-truly-safe [superuser.com]

        I'm sure (well, actually not really) rumours are just FUD (by whom and for what purpose?) but that's the thing, you're through Alice's Looking Glass now... is it FUD disseminated by people who don't want use to use Truecrypt just because it is iuin fact unbreakable or is it someone who's hit on truly suspect facts about Truecrypt? How do y

  • Backups (Score:5, Informative)

    by SJHillman (1966756) on Monday May 21, 2012 @04:37PM (#40069227)

    My last employer offered offsite backups to clients. For the initial seed, we always tried to get them to put it on an external HDD and ship it to us (or at least DVDs). The only major exceptions were clients that were also on FiOS - that was the only case where over-the-net transfer was faster than the backup-and-ship-it method for the initial seed.

  • station wagon has low bandwidth, the tapes have to be written and read.
    • by h4rr4r (612664)

      140MB/s is low speed?

      That is the speed of LTO-5, at best.

      • by rubycodez (864176)
        LTO-5 doesn't do that over sustained time periods.
        • by jedidiah (1196)

          Then put it on a RAID array. I knew people that were doing this more than 10 years ago. They needed to move large amounts of data around. So they would FedEx a RAID array around.

        • by h4rr4r (612664)

          That depends on the drive and how fast you can send it data, more than the format. This is for linear reads and writes of course.

    • Random Access on tapes makes using your HDD as RAM seem like fun; but contemporary tape drives are actually pretty damn fast. It may be necessary to flog the tape minions on occasion, in order to spur them to greater effort; but that isn't tape's fault...
    • by Jeng (926980)

      That only depends on how many tape drives you utilize at once. If you are shipping 200lbs of tape, and using dozens of tape drives then you should have much better bandwidth than if you tried to send it over the internet.

    • by ksandom (718283)

      station wagon has low bandwidth, the tapes have to be written and read.

      No. Bandwidth is how much you can send in one go (tapes/hdds in a car are extremely high). Latency is fairly much how long it takes you to do it. Throughput brings these together.

    • by Animats (122034)

      True. I was at one point involved in converting the Stanford AI Lab archives from 6250BPI tape to a file server. People were loading tapes for weeks. As soon as a tape was loaded, the data went over the Internet to a file server at IBM Almaden for format conversion. The transmission of a tape only took a few seconds. Of course, both IBM Almaden and Stanford have major backbone connections.

  • by fuzzyfuzzyfungus (1223518) on Monday May 21, 2012 @04:50PM (#40069355) Journal
    I don't think that TFS's answer is necessarily the correct one. I'd really prefer to hold Ma Bell's feet to the fire concerning the fact that bandwidth(even in 'optimal' build-out areas, spare me the excuses about the boonies) has enjoyed a deeply underwhelming track record in terms of improvements in cost and quantity compared to most other aspects of contemporary computing.
  • Intercontinental company I used to work for, once or twice a year they'd send an intern over the Atlantic in the SST with a case of tapes.

    When it just positively had to be there asap...

  • by Surazal (729) on Monday May 21, 2012 @04:52PM (#40069385) Homepage Journal

    Yes, never underestimate the bandwidth of a station wagon full of disks hurtling down the highway. The latency, on the other hand, leaves much to be desired, and I've heard the packet loss can be downright fatal.

  • I got my first linux distribution (I don't remember if they were called distributions back then) shipped on tape to the campus computer lab where a group of us brought our computers to copy the files.

    • by couchslug (175151)

      I got mine in 1999 from Cheapbytes:

      http://www.cheapbytes.com/ [cheapbytes.com]

      Corel Linux FTW!

      • by Dogtanian (588974)

        I got mine in 1999 from Cheapbytes: http://www.cheapbytes.com/ [cheapbytes.com]

        Clicked on the link out of curiosity, got a splash screen that looked like it was designed in the 90s. (*) Anyway, clicked on the "Click here to enter the CheapBytes store" and I got...

        Great Success !
        Apache is working on your cPanel® and WHM Server

        If you can see this page, then the people who manage this server have installed cPanel and WebHost Manager (WHM)

        So are they still trading or is this just a zombie remnant? Guessing that their business would have shrunk quite a lot since the days when everyone was on dialup and you'd have had to be on crack to consider downloading even a CD's worth, let alone a DVD. (Used to order Linux discs quite a lot myself, haven't done it since I

        • Guessing that [a CD distributor's] business would have shrunk quite a lot since the days when everyone was on dialup and you'd have had to be on crack to consider downloading even a CD's worth, let alone a DVD.

          Shrunk? Yes, I'll grant. Still useful in places that can't get FTTH, DOCSIS, or DSL? Yes. Satellite and cellular are still capped to about one DVD a month, with single or dual layer depending on which plan you choose.

          • by jedidiah (1196)

            Cheapbytes didn't even really come around until broadband was commonplace. The fact that they are "cheap" is a reflection of that. Before then, you had more expensive CD sets.

            It's cheap for a reason.

      • by jedidiah (1196)

        I got mine in 1994 from Walnut Creek. 6 CDs. 4 Distributions.

    • I am much younger, I got my linux from a dead tree.

      • That is relevant because it was easier/faster to get an up-to date-ish linux cd with a book that had to go through various editors, a printing press and the book stores distribution system then download the cd over a 33.6 modem... and then it was easier to read the book then load /search a document that large in a word processor and stare at a low res CRT screen.

  • by petes_PoV (912422) on Monday May 21, 2012 @05:13PM (#40069655)

    ... is getting it all back OFF again when you want to switch service providers.

    The one thing you want never to happen is that you get locked in to a single cloud service. They might go bust, they might become uncompetitive. They may become politically "unfriendly" or tainted with customers you have no desire to be associated with - or any of a number of other reasons to say "adios".

    Just like with disaster planning, all the processes and procedures, agreements and SLAs are worthless until you've actually PERFORMED the operation and done so without a major service interruption. How many cloud users have gone that far - and how many are locked in but don't know it?

    • We've already seen the unsinkable cloud get sunk. Amazon's never-down cloud has rained out at least once in some regions. Another problem is the cloud provider making changes to their services that impact the way your company operates. My last company was in the process of Googleizing when I left a year ago and Google's already made changes that must have required training and documentation updates. And they can remove apps and services at any time so some unpopular product that happens to be very usefu

  • Ah yes, the TCP/USPS revolution has finally arrived!

  • Aspera and Friends (Score:4, Interesting)

    by PhillC (84728) on Monday May 21, 2012 @06:14PM (#40070375) Homepage Journal

    You you always use a UDP solution such as Aspera [asperasoft.com]. Fast transfer speeds, bandwidth management and they have a specific AWS implimentation. [asperasoft.com]

    Other options to look at include Smartjog [smartjog.com], whose new Bolt product looks quite interesting, Riverbed's Steelhead [riverbed.com] product, Filecatalyst [filecatalyst.com] and Signiant [signiant.com].

    There are many solutions around now to deal with large file transfers for both small and large business. Most of them use UDP instead of TCP/IP, with Checksums to ensure all data is reliable delivered. Even with just 1Mbps upload speeds, something like one of the above named products will be advantageous. I've worked in the media industry for a number of years, and this type of thing is being used in Film and Television all the time. Of course, there are still tapes being shipped around, but in emerging markets, such as Russia for instance, the file transfer really beats a tape being stuck in customs for weeks or months.

  • But then... (Score:5, Insightful)

    by TemperedAlchemist (2045966) on Monday May 21, 2012 @06:22PM (#40070477)

    How did you manage to fix armed FBI storming your servers located in another country problem?

    • by aiht (1017790)

      How did you manage to fix armed FBI storming your servers located in another country problem?

      Unless you're not in the US, and "another country" is the US - what the hell are the FBI doing there?

  • by dave562 (969951) on Monday May 21, 2012 @06:43PM (#40070707) Journal

    I am dealing with this as well, albeit on a different scale. About a year ago, the powers that be decided that they were going to develop a private cloud for the company. Nobody really considered how to migrate 500+TB of data from three separate sites into the new cloud. We are doing a mixture of over the wire replication (for sites with 100Mb+ of bandwidth), physical replication (using NAS devices and tape), and synchronization using DoubleTake for the SQL data and Vice Versa Pro for file system data. It is a massive undertaking, made even more difficult by the fact that we are working with production systems with locked in SLAs that need be maintained.

    For the average person, and even most enterprises, I honestly believe the best way to get into "the cloud" is by following a well planned out, phased approach. The first phase should be using the cloud as a DR target. Only when both sides of the equation are balanced and able to operate independently of each other can you consider doing away with one and moving to the other.

    • by ediron2 (246908)

      Actually, enterprise issues regarding data synchronization quickly make get problematic.

      Have just watched a migration from private mail servers to cloud-based email. Months in, it quickly became apparent that a few short days or even a few weeks of pain associated with migrating users cold turkey (and then importing requested data from Notes once it had become static) would have been astronomically less cost and pain compared to wiring the connector and having two frameworks alive (and borking the sync in

      • by spazdor (902907)

        we all can write the headline and article now for going the other direction the day some cloud provider implodes: Company Widgetcorp declared bankruptcy today. Their tragic fall from stalwart Rusell-2000 midcap manufacturer to receivership happened unexpectedly: Their cloud-based XXX provider shut off servers without warning less than 60 days ago, and Widgetcorp was never able to recover critical processes and data.

        Perhaps the "use the cloud as your DR target" model is a good one after all! I think a good w

  • Is around a gigabyte per second.

    (100 packs of 16*64GB microSDs, in appropriate packaging, swallowed at intervals over the course of a day)

  • .... how did you end up with all that data on the servers in your mom's basement in the first place.

    The 'Cloud' option needs to be a part of your system design in the first place. So you begin to accumulate all that data in The Cloud from the word go.

  • Canada (Score:4, Interesting)

    by DarwinSurvivor (1752106) on Monday May 21, 2012 @10:10PM (#40072189)
    In canada, unless you need low latency, the internet is about the most expensive method you could possibly use to transfer data. source [dslreports.com]
  • by isaac (2852) on Monday May 21, 2012 @11:20PM (#40072655)

    http://aws.amazon.com/importexport/ [amazon.com]

    http://awsimportexport.s3.amazonaws.com/aws-import-export-calculator.html [amazonaws.com]

    It's not rocket science. Yes, shipping drives is the cheapest, fastest option for a lot of people.

    YMMV, speaking for myself, not my employer, etc. etc.

    -Isaac

  • Namely, the increased risk that your data will become collateral damage in the War On Piracy.

"Out of register space (ugh)" -- vi

Working...