Forgot your password?
typodupeerror
Data Storage Networking IT

Never Underestimate the Bandwidth of a Suburban Filled With MicroSD Cards 208

Posted by timothy
from the eggs-in-one-basket dept.
toygeek writes "If you've been in IT long enough, you're bound to have heard the phrase 'Never underestimate the bandwidth of a station wagon filled with backup tapes.' These days moving data has become so much easier; We've surpassed baud rates and are into Gbps fiber on the backbones, and even in some homes. So, what's the modern equivalent to this, and what does it take to make the OC fiber connections cringe? Follow along as we theoretically stuff MicroSD cards into a Chevy Suburban and see what happens, and take sneakernet to a whole new level."
This discussion has been archived. No new comments can be posted.

Never Underestimate the Bandwidth of a Suburban Filled With MicroSD Cards

Comments Filter:
  • Jet full of CDs (Score:5, Interesting)

    by YttriumOxide (837412) <[yttriumox] [at] [gmail.com]> on Saturday September 21, 2013 @11:18AM (#44911703) Homepage Journal

    In my high-school days, we talked about a 747 full of CDs...

    I think it may have something to do with growing up on an isolated island nation... there's not many useful places a station-wagon will go.

    • Re:Jet full of CDs (Score:5, Interesting)

      by JavaBear (9872) on Saturday September 21, 2013 @12:27PM (#44912049)

      I do recall a story about Maersk moving their data centre from the US to Denmark, using one of their planes as a carrier, filled with harddisks.

    • by rossdee (243626)

      How about a rollon-rolloff cargo ship? Or a supertanker? The bulk of the worlds commodities are still transported by sea (even to and from the most isolated of nations like NZ.
      The limitation of a 747 would be the weight it could carry, rather than volume (microSD cards sink so they are heavier than water.

      Do you think Dean Barker can finnally win a race this afternoon (I guess its morning over there in Godzone)

      • by dbIII (701233)

        The limitation of a 747 would be the weight it could carry

        I was a bit surprised a while back to find out that 747s used to have up to a ton and a half of depleted uranium in the tail just as a counterweight. Even though there are bigger planes and they just don't compare to ships they can still carry a lot.
        How many TB/ton can microSD cards fit anyway :)

    • by nurb432 (527695)

      .... CDs....

      Kids these days.

      • .... CDs....

        Kids these days.

        I was in high-school in the 90s... I'm 34 now. Not yet old, but hardly a kid!

  • This is pointless (Score:5, Informative)

    by Anonymous Coward on Saturday September 21, 2013 @11:20AM (#44911709)

    The original saying was coined in a time where reading from a tape *was considerably faster* than reading over a network. Hence, transferring data via sneakernet was quicker, inclusive of the read-write times.

    Now, with multi-gigabit pipes making up the networks, data can be written, pushed, and read again, all at much higher bitrates than reading any storage medium. It's the read-write to physical medium that are the bottleneck with the sneakernet now.

    • by jc42 (318812) on Saturday September 21, 2013 @11:33AM (#44911785) Homepage Journal

      It's the read-write to physical medium that are the bottleneck with the sneakernet now.

      Yeah, but it's competing with high-speed networks that are crippled by the ISPs at both ends using a single fibre to feed an entire neighborhood, and intentionally slowing the speed at the customer's site to a crawl unless you pay an exorbitant rate for a higher speed (which is then unused 99% of the time, and doesn't deliver if 2 or 3 others in your neighborhood are using high speed at the same time).

      It's not surprising that vehicle+SD card could outperform such a network. The ping times can be rather long, though.

      • by HockeyPuck (141947) on Saturday September 21, 2013 @12:08PM (#44911939)

        Yeah, but it's competing with high-speed networks that are crippled by the ISPs at both ends using a single fibre to feed an entire neighborhood, and intentionally slowing the speed at the customer's site to a crawl unless you pay an exorbitant rate for a higher speed (which is then unused 99% of the time, and doesn't deliver if 2 or 3 others in your neighborhood are using high speed at the same time).

        Last I checked an OC-768, as referenced in the article, isn't going to be crippled by the ISP.

        Maybe it's time for you to realize that your uber-ultimate-epic-extreme bandwidth package from your ISP isn't really that fast compared to say what's in the article...

        • by Bengie (1121981)
          Not to mention that an OC-768 is kind of an old standard in the days of relatively cheap 100gb connections and even 8tb/s over a single fiber.
      • Re: (Score:2, Interesting)

        by Zero__Kelvin (151819)
        vehicle+SD cards can't out-perform even a slow connection. They "conveniently" ignored reality to arrive at their numbers. Not only do all those SD cards have to be filled with data prior to transmit and have the data pulled off on the other end, they would have to be done so in a specific order. There would at the very least have to be data duplication and error correction. They didn't calculate that overhead either. He should have stuck with tape, which has much better bit density and read/write spe
        • by sjames (1099)

          And you're ignoring that the cards can be written and read in parallel with a simple metadata header to allow re-ordering at the receiving end.

          • I'm not ignoring that at all. Go ahead and write up a proposal for the specialized hardware as well as the software spec, and the budget for developing the system you allude to, and get back to me when you have the system designed, implemented and tested. I'll let you know how much data I have transferred over the net during that time and we can compare throughput and cost.
            • by sjames (1099)

              Why in the world should I do all that for a simple thought exercise? Just to make you feel better?

              If I had an actual need, I would actually do the work. But if you like, you can go ahead and do a formal estimation of how long it might take me to do that, then rent yourself an OC-768 and see how much data you can actually cram down it. Don't forget to count the time it takes to get the contract negotiated and to get the people actually onsite to terminate the connection.

              We'll be waiting for your report.

              • "Why in the world should I do all that for a simple thought exercise? Just to make you feel better?"

                To participate in a thought exercise you have to put actual thought into it.

                "Don't forget to count the time it takes to get the contract negotiated and to get the people actually onsite to terminate the connection."

                You don't seem to understand the difference between a capability that actually exists currently and one that doesn't. By your logic anything you can imagine is already possible. OC-768 already ex

                • by sjames (1099)

                  You seem to want me to actually build one (since you demanded testing), so I want you to actually build one. Now MUSH!

          • by BlueStrat (756137)

            And you're ignoring that the cards can be written and read in parallel with a simple metadata header to allow re-ordering at the receiving end.

            Are you including the time required for the NSA to make their copies?

            What, you actually thought that ICE/TSA/DHS/Border Patrol roadside checkpoints were looking for illegal aliens?

            Strat

        • by Baloroth (2370816)

          vehicle+SD cards can't out-perform even a slow connection.

          For some non-arbitrary definition of "slow connection." It absolutely could outperform 56.6k dial-up (which still exists and definitely counts as a slow connection).

          Although I'm not sure why TFA chose SD cards, those are pretty much the worst means of bulk data transfer.

    • Re:This is pointless (Score:4, Interesting)

      by Waffle Iron (339739) on Saturday September 21, 2013 @11:35AM (#44911789)

      Now, with multi-gigabit pipes making up the networks, data can be written, pushed, and read again, all at much higher bitrates than reading any storage medium. It's the read-write to physical medium that are the bottleneck with the sneakernet now.

      TFA says that they have 19 million SD cards. If each one is a mid-range 6 megabyte per second speed and we access them all in parallel, that gives 912 terabit per second potential max bandwidth, which almost certainly exceeds any network you're thinking about.

      • Re:This is pointless (Score:5, Informative)

        by mooingyak (720677) on Saturday September 21, 2013 @12:01PM (#44911907)

        Now, with multi-gigabit pipes making up the networks, data can be written, pushed, and read again, all at much higher bitrates than reading any storage medium. It's the read-write to physical medium that are the bottleneck with the sneakernet now.

        TFA says that they have 19 million SD cards. If each one is a mid-range 6 megabyte per second speed and we access them all in parallel, that gives 912 terabit per second potential max bandwidth, which almost certainly exceeds any network you're thinking about.

        19M card readers (or slots or whatever) probably isn't even necessary.. I'm sure working in rotation you'd have a steady stream of cards whose data was fully read before other cards are even unpacked. I believe the optimal number of card readers would be (time to read) / (time to unpack).

    • by jedidiah (1196)

      > The original saying was coined in a time where reading from a tape *was considerably faster* than reading over a network.

      Reading from tape is still faster than reading from a network.

      Locally attached storage is always going to be faster. For large amounts of data, it will likely always be the case that a courier with a hard drive will move the data faster than a network.

    • I do not know about that. You can hook an eternal storage medium up to USB 3 or SATA whatever and get internal hard-drive range speeds.

      • "You can hook an eternal storage medium up to USB 3 or SATA"

        Buh! don't drink the cool-aid from storage vendors... their MTBF are not *that* high.

    • Re:This is pointless (Score:5, Informative)

      by bored (40072) on Saturday September 21, 2013 @02:14PM (#44912645)

      That is true for something like SD, but modern tape drives [oracle.com] can do well over 500MB/sec in compressed streaming mode, and have native uncompressed capacities of 8.5TB.

      That means that even with a midrange tape library with 56 drives, your talking a read/write bandwidth of 27GB/sec (aka 1/3 Tbit/sec). Tape bandwidth scales linearly with the number of drives in the library, and things like the the SL8500 from STK can support up to 640 drives.

      It still faster if you have a PB of data you need shipped from NY to LA to write it to tape, put it on a plane and read it back in LA. Plus, it all fits nicely into a big suitcase. Furthermore, even for smaller amounts of data (say 10-20TB a day) the cost of a tape drive and an next day delivery is going to be significantly less than the Gbit/sec or so of bandwidth required to ship a similar amount of data in most places in the US.

    • by Darinbob (1142669)

      The bottle neck would also be to fill up those microSD cards and putting them in the suburban then reading them all back again. You'd need many computers writing the SDs in parallel with a team of interns collecting them as fast as they fill up.

      Might be simpler to just buy the same number of hard drives as you're trying to copy, copy over onto them (faster than the external USB speeds for sure), then transport the hard drives instead of the microSDs. The advantage is that they store much more easily, they

    • by rtb61 (674572)

      The big reason for the comparison has always been cost. Comparing the cost of shifting that data via the internet, versus the cost of putting it on storage and flying with it to the destination. The argument being we are getting seriously ripped off with data transmission charges across the board when it is cheaper to carry data via sneaker net.

      Of course that doesn't even touch the best function of a 64GB micros SD card, boot from SD card rather than built in storage. 64 GB all in something the size of a

    • The original saying was coined in a time where reading from a tape *was considerably faster* than reading over a network

      Which is still true between many locations on the planet. The big change is that instead of tapes being used for large volumes you see hard drives being posted around instead. Uploading 2TB via ADSL2 with an upchannel of 256kbps or so would suck immensely, and 10Mbps is still making sending the thing by courier a much more attractive idea.

  • by jlf278 (1022347) on Saturday September 21, 2013 @11:25AM (#44911725)
    From the article: "A MicroSD card is only .1 cubic inches, so if all things were equal you could stuff 100 64gb cards into a cubic inch of space! But, that does not seem realistic. In fact it doesn't even seem remotely possible." Perhaps that's because 1 cubic inch = 10 * 0.1 cubic inches and not 100 * 0.1 cubic inches.
  • XKCD (Score:5, Informative)

    by Anonymous Coward on Saturday September 21, 2013 @11:26AM (#44911733)

    http://what-if.xkcd.com/31/
    That is all

    • http://what-if.xkcd.com/31/ That is all

      Thing is, they are only talking about the internet traffic, not the total data available on the internet. For example, abandoned blogs. There must be loads of blogs that people have started, then abandoned because nobody was reading them, or they moved to facebook, or whatever. Photo sites, parts databases, etc, There must be terabytes of stuff that is never, or rarely accessed. And that's only web pages. What about FTP sites, and others?

      On the other hand, there will be a lot of duplication of data

  • by queazocotal (915608) on Saturday September 21, 2013 @11:26AM (#44911737)

    Swallowing capsules containing a terabyte (about 12mm in diameter, and 15mm long of microSDs) is quite plausible.
    You can easily swallow a hundred of these, and it'll come out over the next 2 days.
    100TB/2 days = 600 megabytes a second.

    • by Anonymous Coward on Saturday September 21, 2013 @11:37AM (#44911801)

      Something something memory dump.

    • Re: (Score:2, Insightful)

      by EmagGeek (574360)

      If you're trying to smuggle your own data across the border without it being copied by the NSA (as is routine for entry into the US for CBP to confiscate and make an image of your laptop HDD for NSA), swallowing a MicroSD is not so implausible or impractical.

      • by phayes (202222) on Saturday September 21, 2013 @12:17PM (#44911997) Homepage

        Yeah, right. Border authorities couldn't possibly detain someone on suspicion of smuggling data internally & wait for it to come out like they already do for drug mules. No, no, it could never happen...

        It'd be nice if you wouldn't demean words like "routine" into meaninglessness and stop confusing the NSA with the US customs authorities. Yeah we've all heard of incidents where laptops & drives have been confiscated but this is not a routine occurrence -- If you want to claim otherwise deliver a reliable reference giving the total number of incidents per year (you're the one claiming it's "routine" so it's on you to justify your claim). I'll divide that by the number of border crossings & we'll all have a benchmark on how often it happens per border crossing & just how "routine" it is.

        • If you want to claim otherwise deliver a reliable reference giving the total number of incidents per year

          You can argue frequency all you want, but so long as the unconstitutional searches are > 1 *and* well-publicized, the panopticon effect becomes real.

        • by Kaenneth (82978)

          That's why you phsyically carry the random one-time-pad with you; while sending the encrypted data via a different route.

    • Johnny Mnemonic but now days it will need to be like 160TB-320TB or even 160PB to be really big.

    • by davidbrit2 (775091) on Saturday September 21, 2013 @11:48AM (#44911849) Homepage
      I've got Crohn's disease, so my ping times are faster.
  • by Ygorl (688307) on Saturday September 21, 2013 @11:29AM (#44911749)
    As they point out in the article, the tremendous bandwidth achieved does not include the logistics or time required to initially copy the data onto SD cards, and then back off of the cards upon reaching the destination. Still, beats a flock of parrots trained in Morse code.
  • by sanosuke001 (640243) on Saturday September 21, 2013 @11:29AM (#44911755)
    I did something similar after a conversation at work using a 53' Semi Trailer and 4TB HDDs.

    Semi = 630" x 94" x 102"
    HDD = 4" x 1" x 5.75"
    Total HDDs = 262628
    Total Storage = 1050512 TB
    Bandwidth from NYC to SF = 55.58 Tbps (42 hours according to google maps)
  • by kannibal_klown (531544) on Saturday September 21, 2013 @11:30AM (#44911761)

    Great, you've managed to transfer X terrabytes of data across state lines. I salute you.

    Now... go find this list of files that I need. Also that collection of data that our servers need to process. What, that's going to take you HOW long?

    Yes might be quicker to send the data from point A to point B by just shipping disks... but only for archiving purposes. If you actually need to access the data, then you still have that last mile (or 10') of having to load that data into a system or network.

    Now, if you were shipping a truck load of servers, or maybe a car full of NAS devices that you could just plug into a network and be done with it... then it's not too bad. Then you can start electronically searching the data within minutes, index it, use it for your data-store, whatever.

    But just a car load of disks for non-archival purposes... you're asking for a headache.

    • What if added the stipulation that the data must be transferred in a single filesystem. This thing:

      http://www.synology.com/products/product.php?product_name=DS2413%2B [synology.com]

      ...scales up to 96 TB w/ an expansion device and easily fits in a suburban. Using their estimate for travel time from NY to LA that comes to ~0.66 GB/s or ~6.6 Gbps. Fiber beats that by a factor of six. However, since they're standard NAS devices, you should be able to access them to read the data w/o much trouble, and you could probab
    • by jedidiah (1196)

      Use an array enclosure and plug it directly into the system that needs to use it. 8-bay arrays are cheap and plentiful already. You don't have to get into anything "expensive and fancy".

      Even a "car full of disks" is doable if you already have them set up to be plugged in at the destination. Many array enclosures have been built with this in mind for decades now.

    • "Yes might be quicker to send the data from point A to point B by just shipping disks... but only for archiving purposes."

      I can tell you I recently have had to resort to move hard drives by courier to move virtual machine images between two datacenters... for a telco, no less!

      So you can bet is an option, not only for archiving.

  • by Ignacio (1465)

    This is neither news nor does it matter. You should consider changing the site's tagline to "We'll post anything!".

  • they forgot the time it takes to transfer data onto the SD cards, remove them, and fill the truck. add another 24 hours

    • Exactly. For most scenarios a high-bandwidth link is not that useful if the latency goes to hell.

      Even for my normal internet usage, I'd rather take a 5Mb/s connection with 2ms latency (to ISP's default GW) than a 100Mb/s connection with 20ms latency.

      • by dbIII (701233)

        Exactly. For most scenarios a high-bandwidth link is not that useful if the latency goes to hell.

        No, since most scenarios will be "finish by time X" and not "start at time X".

  • For real-time gaming, this would be awful. Well, except compared to the average American ISP.

  • FTA:

    These days you can't really get a good sturdy station wagon, but the modern equivalent seems like it would be the SUV. Since Chevrolet Suburbans have been around for so long, I'm going to pick that.

    Subaru still makes the Legacy station wagon which is sold in the US as the gussied up Outback. That would be a fairer vehicle for comparison. It has 71.3 cu. ft. of space with the seats down vs. the stated 137 cu. ft. of a Suburban so the end result needs to be scaled by 0.52. The stated 68057Gbps bit rate should really be 35390Gbps in a fair evaluation.

  • Petabyte tablet (Score:4, Informative)

    by Ken_g6 (775014) on Saturday September 21, 2013 @11:57AM (#44911889) Homepage

    In 3001: The Final Odyssey, Arthur C. Clarke wrote about a petabyte tablet as though it were the ultimate storage medium, something humanity finally arrived at after a millennium. In the book it was enough to store the contents of a human brain!

    16,000 or so microSD cards could store a petabyte in roughly a 1-foot by 1-foot by 2-inch space, probably leaving enough room to wire them up as well. Of course, it would cost nearly a million bucks, not counting the hardware necessary to wire it up to be accessed. But, still, I find that very impressive.

    • by wjcofkc (964165)
      I remember reading Vernor Vinge's A Fire Upon The Deep. There is one scene where a download is taking a place at several megabytes a second. A great deal of dramatic effect was put on "megabytes". The gigabytes of data that resulted from the download was also given an impossibly dramatic effect.
      • Re: (Score:2, Interesting)

        by Anonymous Coward

        If that's the scene I'm thinking about, they were trying to pull incredibly valuable data off an ancient computer on a dead planet. Considering its condition, that kind of file recovery speed would make me happy, too.

      • by dbIII (701233)
        "All this technology and it still takes forever to get anywhere."
    • In the book it was enough to store the contents of a human brain! ...
      Of course, it would cost nearly a million bucks

      Let's see, if we run Moore's Law out until this costs $500, that's .... the end of 2029. Curiously enough, exactly the year that Kurzweil has been predicting the singularity for over a decade.

  • by Arancaytar (966377) <arancaytar.ilyaran@gmail.com> on Saturday September 21, 2013 @12:01PM (#44911905) Homepage

    Assuming 2500kg for the car, 0.0005kg per cards, $50000 for the car and $50 per card:

    Not only a bandwidth of 68Tbps, but a mass of 12 metric tons (4.8 empty cars).

    Also a market value of $0.96B, or the equivalent of 19,141 new cars, plus one car with a broken suspension.

  • What about round trip latency??  How long does it take me to get my 10GB Full HD video after I've clicked??
  • I first heard it as "truckload of CD-ROMs".

  • From the article:

    A MicroSD card is only .1 cubic inches, so if all things were equal you could stuff 100 64gb cards into a cubic inch of space!

    Wouldn't that be 10? You know, with the decimal and all that.

    Seriously, off by a factor of 10 is still a lot, no matter what scale you're talking about. Shouldn't someone check this stuff?

    Beyond that, the time to read those things has got to be enormous, MicroSD can only be read at 104 MB/s, how many cards do you need to read in parallel to match a decent backbone l

    • Try doing a DC Promo of a very, very large Active Directory (think Air Force large) to an island with a satellite link of 512K then tell me the SneakerNet is dead.

      Yeah we did the DC Promo on a machine here then sent the disks via the next plane. I think that qualifies as a SneakerNet for today's kids.

  • We can overnight a padded envelope full of 32GB drives anywhere in the country. That's hard to beat when you need to send a few dozen gigabytes in 12 hours.

  • As I read this, I'm on a trip to Houston to retrieve several terabytes of data for Clonebox. At 90 minutes each way, if I bring back 8TB, that's what, 60 Gbps. There's no way I could transfer that data over my cable modem.

  • Short of an actual man in the middle attack, with rifles and warrants, your information doesn't get mirrored to the NSA's data center. Being able to see your packet in transit means security.

    • by rusty0101 (565565)

      The problem with sneakernet solutions is rarely the read/write time at either end. In almost all cases if you're organization is large enough to make use of high capacity circuits, you probably can't control the end-to-end security of your sneakernet for the data transfer any more.

      You may use bonded couriors, but you very likely can't controll such things as bridge failures (rare, but anyone else wonder how much was paid for the cargo in the trucks that were on the I-35W bridge that collapsed August 1, 2007

  • A few years ago, I was involved in the conversion of the Stanford AI lab tape archive to modern media. This involved reading thousands of reels of 1/2" magnetic tape. It was a slow process. Volunteers were loading a tape onto a tape drive every 15 minutes for weeks. After each tape was loaded, its contents were sent over an Internet connection in under a minute. It took much longer to wind through the tapes than to transmit the data.

    The data went to a server farm at IBM Almaden Research, where the file s

    • by bored (40072)

      A few years ago, I was involved in the conversion of the Stanford AI lab tape archive to modern media. This involved reading thousands of reels of 1/2" magnetic tape. It was a slow process. Volunteers were loading a tape onto a tape drive every 15 minutes for weeks.

      Tape has changed a lot since the early 80's. Modern tape drives [oracle.com] can sustain over 500MB/sec compressed each. And they tend to live in automated tape libraries [oracle.com] that load and unload the tapes without human intervention. A library like that can do T

  • I think sneaker-net is funny phrase from the days of 10baseT networking being slower than just copying to another media and walking it over. Also used when the stupid networking just wouldn't work...

  • Tape drives like T10kD and LTO6 have sustained transfer rates that are individually larger than single (non SSD) hard drives. Combined into tape libraries, they are capable of outrunning the fastest RAID arrays on the market.

    That of course assumes you are using them as a backup/archive medium and streaming data from a RAID to the tape drive rather than trying to use them in random access mode.

  • Otherwise you don't have a chance reading them in the right order when restoring the backup.

  • http://www.dansdata.com/gz105.htm [dansdata.com]

    The capacity of MicroSD cards has improved a bit since then, resulting in a moderate increase in achievable bandwidth; but other than that the analysis is still essentially the same.

  • Did no one else immediately think of the weight as soon as the author started talking about filling an SUV with microSD cards? I'm reminded of the saying '100lbs of pillows/feathers is still 100lbs', in reference to how people seem to overlook that very light objects are still heavy if you carry enough of them.

    While the exact weight of each of the 19 million microSD card would vary a nice starting point is about 0.4 grams plus or minus 0.1 based on general specs. That's well over 16,000 lbs or 8 tons of
  • You cannot eavesdrop on a station wagon full of SD or even micro-SD cards or sneaker net in general.
  • You'd get a very large amount of out-of-sequence deliveries and packet loss. Better get a Toyota Hi-Lux or something like that if you want your data to actually arrive in a predictable amount of time.
  • A micro SD card weighs about 0.25 grams, so their calculated 19,141,092 micro-SDs weigh in at 4785 kg. The maximum load of a Chevy Suburban is 2561 pounds (1161kg). Assuming you have a 75kg driver, that lonly leaves 1086 kg for micro-SD cards. So you can only carry about 4,346,596 micro SD cards, less than a quarter of what the authors estimated. The bit capacity is 278 petabytes.

  • ...and a play ticket to Hong Kong (with transfer to Moscow).

Programmers do it bit by bit.

Working...