'Digital Universe' To Add 1.8 Zettabyte In 2011

1sockchuck writes "More than 1.8 zettabytes of information will be created and stored in 2011, according to the fifth IDC Digital Universe study. A key challenge is managing this data deluge (typified by the Large Hadron Collider at CERN, which generates 1 petabyte of data per second)."

LHC data is _not_ stored in the digital universe

[rbrausse]

the experiments may generate the PB per second but most of the data is rejected before it hits any storage system...

Re:LHC data is _not_ stored in the digital univers

[Infiniti2000]

Indeed, only about 25PB are stored every year from the LHC.

Re:

[AliasMarlowe]

Indeed, only about 25PB are stored every year from the LHC.

No. They store all of it, but mostly in /dev/null

Re:

[Joce640k]

Shouldn't it be "Pebibytes"? We're supposed to be geeks.

Re:

[Daimanta]

Shouldn't it be "Pebibytes"? We're supposed to be geeks.

Yes, but we're not morons.

Re:

[AliasMarlowe]

And as geeks we understand that the English language isn't governed by a committee of Swiss engineers.

Unfortunately, they're English engineers...

Re:

[GrumpySteen]

I suppose we should be grateful that data isn't measured in petahogsheads.

Re:

[the_other_chewey]

Yes, and as geeks we understand that the quoted number is an approximate
one and the data generated are not inherenly binary, so there is no need
for either the precision of "exactly one PiB" or the context of "this is binary".

In fact, the decimal prefix is the much more sensible one to use here.

1.8 Petabyte per second...

[LordofEntropy]

Wow that's a lot of data. Can't wait to see more of the results published.

Re:

[cashman73]

How many libraries of congress is that?

Re:

[daem0n1x]

And 99,9999% of it is completely useless.

Re:

[perryizgr8]

its a decimal, not a comma. why do so many people make the same stupid mistake?

Re:

[daem0n1x]

Stupid is to think English is the only language in the world.

Re:

[perryizgr8]

decimal notation changes with language?!? that's a new one. even if it does, the rest of your comment WAS in english.

Re:

[daem0n1x]

decimal notation changes with language?!?

Yes it does.

that's a new one.

Only for you

even if it does, the rest of your comment WAS in english.

The decimal separator on my keypad is comma. If you weren't so swift in calling me stupid, I would have explained it earlier.

Re:

[perryizgr8]

ok, my bad.

Hellagood

[slipnslidemaster]

That's 1.8 x 10^-6 hellabytes for those of you keeping track.

Re:

[BrokenHalo]

That unit is as good as any. We're rapidly running out of prefixes here, and we still need formal definitions for units such as bucketload, shitload, shitton[ne] and so forth.

Re:

[slipnslidemaster]

The key to getting a prefix established...is to just start using it.  The rest of the world and ultimately standardizing bodies will adopt it eventually.

http://www.facebook.com/pages/The-Official-Petition-to-Establish-Hella-as-the-SI-Prefix-for-1027/277479937276?ref=search&sid=1050298974.3729275378..1

Re:

[BrokenHalo]

Fair enough, but I doubt if you're going to win any converts by insisting on posting your comments with horrible monospaced fonts just to grab attention... :-}

Re:

[perryizgr8]

and why in hell does a /. tab not close immediately but does some shit for a few miiliseconds before closing?

Re:

[tom17]

Why two? For redundancy?

Re:

[L4t3r4lu5]

rel=nofollow dude. Your linkwhoring is pointless here.

Can I have that in LoCs

[Bob the Super Hamste]

Can we get that in a proper measurement like Libraries of Congress.

Re:

[Phurge]

Using 1 LoC = 20 TB, then 1.8 ZB = 96,636,764 LoCs

Or as wolfram alpha says 1.8 ZB = 144,000 x estimated information content of all human knowledge.

Re:

[Phurge]

oops 1.8 ZB = 144 x estimated information content of all human knowledge.

Re:

[Cylix]

Or as woflram and heart says, 0 = 144,000 x 0 (estimated worth of human knowledge).

How much is redundant

[Bob the Super Hamste]

I wonder how much of that data is redundant. I know that for one of my side projects I have "redundant" data that I got from the Minnesota DNR, various MN counties, the state legislature, and the federal gov. Even after it had been preprocessed and trimmed down so it only has what I care about it is still around 12GB of vector data which is about 1/3 the original size.

bah

[PJ6]

that's just Netflix.

57.5 billion 32GB iPads required to store this :D

[Mightee]

to store this amount of data, you need 57.5 billion 32GB iPads which will cost around $34.4 trillion — and that's equivalent to Gross Domestic Product (GDP) of United States, Japan, China, Germany, France, the United Kingdom and Italy combined. :D

Re:

[networkBoy]

You are right, of course.
That'll be 1,500,000,000,000 double density 8 inch floppies worth of data. Much more efficient, and at the last price I paid for them, (about 12 years ago now) they were $12.00 each (not per box, each)) so that is $18,000,000,000,000, or roughly 52.325581395348837209302325581395% of the cost of the iPads.
-nB

Re:

[L4t3r4lu5]

Don't worry, the large size won't be an issue. You can put it in a ZIP and then put that into another ZIP and so on.

That's just stupid. They use the same compression algorithm!

Put it in a ZIP in a TAR in a RAR in a 7z in an ACE in a bZip in a CAB in a dmg in a a ARJ, and finally save it as a GIF. You can't use JPEG as it's lossy.

Re:

[Noughmad]

No, man. It's just ZIP's all the way down.

Great

[Dunbal]

More space to be filled by Russian mining bots. Oh wait which universe is this?

Large Hadron Collider data anomaly?

[L4t3r4lu5]

So it generates 1PB of data per second, yet from the article "[T]he data comes from the four machines on the LHC in which the collisions are monitored â" Alice, Atlas, CMS and LHCb â" which send back 320MB, 100MB, 220MB and 500MB"

That's a few orders of magnitude short of 1 Petabyte, folks. Where are these numbers coming from?

Re:

[L4t3r4lu5]

Those numbers add up to the 25PB/year that they store. The other 99.9999% (really) is filtered our at the detectors before being "sent back".

Ah, thanks for that. I was under the impression that the computer centre would be discarding the unused data, but if the detectors are smart enough to do it then all the better!

Re:

[Eivind]

One is the raw-amount, simply the number of sensors multiplied with the frequency each sample at multiplied with the size of each sample.

But the article itself say they filter and store only the interesting stuff, which is, as we can see from the later numbers, a triflingly small fraction of the entirety.

A camera that can do full-HD at 30fps captures 186MB/s afterall, but it does not follow that a facility with 3 such security-cameras need to store however many petabytes that becomes, in order to have a rec

Re:

[AJH16]

As eivind pointed out, the PB a second is raw data. The LHC utilizes 2 layers of in hardware filtering and another layer of software filtering (as I recall from a while back at least) in order to trim the data down to a quasi-reasonable data stream that can be effectively stored and analysed.

Re:

[MartinSchou]

If you drive 60 miles/hour for 30 seconds, you haven't driven 60 miles. One is a measure of speed, the other is distance.

Same with this. 1 PB/s is speed. 1,140 MB is the amount of data. All it really means is that these 1,140 MB are generated (and possibly collected) in 1.06 microseconds.

Wow, great compression at the LHC!

[brian0918]

(typified by the Large Hadron Collider at CERN, which generates 1 petabyte of data per second)

So, the LHC produces 1 petabyte per second, and given that there are 30+ million seconds in the year, that means the LHC produces 30+ zettabytes a year. Clearly there is a problem with your typification.

Bogus

[vlm]

I call bogus on this.

10e21 / 10e10 = 10e11 bytes/living human being.

The global GDP and global hard drive manufacturers simply cannot support a 100 GB hard drive per person per year... Cheapest option per byte is probably 1 TB drive for every 10 people. My basement therefore balances against a small African village, but there's plenty of small African villages, and only one me.

Even if all the ACTIVE /.-er types have a basement like mine, and they do not, there are simply not enough of us. And on a global G

Re:

[stevelinton]

Do the sums. 1 ZB = 10^9 TB. a TB had drive costs c US$50, probably less in quantity, so information storage is US$50 billion/year industry.
Doesn't seem implausible to be honest.

Re:

[weffew...]

Most of CERN's data isn't on hard drives anyway - it's got the biggest tape system in Europe that I know of (I do high end storage for a living).

A photo, for funsies: http://www.flickr.com/photos/naezmi/3309812634/ [flickr.com]

C

LHC data will not be information for years to come

[recharged95]

And most of it will be junk data.

Data != Information. And then there's Metcalfe's law applied to information value.