Digital Big Bang — 161 Exabytes In 2006 176
An anonymous reader tips us to an AP story on a recent study of how much data we are producing. IDC estimates that in 2006 we created, captured, and replicated 161 exabytes of digital information. The last time anyone tried to estimate global information volume, in 2003, researchers at UC Berkeley came up with 5 exabytes. (The current study tries to account for duplicating data — on the same assumptions as the 2003 study it would have come out at 40 exabytes.) By 2010, according to IDC, we will be producing far more data than we will have room to store, closing in on a zettabyte.
And here I thought Malthus was dead (Score:5, Insightful)
The awesome information we retain (Score:5, Insightful)
Supply and demand (Score:5, Insightful)
"producing far more data than we will have room to store"
That's like saying, for the last 2 months, my profit has increased by 10%. If my profit keeps increasing at 10% per month, then pretty soon I'll own all the money in the world, and then I'll own more money than exists! Damn I must stop making money now before I destroy the world economy!!!
Who are these people who draw straight lines on growth curves? Why do people print the garbage they write and why weren't they the first against the wall after the dot com bust?
The only things that seem certain are death, taxes, entropy and stupid people...
Re:Supply and demand (Score:4, Insightful)
Though with data, some people, or even companies, are merely sinks. They store huge amounts of data, mostly for auditing purposes. Access logs for webservers. Windows NT event logs. Setup logs for Windows Installer apps. For ISPs, a track record of people who got assigned an IP address, in case they get a subpoena. Change logs for DoD documents. Even CVS for developers, to keep track of umpteen old versions of software. Even the casual Web browsing session replicates information in your browser cache. Many more of these examples could be given.
We also need to produce more and more hardware to store these archived data, the most obiquitous of which is the common hard drive. In the end, we'll need more metal and magnetic matter than the Earth can provide.
Martian space missions, anyone?
Re:It was only 9 megs (Score:5, Insightful)
But seriously, I wonder what percentage of this data is text. I'd guess it is a very very small amount. When I had a film camera, in twenty years I bet I took less than 100 rolls of film. With digital cameras I've take thousands of pictures, sometimes taking a dozen or more of the same subject, just because the cost to me is practically zero. Now there are vendors that will let me upload large numbers of these amateurish photos for free, and let's pretend that there are enough people interested in seeing my pictures that these companies can pay for this storage with advertising. That's scary.
Excluding attachments I think it would be practically impossible for anyone to use up Googles 2 gig of storage, but I've heard of people using it up in little more than a week by mailling large attachments back and forth (oh yeah, I HAVE to have every single iteration of that Word document, sure I do!)
But what's scarier is that for some nominal fee (like $20 a year) they place no limit at all on my ability to hog a disk drive somewhere. I know people who are messed up in the head enough to want to test these claims. Give them 5 gig for photos and they've filled it up in a week, give them "unlimited" and they upload pure junk to see if they can break the thing.
Like any house of cards, this thing is gonna come down sooner or later. I just hope that people who are making sensible use of these online services don't lose everything along with the abusers.
Internet | uniq (Score:3, Insightful)
My work machine that I backed up a couple weeks ago, was a 30MB zip file, and 3/4 of that was my local CVS tree. So out of a 30GB, less then 1/3000th was not OS, software, or just copied locally from a data store.
At home, I've saved every email, every picture, everything from my Windows, Linux, OSX and every other box I've every had since ~1992, and that's barely a few GB uncompressed.
The amount of non-duplicate useful material is far far smaller then your would think.
Re:And here I thought Malthus was dead (Score:1, Insightful)
The number is way too low! (Score:3, Insightful)
If we consider all digital data, not just the stuff that flows over the internet, then this is way too low. Consider the data in all the DTVs, GPS receivers etc.
A top-end GPS is grinding over 10^9 bits per second in its correlators (about 50 correlator channels x 20Mbps or so sampling rate). That ends up being approx 3x10^15 bytes per year per GPS... or 40,000-odd top-end GPSs would be grinding 1.61x10^20 bytes per year. There are far more than 40k high end GPSs in the world, so the budget is already blown...
Low SNR (Score:5, Insightful)
As interesting as the sheer volume is, most of it is garbage. I'd rather have 50 terabytes of organized and accurate information than 500 exabytes of data that isn't organized, and even if it were, it's accuracy is questionable at best. In essence, even if you manage to find what you want, the correctness of that information is likely to be very low.
I've long said we are not in the information age, we are in the data age. The information age will be when we've successfully organized all this crap we're storing/transmitting.
Malthus has just gone down to the shops (Score:3, Insightful)
I remember when software came on cassettes and when food came from close to where you live.
When floppy disks were too small, we made higher-density floppy disks, and we still needed a whole box of them.
When there wasn't enough of a particular food, we got it shipped from further away.
When CD-ROMs came out, we still ended up not only filling them but spreading things over multiple CDs.
When the imported food got too expensive, we started using chemical fertilisers to grow more of them closer to home and more cheaply.
We had to invent bigger CDs. DVD became HD-DVD and Blu-Ray. People are already complaining that they're not big enough.
We got bigger trucks and bigger boats to cover food with more preservatives and ship it here from further away, and the more of this we bought, the cheaper it got.
You got that bigger hard disk, so you could amass data and store it forever. Remember how you said you'd never fill it up? Then broadband happened, and P2P happened, and fill it up you did.
You didn't worry about it, though, the same way you didn't worry about not having enough food, either. Your supermarket is awash with thousands of varieties of food, from wherever it's cheap, and you can eat as much as you want of whatever you want.
Because everything is more available, more quickly and more easily, you now have more stuff than you could ever use. Nowadays, people don't think twice about Tivo-ing or downloading something that they're never even going to watch. As the technology gets better - as disks get bigger, and as networking gets faster - this is only going to become more prevalent.
But there is a physical limit to what can be done. Do you need a new hard drive, or a new router? What metals and chemicals are required to make them? How much energy is required? Where are they built, and how do they get to you? There's only a finite amount of this stuff in the ground, and none of this is invincible to exponential growth. The people who think this can go on forever, or even for the rest of their natural lives, are kidding themselves.
Eventually, these materials will be harder to get, things will start to become more difficult to make and more expensive, and everyone will be complaining about how expensive their last computer was. Really, though, I don't even want to know these people. They've gotten their priorities all wrong.
The parent poster says we won't be running out of anything. All that's really happened is that we haven't run out yet. The planet simply can't sustain the 6.5 billion of us there are now, let alone the billions more to be born in the next few decades. The problem is that when there isn't enough to go around, some of us will be lining up for new video games and iPods, and some of us will be lining up for food, water and fuel.
I should warn you to choose wisely, but really, what do I care? Choose unwisely, and leave more for the rest of us.
Re:And here I thought Malthus was dead (Score:2, Insightful)
They wouldn't fit comfortably, and you'd certainly have to stack them. The acceleration of gravity would increase as more humans were added to the mass of the Earth.
Possibly, you'd have to import food from throughout the universe... I'm not sure if conservation of mass applies to a planet and all living (or not living) entities on it... Debris from space that enters Earth's atmosphere may or may not be useful in reproducing humans.
Of course, as the mass approaches infiniti the universe would begin to be pulled toward Earth eventually ending in one big-assed mass. Maybe. Smoke some weed, drop some acid, and figure it out for yourself. Most of you are so left-brained a little mental exploration would probably be good for you.
Re:The number is way too low! (Score:4, Insightful)
If a NMEA lat-lon string gets spit out of the serial port of a GPS and there's nothing there to capture it, it is not part of their count. They're not counting bitrate on data generators and multiplying times bandwidth. They're counting discrete blocks of saved data. You cannot arrive at the latter from the former, just like you can't tell how much water is behind Hoover Dam on average during the year by measuring the average daily flow rate of the Colorado river and multiplying by 365.