Avoiding a Digital Dark Age 287
al0ha writes to recommend a worthwhile piece up at American Scientist on the problems of archiving and data preservation in an age where all data are stored digitally. "It seems unavoidable that most of the data in our future will be digital, so it behooves us to understand how to manage and preserve digital data so we can avoid what some have called the 'digital dark age.' This is the idea — or fear! — that if we cannot learn to explicitly save our digital data, we will lose that data and, with it, the record that future generations might use to remember and understand us. ... Unlike the many venerable institutions that have for centuries refined their techniques for preserving analog data on clay, stone, ceramic or paper, we have no corresponding reservoir of historical wisdom to teach us how to save our digital data. That does not mean there is nothing to learn from the past, only that we must work a little harder to find it."
Won't matter (Score:4, Insightful)
Our landfills will provide all the info they need.
sadly lawyers are working tirelessly (Score:2, Insightful)
to ensure this never happens. This is the same reason why DVD's and Bluerays will never work in 100 years time.
DRM will destroy any record of our current culture, but looking around at the abyss, I really have to say its for the better.
But I already feel bad for the eventual people that will spend far too much time trying to recover "scary movie part 15" or some other 'gem' from our time. But much like 'abandonware' and other areas of trying preserve machine code, lawyers will always race in to make sure all copies are lost forever.
Support things like SIMH while you still can!
The Middle Ages didn't have the DMCA (Score:5, Insightful)
Re:Won't matter (Score:3, Insightful)
right on, and for that matter it is silly to say we don't have paper records any more. We have even more of them than ever before. Receipts, leases, mortgages, contracts, invoices, manifests, packing slips, explanation of benefits (EOB), licenses, warranties, guarantees, manuals....fuck, if anyone thinks digital age means less just order a single piece of software on Amazon and by the you take everything out of the box you'll have generated at least eight items on the list I just mentioned. God damn!
perfect example: Geocities (Score:5, Insightful)
It is indeed a big problem. The problem was illustrated recently when Yahoo suddenly pulled the plug on Geocities, wiping out a vast cultural archive that went back to the early days of the internet, a lot of valuable information was lost as a result of that. Yahoo's blatant arrogance caused me to refuse to ever use any of their products again. Geocities was actually a fairly nice service, often people criticised it because of the ads, but how do you pay to continue to offer a free service. The loss of geocities was a perfect example of the need for a permenant store or online archive of information, personal websites and so on that can be maintained as a cultural legacy and informational resource.
ffs.. the "zomg how to preserve" story -again-!? (Score:4, Insightful)
Seriously, Slashdot.. until there's a revolutionary insight into this matter.. quick posting these stories ad nauseum.
For further commentary, see previous stories... here's one.. it's from september 2009 and -nothing has changed-.
http://ask.slashdot.org/story/09/09/29/1646251/Archiving-Digital-Artwork-For-Museum-Purchase [slashdot.org]
To forget is good (Score:3, Insightful)
IMHO we'll find that our problem is that we drown in a sea of useless information because we can't find the islands of relevance. Trying to archive everything will only lead to failing to archive anything. On the other hand I doubt that we'll lose much important information despite failing at organized preservation attempts, because important information is copied all the time, which is the only way for information to survive quickly changing technologies and file formats anyway.
In a more philosophical light, I think that forgetting is good for us. It frees us from the constraints of our past and makes way for new ideas. Archives are backwards-facing, but we all live in the future, all the time.
Forecast: Cloudy forever (Score:5, Insightful)
I think that many people are failing to appreciate the longevity of information preservation
that cloud computing (more specifically, redundant, geographically distributed network storage) can bring.
If we get the protocols right, and insist on open standards for data interchange, we can obtain
properties such as:
Data bundles that know how to move themselves to more recently commissioned, and/or more
reliable hosts.
Data bundles that know how to check in with copies of themselves, to make sure there are enough of
them alive, and that they are adequately geographically distributed, at every given moment.
If not, then more baby copies of the same data would be produced and stored elsewhere automatically.
There are other issues to longevity of course, like maintenance of software that understands different
versions of data etc. Not trivial but very doable.
How long an individual disk or SSD or stone tablet lasts is COMPLETELY IRRELEVANT to
the prospects for information longevity, given the network, and new levels of automated distribution
that will take place on it going forward.
Great! Now I'll have to buy the White Album again (Score:4, Insightful)
We will naturally make multiple copies of everything we consider important, continually transcribing important data onto the latest generation data storage media. (Consider what was the very first publication printed on Gutenberg's big invention.) Unfortunately, that's not necessarily what will be considered important many generations into the future.
I have every confidence that, far into the future, we will have or be able to develop the capability to read any media we preserve today. The problem then becomes how to determine what data we should should preserve now rather than how to preserve it. What do we know now that will be important and useful to someone 10^n years from today?
Re:perfect example: Geocities (Score:3, Insightful)
I checked archive.org backups of geocities. half of the sites are not backed up correctly. Mine was never backed up, it seems, at all. With most sites 90% of the files are missing. Is archive.org the solution? Apparently not.
Self-correcting problem (Score:4, Insightful)
we are generating data far, far faster than we can save. We have for some time, and while trends for storage are catching up, we will always be able to generate more than we store, as a function of how computing and communications work.
So what to save? The Director of the NLM had a unique insight on this exact question: [paraphrasing] "What is used, is saved." Basically, its the utility of information, that information that people find useful and actually use is the best proxy for long term value. The good thing is that all people are motivated to store and maintain the data they find useful, or their constituents or customers desire. As long as people keep wanting data, it will be stored and available.
This is a very different situation to real-world archeology. In the digital, connected world we can access data today once it's publicly available, evaluate it and use it if we want. There is no dust that covers old data, it does not get buried...
Re:PLASTIC PUNCH CARDS (Score:3, Insightful)
The army had a program to design a means of storing data in case of really being nuked back into the stone age. They chose punched metal tape. Most plastic doesn't last long whn exposed to sunlight or weather, and the downside of a card deck is obvious the first time you drop one down the stairs. It's a clevel idea really, since you can read punch tape manually if you have to, and it's far faster than cards.
Re:perfect example: Geocities (Score:3, Insightful)
Don't trust cloud computing. (Score:3, Insightful)
Practical example : Classics emulation (Score:4, Insightful)
The main way ancient writing reached us is because someone copied it. Lots of copies. {...} With all the copy protection and encryption on our media today, can we ever copy the data and be able to decipher it again?
(And as another example of copies being important for preserving : Fritz Lang's Metropolis [wikipedia.org] got recently another 30 minutes of its missing part recovered from a copy located in Argentina)
After a long enough time, virtually any DRM measure end-up being broken. What only matters is time, resources and some clever tricks (to avoid waiting until universe heat-death while bruteforcing a 4096bit key).
So DRM has only 2 direct effects :
- it annoys legitimate users everywhere with no practical reason.
- it forces the basement-dwelling teen with too much free time on their hand to wait until 2 weeks before official launch date, instead of 3 weeks before, because it took 1 week to the pirates to find a way to break the DRM.
This implies 2 results :
- That the 99.99% of pirate users, will never ever interact with the DRM nor be affected by it in any way.
- The important part : DRM protected piece of data will get copied, eventually and a lot. Lots of copies will exist and virtually 99.99% of these copies will be the "pirated" copies. Be it legal backup or unlicensed copies.
So in the end, the DRM-protected data will survive, only not the DRM version itself, but the DRM-free version as found on The Pirate Bay and similar. Case in point : Classics emulation.
Most of the companies which produced the game we played as children are now belly. Of the few remaining, few of them have kept the assets of their old production. Few of them are interested in doing anything with these old assets. The few who do, generally do modern re-imaging and re-interpretation, rather than re-issuing the old.
So in short, if you ever wanted to pull back some of your children memories out of the grave, don't count on the original companies. ... as image of pirated disks. It's practically sure that, if in 2010 you want to play the same game as in 1985, you'd probably see a cracktro in the beginning.
Some time you can find still working vintage equipment and media - but these will eventually break.
Today, the biggest part of these oldies are available
All your Commodore C64, Amiga, etc. favourite games are currently best sourced from download site which contain warez copies that were carried over back from that era, while at the same time the companies went belly up and/or let their assets rot.
So, in 25 years, when most of the current media companies have either disappeared, or completely forgotten about today's media, your children's best way to find a copy of them to remember fond memories, would be finding a copy which will be the digital descendant of what's today on pirate bay.
Yes, **AA, today's EVIL pirate, might be tomorrow's heroic archivist.
In 25 years, when the current maker of
Re:Won't matter (Score:3, Insightful)
I would suspect that, in the future, our ability to understand and figure things out will be far higher than it is today. Especially since the question of what a DVD is for is clear - not just to us now, but I would imagine even to someone who had no preconceived notion would be able to piece together the clues into what it might have been used for.
A reflective on one side, perfectly round disc? Looking at it under a microscope would no doubt show the presence of the "peaks and valleys" of digital data, and I think it would shortly fall into place.
Of course, a thought experiment such as this is nearly impossible to do, as I don't know anyone intelligent enough that also has never experience optical storage.
It's just that, as time passes, and our perspective of the world zooms out (coinciding with our understanding of the world), it becomes much easier to see how things are connected together. In the above example, part of the trouble with the Mayan civ is that we know so little about them and their world. It is not that they were complicated, or smarter than us, or were able to figure out things better than us (Yes, I'll see you all in 2013); the real issue is that we do not know enough about their fundamental culture in order to deduce what they were using things for. Certainly, using rope as a form of writing is an incredibly unusual way to write.
However, time marches on, and someone figured it out. Just like they will in the future.
In a way, it would be interesting if, in the future, someone did confusingly stumble across a shattered DVD, and, having analysed the data, finds a young man's porn collection, relentlessly locked down with encryption, it takes an unusually long 15 seconds to decrypt, and his reward is just the disclaimer:
"If you're reading this, I'm probably dead."
The researcher can't help but be gripped by the strange coincidences that must have lined up to bring this to him here.
Sorry, sometimes I like to write fiction.
-Red
PS. Certainly, the obvious counter to my thoughts is the human ability to look at ONLY either the forest or the trees - certainly, I've missed the "Tree" part of the "forest" when learning and figuring things out in the past, and will continue to in the future.
Re:The fight is lost (Score:3, Insightful)
"And what parts of those digital records would be *important* information? c'mon, you are talking about personal crap."
What do you think History is but a lot of "personal craps" tied together?
Re:Perhaps the way we think.. (Score:5, Insightful)
Many of the laws that overly stymie information flow (DMCA etc.), I think, are just a knee jerk reaction in the way printing presses were suppressed, and controlled until everyone realised the benefits of having them opened up.
Barbarians have always burned down libraries. No reason to think they'd stop doing that just because they wear ties these days.
Re:924 Years and nothing has changed (Score:4, Insightful)
Because they forgot key parts of the process:
The UK fouled up by inventing new proprietary storage formats which needed custom hardware and software to read and process the data. The laserdisc needed a special laserdisc player and a BBC Micro. The BBC who produced this were years ahead of their time and had to invent a lot of stuff. Unfortunately the rest of the world invented a lot of different stuff, which is what we use today.
And how many of these systems were produced? I don't know, but they cost 4000 pounds each which is a significant investment for a school and certainly the high price reduced the number of items which were sent into the community.
Even though we have extracted the data from the original formats (and also obtained improved images by re-mastering original video footage) it seems that one of the main impediments to putting this data online is copyright - the contents of the 1986 project won't be out of copyright until 2090!
The above two points come together with "keep converting to new formats". If your stuff is all proprietary, it may be hard to convert to new formats. If your stuff is copyrighted, you may be able to convert it but you can't distribute it, and widespread distribution is one of the requirements of effective data preservation.
The data which was produced in 1986 wasn't lost and won't be lost. People are working with it and upgrading it. However, you won't be able to see it, primarily due to the shortsightedness of the original project.
So loss of digital data is not so much a technical problem, more a social problem, of shortsightedness in creation, distribution and copyright.
Kinda like the BBC's lost videotapes of Monty Python (or was it Dr Who?) ... priceless recordings were allowed to degrade and become unusable, were thrown away, or were overwritten ("media re-used"). I don't mean to point the finger only at the BBC - NASA did it too. Lack of foresight, folks.
Re:Won't matter (Score:3, Insightful)
Unfortunately, as someone who's tried to challenge a tax audit for old expenses, I can tell you that a lot of those records do not keep well. Many poor quality printouts are not likely to last even a few years due to the poor quality of the paper and the ink. This is especially true of receipts, which are on the cheapest printers possible.
like an odd sock (Score:4, Insightful)
typically kdawson posts it, what a tard.
Re: One Site to Archive Them All (Score:1, Insightful)
and One SIte to censor them