Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Data Storage

Avoiding a Digital Dark Age 287

al0ha writes to recommend a worthwhile piece up at American Scientist on the problems of archiving and data preservation in an age where all data are stored digitally. "It seems unavoidable that most of the data in our future will be digital, so it behooves us to understand how to manage and preserve digital data so we can avoid what some have called the 'digital dark age.' This is the idea — or fear! — that if we cannot learn to explicitly save our digital data, we will lose that data and, with it, the record that future generations might use to remember and understand us. ... Unlike the many venerable institutions that have for centuries refined their techniques for preserving analog data on clay, stone, ceramic or paper, we have no corresponding reservoir of historical wisdom to teach us how to save our digital data. That does not mean there is nothing to learn from the past, only that we must work a little harder to find it."
This discussion has been archived. No new comments can be posted.

Avoiding a Digital Dark Age

Comments Filter:
  • Won't matter (Score:4, Insightful)

    by countertrolling ( 1585477 ) on Tuesday February 23, 2010 @07:50PM (#31252808) Journal

    Our landfills will provide all the info they need.

  • by Anonymous Coward on Tuesday February 23, 2010 @07:56PM (#31252880)

    to ensure this never happens. This is the same reason why DVD's and Bluerays will never work in 100 years time.

    DRM will destroy any record of our current culture, but looking around at the abyss, I really have to say its for the better.

    But I already feel bad for the eventual people that will spend far too much time trying to recover "scary movie part 15" or some other 'gem' from our time. But much like 'abandonware' and other areas of trying preserve machine code, lawyers will always race in to make sure all copies are lost forever.

    Support things like SIMH while you still can!

  • by MagikSlinger ( 259969 ) on Tuesday February 23, 2010 @07:58PM (#31252900) Homepage Journal
    The main way ancient writing reached us is because someone copied it. Lots of copies. Sometimes translated into another language and back, for example, a lot of Greek learning went into Arabic and came back out into Latin or Greek. With all the copy protection and encryption on our media today, can we ever copy the data and be able to decipher it again?
  • Re:Won't matter (Score:3, Insightful)

    by rubycodez ( 864176 ) on Tuesday February 23, 2010 @08:00PM (#31252924)

    right on, and for that matter it is silly to say we don't have paper records any more. We have even more of them than ever before. Receipts, leases, mortgages, contracts, invoices, manifests, packing slips, explanation of benefits (EOB), licenses, warranties, guarantees, manuals....fuck, if anyone thinks digital age means less just order a single piece of software on Amazon and by the you take everything out of the box you'll have generated at least eight items on the list I just mentioned. God damn!

  • by Eravnrekaree ( 467752 ) on Tuesday February 23, 2010 @08:04PM (#31252976)

    It is indeed a big problem. The problem was illustrated recently when Yahoo suddenly pulled the plug on Geocities, wiping out a vast cultural archive that went back to the early days of the internet, a lot of valuable information was lost as a result of that. Yahoo's blatant arrogance caused me to refuse to ever use any of their products again. Geocities was actually a fairly nice service, often people criticised it because of the ads, but how do you pay to continue to offer a free service. The loss of geocities was a perfect example of the need for a permenant store or online archive of information, personal websites and so on that can be maintained as a cultural legacy and informational resource.

  • by Animaether ( 411575 ) on Tuesday February 23, 2010 @08:04PM (#31252978) Journal

    Seriously, Slashdot.. until there's a revolutionary insight into this matter.. quick posting these stories ad nauseum.

    For further commentary, see previous stories... here's one.. it's from september 2009 and -nothing has changed-.

    http://ask.slashdot.org/story/09/09/29/1646251/Archiving-Digital-Artwork-For-Museum-Purchase [slashdot.org]

  • To forget is good (Score:3, Insightful)

    by Anonymous Coward on Tuesday February 23, 2010 @08:09PM (#31253026)

    IMHO we'll find that our problem is that we drown in a sea of useless information because we can't find the islands of relevance. Trying to archive everything will only lead to failing to archive anything. On the other hand I doubt that we'll lose much important information despite failing at organized preservation attempts, because important information is copied all the time, which is the only way for information to survive quickly changing technologies and file formats anyway.

    In a more philosophical light, I think that forgetting is good for us. It frees us from the constraints of our past and makes way for new ideas. Archives are backwards-facing, but we all live in the future, all the time.

  • by presidenteloco ( 659168 ) on Tuesday February 23, 2010 @08:10PM (#31253030)

    I think that many people are failing to appreciate the longevity of information preservation
    that cloud computing (more specifically, redundant, geographically distributed network storage) can bring.

    If we get the protocols right, and insist on open standards for data interchange, we can obtain
    properties such as:

    Data bundles that know how to move themselves to more recently commissioned, and/or more
    reliable hosts.

    Data bundles that know how to check in with copies of themselves, to make sure there are enough of
    them alive, and that they are adequately geographically distributed, at every given moment.
    If not, then more baby copies of the same data would be produced and stored elsewhere automatically.

    There are other issues to longevity of course, like maintenance of software that understands different
    versions of data etc. Not trivial but very doable.

    How long an individual disk or SSD or stone tablet lasts is COMPLETELY IRRELEVANT to
    the prospects for information longevity, given the network, and new levels of automated distribution
    that will take place on it going forward.

  • We will naturally make multiple copies of everything we consider important, continually transcribing important data onto the latest generation data storage media. (Consider what was the very first publication printed on Gutenberg's big invention.) Unfortunately, that's not necessarily what will be considered important many generations into the future.

    I have every confidence that, far into the future, we will have or be able to develop the capability to read any media we preserve today. The problem then becomes how to determine what data we should should preserve now rather than how to preserve it. What do we know now that will be important and useful to someone 10^n years from today?

  • by Eravnrekaree ( 467752 ) on Tuesday February 23, 2010 @08:28PM (#31253278)

    I checked archive.org backups of geocities. half of the sites are not backed up correctly. Mine was never backed up, it seems, at all. With most sites 90% of the files are missing. Is archive.org the solution? Apparently not.

  • by drDugan ( 219551 ) * on Tuesday February 23, 2010 @08:29PM (#31253288) Homepage

    we are generating data far, far faster than we can save. We have for some time, and while trends for storage are catching up, we will always be able to generate more than we store, as a function of how computing and communications work.

    So what to save? The Director of the NLM had a unique insight on this exact question: [paraphrasing] "What is used, is saved." Basically, its the utility of information, that information that people find useful and actually use is the best proxy for long term value. The good thing is that all people are motivated to store and maintain the data they find useful, or their constituents or customers desire. As long as people keep wanting data, it will be stored and available.

    This is a very different situation to real-world archeology. In the digital, connected world we can access data today once it's publicly available, evaluate it and use it if we want. There is no dust that covers old data, it does not get buried...

  • by lgw ( 121541 ) on Tuesday February 23, 2010 @08:52PM (#31253572) Journal

    The army had a program to design a means of storing data in case of really being nuked back into the stone age. They chose punched metal tape. Most plastic doesn't last long whn exposed to sunlight or weather, and the downside of a card deck is obvious the first time you drop one down the stairs. It's a clevel idea really, since you can read punch tape manually if you have to, and it's far faster than cards.

  • by Ltap ( 1572175 ) on Tuesday February 23, 2010 @09:04PM (#31253712) Homepage
    Part of the problem is of manpower - geocities was just so massive, and Yahoo gave them very little time to archive anything properly, so most of it was simply a dash to copy as much as they could before it was deleted. When you look at public domain audio, video, and texts, you'll see that things have been done much better.
  • by Lazarian ( 906722 ) on Tuesday February 23, 2010 @09:18PM (#31253864)
    If you want to preserve your data, backup your data yourself, and keep it on its own storage medium. There seems to be a growing impetus where "cloud computing" and "thin clients" are envisioned to replace traditional architectures where data is stored and decoded by the individual who owns/created it. I'd rather store my data myself than ask permission to access it through the equivalent of a 1980's green screen dumb terminal from some corporation who's interests run contrary to mine.
  • by DrYak ( 748999 ) on Tuesday February 23, 2010 @09:22PM (#31253898) Homepage

    The main way ancient writing reached us is because someone copied it. Lots of copies. {...} With all the copy protection and encryption on our media today, can we ever copy the data and be able to decipher it again?

    (And as another example of copies being important for preserving : Fritz Lang's Metropolis [wikipedia.org] got recently another 30 minutes of its missing part recovered from a copy located in Argentina)

    After a long enough time, virtually any DRM measure end-up being broken. What only matters is time, resources and some clever tricks (to avoid waiting until universe heat-death while bruteforcing a 4096bit key).
    So DRM has only 2 direct effects :
    - it annoys legitimate users everywhere with no practical reason.
    - it forces the basement-dwelling teen with too much free time on their hand to wait until 2 weeks before official launch date, instead of 3 weeks before, because it took 1 week to the pirates to find a way to break the DRM.

    This implies 2 results :
    - That the 99.99% of pirate users, will never ever interact with the DRM nor be affected by it in any way.
    - The important part : DRM protected piece of data will get copied, eventually and a lot. Lots of copies will exist and virtually 99.99% of these copies will be the "pirated" copies. Be it legal backup or unlicensed copies.

    So in the end, the DRM-protected data will survive, only not the DRM version itself, but the DRM-free version as found on The Pirate Bay and similar. Case in point : Classics emulation.
    Most of the companies which produced the game we played as children are now belly. Of the few remaining, few of them have kept the assets of their old production. Few of them are interested in doing anything with these old assets. The few who do, generally do modern re-imaging and re-interpretation, rather than re-issuing the old.

    So in short, if you ever wanted to pull back some of your children memories out of the grave, don't count on the original companies.
    Some time you can find still working vintage equipment and media - but these will eventually break.
    Today, the biggest part of these oldies are available ... as image of pirated disks. It's practically sure that, if in 2010 you want to play the same game as in 1985, you'd probably see a cracktro in the beginning.

    All your Commodore C64, Amiga, etc. favourite games are currently best sourced from download site which contain warez copies that were carried over back from that era, while at the same time the companies went belly up and/or let their assets rot.

    So, in 25 years, when most of the current media companies have either disappeared, or completely forgotten about today's media, your children's best way to find a copy of them to remember fond memories, would be finding a copy which will be the digital descendant of what's today on pirate bay.
    Yes, **AA, today's EVIL pirate, might be tomorrow's heroic archivist.

    In 25 years, when the current maker of

  • Re:Won't matter (Score:3, Insightful)

    by Redlazer ( 786403 ) on Tuesday February 23, 2010 @09:34PM (#31254036) Homepage
    While I agree to some extent, an advanced culture is advanced not just with it's technology, but also in the way it thinks.

    I would suspect that, in the future, our ability to understand and figure things out will be far higher than it is today. Especially since the question of what a DVD is for is clear - not just to us now, but I would imagine even to someone who had no preconceived notion would be able to piece together the clues into what it might have been used for.

    A reflective on one side, perfectly round disc? Looking at it under a microscope would no doubt show the presence of the "peaks and valleys" of digital data, and I think it would shortly fall into place.

    Of course, a thought experiment such as this is nearly impossible to do, as I don't know anyone intelligent enough that also has never experience optical storage.

    It's just that, as time passes, and our perspective of the world zooms out (coinciding with our understanding of the world), it becomes much easier to see how things are connected together. In the above example, part of the trouble with the Mayan civ is that we know so little about them and their world. It is not that they were complicated, or smarter than us, or were able to figure out things better than us (Yes, I'll see you all in 2013); the real issue is that we do not know enough about their fundamental culture in order to deduce what they were using things for. Certainly, using rope as a form of writing is an incredibly unusual way to write.

    However, time marches on, and someone figured it out. Just like they will in the future.

    In a way, it would be interesting if, in the future, someone did confusingly stumble across a shattered DVD, and, having analysed the data, finds a young man's porn collection, relentlessly locked down with encryption, it takes an unusually long 15 seconds to decrypt, and his reward is just the disclaimer:

    "If you're reading this, I'm probably dead."

    The researcher can't help but be gripped by the strange coincidences that must have lined up to bring this to him here.

    Sorry, sometimes I like to write fiction.


    PS. Certainly, the obvious counter to my thoughts is the human ability to look at ONLY either the forest or the trees - certainly, I've missed the "Tree" part of the "forest" when learning and figuring things out in the past, and will continue to in the future.

  • by turbidostato ( 878842 ) on Tuesday February 23, 2010 @09:34PM (#31254038)

    "And what parts of those digital records would be *important* information? c'mon, you are talking about personal crap."

    What do you think History is but a lot of "personal craps" tied together?

  • by CharlyFoxtrot ( 1607527 ) on Tuesday February 23, 2010 @10:26PM (#31254488)

    Many of the laws that overly stymie information flow (DMCA etc.), I think, are just a knee jerk reaction in the way printing presses were suppressed, and controlled until everyone realised the benefits of having them opened up.

    Barbarians have always burned down libraries. No reason to think they'd stop doing that just because they wear ties these days.

  • by elronxenu ( 117773 ) on Wednesday February 24, 2010 @12:21AM (#31255422) Homepage

    Because they forgot key parts of the process:

    • Keep it simple
    • Make lots of copies which are readily available
    • Keep converting to new formats over the years

    The UK fouled up by inventing new proprietary storage formats which needed custom hardware and software to read and process the data. The laserdisc needed a special laserdisc player and a BBC Micro. The BBC who produced this were years ahead of their time and had to invent a lot of stuff. Unfortunately the rest of the world invented a lot of different stuff, which is what we use today.

    And how many of these systems were produced? I don't know, but they cost 4000 pounds each which is a significant investment for a school and certainly the high price reduced the number of items which were sent into the community.

    Even though we have extracted the data from the original formats (and also obtained improved images by re-mastering original video footage) it seems that one of the main impediments to putting this data online is copyright - the contents of the 1986 project won't be out of copyright until 2090!

    The above two points come together with "keep converting to new formats". If your stuff is all proprietary, it may be hard to convert to new formats. If your stuff is copyrighted, you may be able to convert it but you can't distribute it, and widespread distribution is one of the requirements of effective data preservation.

    The data which was produced in 1986 wasn't lost and won't be lost. People are working with it and upgrading it. However, you won't be able to see it, primarily due to the shortsightedness of the original project.

    So loss of digital data is not so much a technical problem, more a social problem, of shortsightedness in creation, distribution and copyright.

    Kinda like the BBC's lost videotapes of Monty Python (or was it Dr Who?) ... priceless recordings were allowed to degrade and become unusable, were thrown away, or were overwritten ("media re-used"). I don't mean to point the finger only at the BBC - NASA did it too. Lack of foresight, folks.

  • Re:Won't matter (Score:3, Insightful)

    by Antique Geekmeister ( 740220 ) on Wednesday February 24, 2010 @12:36AM (#31255506)

    Unfortunately, as someone who's tried to challenge a tax audit for old expenses, I can tell you that a lot of those records do not keep well. Many poor quality printouts are not likely to last even a few years due to the poor quality of the paper and the ink. This is especially true of receipts, which are on the cheapest printers possible.

  • like an odd sock (Score:4, Insightful)

    by timmarhy ( 659436 ) on Wednesday February 24, 2010 @01:42AM (#31255928)
    this is the same shit story that keeps popping up on /. ever 6 months or so.

    typically kdawson posts it, what a tard.

  • by Anonymous Coward on Wednesday February 24, 2010 @03:48AM (#31256596)

    and One SIte to censor them

Computers are unreliable, but humans are even more unreliable. Any system which depends on human reliability is unreliable. -- Gilb