Storing Data For the Next 1,000 Years

An anonymous reader writes "This may be an interesting take on creating long-term storage technologies. A team of researchers at UCSC claims to have come up with a power-efficient, scalable way to reliably store data for a theoretical 1,400 years with regular hard drives. TG Daily has an article describing this technology and it sounds intriguing as it uses self-contained but networked storage units. It looks like a complicated solution, but the approach is manageable and may be an effective solution to preserve your data for decades and possibly centuries." Nice to see research on this using the kinds of real-world figures for disk lifetimes that recent studies have been turning up.

Only half the problem

[Raindance]

Part of the solution to very long-term storage, of course, has to involve a method to read the data you've archived.

I tend to think systems such as the one described in the article aren't good long-term solutions. If their math works on the failure rates, that's fantastic- but just try to hook up a 2028 computer to one of these things to pull the data off.*

(Ever tried to get data off an obsolete tape backup?)

I think the most reliable archival system is going to be an active one, where data is saved on modern storage hardware and always copied to more modern tech as it arrives.

The other side of this is, for anything more advanced than text-- given that you can get at the data, what do you open it with? File types die over time and it's basically impossible to find programs to open certain files nowadays, much less such programs that will run on a modern OS. I think the answer to this has to be virtualization. Store the data *and* programs that can open the filetypes you need opened inside a portable virtual machine (e.g., a Windows vmware image). Over time, you may have to layer virtual machines inside virtual machines as OSes grow obsolete. But that's okay- virtualization is only going to become more elegant, and the end result is that you'd have your data in its original environment, completely accessible by native programs.

*Some elements of this problem could be solved by having backup servers use wireless and filesharing protocols that might stand the test of time- e.g., 802.11n and SAMBA. No need to just pick one 'most likely to be future-proof' combination, either: run bluetooth and serial access, webdav and a http fileserver, etc. Still, *not* storing data on modern hardware is always going to be a risky kludge.

There's probably room for a lucrative business based around this-- figuring out the most elegant way to archive and retain meaningful access to data under various computing/disaster scenarios. Hey, I do consulting. :)

From TFA, quite sick, really.

[DaedalusHKX]

From TFA:

Santa Cruz (CA) - Have you ever thought how vulnerable your data may be through the simple fact that you may be storing your entire digital life on a single hard drive? On single drive can hold tens of thousands of pictures, thousands of music files, videos, letters and countless other documents. One malfunctioning drive can wipe out your virtual life in a blink of an eye. A scary thought. On a greater scale, at least portions of the digital information describing our generation may be put at risk by current storage technologies. There are only a few decades of life in tape and disk storage these days, but a team of researchers claims to have come up with a power-efficient, scalable way to reliably store data with regular hard drives for an estimated (theoretical) 1400 years.

My "digital life"? Scary to lose it? Man.. these people never heard of backups, or having a real life, eh? Jeez, I can store my whole "digital life" on a 1 gig USB key, with room to spare.

I've lost my backups more times than I can count, my computers are toys, mostly for communication and play. Amazing how many people put their whole LIVES on a hard disk. Remarkable actually. What would I lose? About a dozen passwords and I'd need to reinstall and re-customize my system... OH WAIT... I backed up the important scripts and source code to a DVD.. TWO in fact. Bummer, guess I don't have to cry endless tears over the loss of my "digital life".

Re:Only half the problem

[LoudMusic]

(Ever tried to get data off an obsolete tape backup?)

I think the most reliable archival system is going to be an active one, where data is saved on modern storage hardware and always copied to more modern tech as it arrives.

Oh man, the headaches involved here. It only takes five years and archived data is obsolete. And yes, virtualization can help, but in the past I've resorted to keeping an entire system available, off-line, to guarantee that the client be able to open their data. Sometimes you get lucky and there's either a plug-in for the old app to export to the new app, or one for the new app to import from the old app. But even on the rare chance that one is available, I've never seen a 100% conversion - even on simple stuff.

Maybe old data was meant to die.

Re:What about filling it up?

[Blkdeath]

Unless 10 PB (petabytes) means something other than what I think (10,000 terabytes), where did they get the $4700 number? I even read their definition of static cost [usenix.org] (You have to go up a few paragraphs) and I still don't know.

Table 3: Comparison of system and operational costs for 10 PB of storage. All costs are in thousands of dollars and reflect common configurations. Operational costs were calculated assuming energy costs of $0.20/kWh (including cooling costs).

Does $4.7 million sound a bit more realistic?

Re:Sometimes old tech is best

[evanbd]

You could, of course, update the technology a bit: Rosetta Project [rosettaproject.org]. High density, readable with a high quality microscope, and partially readable with the naked eye -- the spiral of shrinking text should make the usage instructions obvious: "get a magnifying glass, there's more here."

Reinventing Honeycomb

[zdzichu]

So, they are proposing Sun StorageTek 5800 [sun.com] (codenamed Honeycomb) as their research?

Compare article with this whitepaper [sun.com], especially Figure 13 on page 28. Networked nodes with 4 disks each, grouped in cells of 16 + 1 management node. Each object is stured redundantly on disks of different storage nodes. Everything self-contained, accessible by nice API. Oh, and the software is Open Source.

Re:Only half the problem

[dbIII]

Oh man, the headaches involved here. It only takes five years and archived data is obsolete

Only in the MS Windows world. For the rest of us if it predates ASCII we can use "dd" to convert from EBCDIC if we have to. The tapes from 1982 I recently read in however were transcribed to new media for me first in case the media had become damaged over time and because I'm not familiar with 9 track drives. It was a direct copy so the data format was retained even if it was on new media (IBM3490 format but done this year so new).

Re:Uh, what?

[Alex Belits]

I think you could make an argument that Russia and China were theocracies for much of the Stalinist period. For example I read that Mao apparently gave a speech which was interpreted as him saying that quarks were the fundamental constituent of matter. After that Chinese physicists were careful not to publish papers that might contradict the great man. In Russia Lysenkoism was famously the officially supported theory of agriculture.

No.

Government providing support to stupid opinions and doctrines does not make them a religion -- for something to be a religion it has to specifically include belief in a supernatural deity. I remember that in USSR saying that someone believes in god was the ultimate insult to his intelligence.

Re:Sometimes old tech is best

[Anonymous Coward]

Most religious people in the West know who was Gutenberg, what did he invent, with which intention, and how important it was for western literacy. But "non-religious" (rather anti-religious) people have been reading so much crap from sex and trash magazines that they no longer know how important were religious people in preservation/nurturing of literacy, classical culture, and philosophy.

Re:Only half the problem

[electrictroy]

The BBC recently ran into this problem where they discovered old 1930s-era records with 30-line video stored on them. They knew what was on the records, but they didn't have the hardware required to play them. Museums have run into similar problems with old cylinder-shaped records, and more-recently, capacitance-encoded video records (RCA CED).

It's not enough to know what's on the thing; you also need the hardware to read the gadget, and that hardware is often unavailable due to its failure to succeed, or extreme age.

After the Roman empire fell, the only thing that survived was rock and paper... which meant text. Pictures were lost; music was lost; even marble statues failed to survive (they were used as building material). Thousands of years of music and art just disappeared.

Re:Uh, what?

[hypnagogue]

Well the Old Testament was written by backward Taliban types in the dark ages. What do you expect? Something I didn't realise about the Old Testament until recently is that when they talk of the the Philistines binding Samson in 'chains of iron' it's because the Philistines had managed to master the technology to use iron but the Israelites hadn't.

1 Samuel 13:19 Now there was no blacksmith to be found throughout all the land of Israel, for the Philistines said, "Lest the Hebrews make themselves swords or spears."

It wasn't technology gap, it was arms control enforced during centuries of oppression. They certainly did have the technology, as the technology itself is described in dozens of passages. (Deu 4:20, 1 Sa 12:31)

There's lots of other signs that they were not exactly academically inclined either, like the biblical value of 3 for Pi which was less accurate than the value the competing civilisations knew.

1) A round bathtub is not the same thing as a circular bathtub, 2) even if it was circular you forgot to account for the annulus.

But don't worry your arrogant little head about it. Other people are stupid and you are smart.