Archiving Digital History at the NARA

val1s writes "This article illustrates how difficult archiving is vs. just 'backing up' data. From the 38 million email messages created by the Clinton administration to proprietary data sets created by NASA, the National Archives and Records Administration is expecting to have as much a 347 petabytes to deal with by 2022. Are we destined for a "digital dark age"?"

16000 formats?!?

[gardyloo]

Hm. This sounds like a job for OpenOffice...

ha

[The Big Ugly]

"Archiving Digital History at the NARA"

You'll have to pry it from my cold, dead hands!

Ohhhh, NARA, not NRA....

Google to the rescue!!!

[feloneous cat]

With the new GoogleNARA...

nara.google.com

Oh, wait... I'm getting ahead of myself...

Have a look at the Fedora Project

[pangloss]

http://www.fedora.info/ [fedora.info]
(Not to be confused with the Linux distribution)

From the website, Fedora is "a general purpose repository service...devoted to...providing open-source repository software that can serve as the foundation for many types of information management systems".

Problem for some is that Fedora can be a little hard to grok. It's not an out-of-the-box repository to install and run, like the repository application mentioned in the article (DSpace). It's an architecture for building repository software. Once you understand the potential for building applications on top of Fedora, you start to see some light at the end of the tunnel for just the sort of issues the article raises.

Re:Dark Ages are ahead! All aboard

[screwthemoderators]

I think it may be worse than that- that there will be a huge proliferation of false information, sensationalistic 'infotainmnet,' advertising, propaganda, etc... Why, historians of the future may be depending on /. as their main source of of information! Think of what a tragedy that would be!

Relevant, interesting post

[Council]

Here is a relevant post by Ralph Spoilsport [slashdot.org] on an earlier article, which can be found here [slashdot.org]. I am reproducing it here in full because it is very interesting and highly relevant.

this is actually a BIG question

And one that I have railed about for many years.
I have been in the same position the Author discussed, and I have come to ONLY negative conclusions. In a few words, and I hate to say this, but buddy:

WE'RE FUCKED.

Digital is a loser's proposition. backing up to analogue or even digital data on analogic substrates (such as DV tape) fail. Simply nad purely.

The *only* thing that comes close is some kind of RAID, and those, even with the plummeting price of storage, are still too expensive given the needs.

Also, a RAID assumes a continuity of several things that are not likely to be continuous:

With Video:
Framerate, number of lines, colour depth, aspect ratio, file format, compression format, Operating system compatibility, etc etc etc. All of these things are variables.

With Audio:
sample rate, compression format, bit depth, file format, etc.

Basically all of it points to very bad places.

I am fairly well convinced that our age will simply disappear. They will find our garbage, the few books not pressed on acidic paper, our paintings (fat lot of good the abstract stuff will mean to them) and drawings, that's about it. the rest will just be shiny little bits of crap in the landfill.

Since we will have used up all the dense energy forms, they will be appalled at the energy requirements just to get the few remaining museum piece devices to work. Archiving the 21st century will be impossible. To the 25th century, the 21st century will be seen as a dark age - not only for the holocaust of the die caused by the failure of the petroleum based economy, but from the simple fact that very little of the information formats we are totally geared into will survive, including this note on /.

His problem of saving personal video is just the tip ofthe iceberg. His problem is the problem of our very civilisation, writ small.

That's why I am abandoning video, and going back to painting. In 500 years, my painting CAN survive. the video simply won't.

RS

And don't give me shit about my karma or whatever. My karma's fine, I don't care about it. I'm copying this because it's interesting and contributes to the discussion.

What do you think about Ralph's thoughts?

Re:16000 formats?!?

[parasonic]

Or this [billablog.com].

strip MS HTML from Outlook mails

[rduke15]

I don't know about the NASA data sets, but they could certainly save a few petabytes by stripping the stupid HTML part of all Outlook emails...

The Solution:

[DarkEdgeX]

NARA needs to open up tons and tons of GMail accounts. Where do I send my invites so I can contribute?

Re:Data loss will always be a possibility

[writermike]

It happened with the Great Library of Alexandria, with pagan libraries throughout the Christian era, and more recently has happened with antiquities in Afghanistan and Iraq. The only thing that can reliably preserve data is large scale, geographically widespread distribution of copies.

True. But I hardly think Alexandria was lost to the tap of the Y key, a pregnant pause, then an "oops."

industrial espionage would be sillier

[mbius]

"Give them to me."

"What do you want??"

"That Gem...and the Holograms."

Re:Data loss will always be a possibility

[Tristor]

No, but it could have been lost to the strike of flint, a pregnant pause, then an "glukús theométôr" (Sweet Mother of God, for you people that suck). (Note: I spent like 20 minutes transliterating that to Latin just so I could post it on /. because it hated the Greek charset. I have no life.)