Error-Proofing Data With Reed-Solomon Codes 196
ttsiod recommends a blog entry in which he details steps to apply Reed-Solomon codes to harden data against errors in storage media. Quoting: "The way storage quality has been nose-diving in the last years, you'll inevitably end up losing data because of bad sectors. Backing up, using RAID and version control repositories are some of the methods used to cope; here's another that can help prevent data loss in the face of bad sectors: Hardening your files with Reed-Solomon codes. It is a software-only method, and it has saved me from a lot of grief..."
Re:It can make files a bit hard to read, though (Score:5, Insightful)
It really depends where you store the FEC, some techniques store the fec separately others concatenate and others interleave the FEC. Each method has its own advantages and disadvantages.
Version control != backups (Score:3, Insightful)
huh??? (Score:3, Insightful)
I have been around this industry quite a while, and I call bullshit on that.
Yes, RS should be a file system service. (Score:4, Insightful)
I agree. I'm willing to have a small loss in speed and a small increase in price to have better data integrity.
There is already data integrity technology embedded in hard drives, and I support making it more robust.
Re:Version control != backups (Score:3, Insightful)
Well, you shouldn't commit until you believe you have it in a state where the changes are usable (i.e. don't break the tree), but beyond that, I'd rather see more commits of smaller amounts of code than giant commits that change ten thousand things. If you end up having to back out a change, it's much easier if you can easily isolate that change to a single commit. My rule is commit early, commit often. I'm not the only one, either:
http://blog.red-bean.com/sussman/?p=96 [red-bean.com]
Re:Harden Files (Score:1, Insightful)
It never ceases to amaze me that the juvenile "heh heh heh.. he said 'harden'" response always gets modded funny. Mods, here's a tip: These kinds of jokes aren't funny unless you are a) 13 years old or b) really drunk.
Re:Version control != backups (Score:4, Insightful)
The best solution is for developers to use their own private branches. Then they can commit as much as they want, and integrate into the main branch when they're ready. Unfortunately subversion has crappy support for integration (even with version 1.5 AFAICT) compared to something like perforce.
Re:As I understand it... (Score:4, Insightful)
You're asking the wrong question.
The right question is: Given a 1Gb file, how much "mutation" do you have to do to it to produce a file with the same hash? And the answer to that is: Enough to make the data unrecoverable no matter what you do.
Datarecovery "data". (Score:5, Insightful)
Working for a datarecovery company, I know that about half the cases where data is lost the whole drive "disappears". So, bad sectors? You can solve that problem with reed solomon! Fine! But that doesn't replace the need for backups to help you recover from: accidental removal, fire, theft and total disk failure (and probably a few other things I can't come up with right now)... .
Re:As I understand it... (Score:3, Insightful)
Re:I wish PAR2 would have kept improving... (Score:2, Insightful)
Agreeing with Fnord666, the software does not use an open algorithm. The general tone of the site is "use this software it is awesome, don't argue". There doesn't seem to be verification of its awesomeness. Furthermore, the program author's tone in many of the forum posts is abrasive and near combative when people question it.
PAR2 is proven but limited. This
Is this really the case, that no one has taken PAR2 to the next level? Judging from the lack of links in these comments to the flamebaiting posts of "we've been doing this for years" there isn't much progress.
I want PAR3.
Re:Drives already do this (Score:3, Insightful)
Thing is, the "overcompressed" MP3 recorders are good enough. Most people use them to record lecture notes, or a meeting, or just talking to themselves. Those are about the only reasons to really need a portable recorder, and for those uses, mp3 is very good. Just because it's low bitrate doesn't mean it's bad, and just because your DAT recorder had higher quality doesn't mean it's more fit for the purposes it would be used for. Seriously... running and recording? Why would you ever want to do that?