How To Verify CD-R Data Retention Over Time? 303
Peter (Professor) Fo writes "I've recently had two CD-Rs reported to me as faulty which are just 3 years old. This is worrying — I suspect the failure rate for this batch could be 10%. When researching CD longevity there is old and unreliable information; pious 'how to cosset your discs so they last 100 years' blurb; and endless discussions of what sort of dye to use, don't use cheap media, burn slower (or don't), but not much by way of hard facts besides there's a lot of data loss going on.
Does anyone know of a generic utility (win or *nix would suit me) that can map sector readability/error rates of CDs? I'd like to measure decay over time in my environment with my media and my other variables; and I expect others would too."
dvdisaster (Score:5, Informative)
You should probably try dvdisaster [dvdisaster.net]. it can test media, and can create (on disk or external) redundancy data, which can be used to recover later.
It's also open source, so you could probably coerce it to export some more information
par2 (Score:5, Informative)
Error-Proofing Data With Reed-Solomon Codes (Score:5, Informative)
Save them again with long lasting solutions (Score:4, Informative)
If you consider your data worth it, have enough time and enough money, you should probably re-burn/re-save them to long lasting media.
There was a previous post on askslashdot about this subject.
http://ask.slashdot.org/article.pl?sid=08/08/27/2119252 [slashdot.org]
My suggestion was to use Plasmon "Century-Disc" :
http://slashdot.org/comments.pl?sid=914095&cid=24784787 [slashdot.org]
(even though I have never tried it myself)
Add PAR2 files (Score:2, Informative)
It is based on the same recovery tech that RAID systems w/ parity drives use, and is mostly used to repair Usenet downloads. I usually put 4GB of data and 400MB of PAR2 recovery files on the disc. This will allow ~10% data loss before recovery is not possible. Also I dont have to worry about the TrueCrypt vol becoming damaged and unusable as well.
Try the CDFreaks Forums (Score:5, Informative)
The obsessed people at CDFreaks can help. Here's a link to their FAQ on CD-R media:
http://club.cdfreaks.com/f33/media-faq-61943/ [cdfreaks.com]
In other places in the cdfreaks forums, you'll find links to tools that can read the C1/C2 error rates. One of the simplest is "readcd", part of the "cdrecord" programs on Linux.
In the DVD world, Lite-On and Plextor both make proprietary programs to read the media-level error rates which only work with their own drives. Lite-On has a Linux version of theirs.
Professional Advice (Score:5, Informative)
What others say about is CDCheck is true, use it along with this advice. Use Plextools Pro on a PX-716 drive if you can find one. It seems to be more accurate than Nero tools. Use Plextools to check the C1, C2, and CU rates. If the graph is half-way to the top of the reading, back that disc up. As cheap as media is, I suggest burning more than one copy, storing the image on an external archive hardrive. When burning, don't use overburning. You lose some integrity for error correction.
Store your media in a cool dry place, on it's side. Avoid humidity, light, and heat when you can. Remember, the best analysis tools in the industry are very expensive for individuals. Take a look at CATs if you are interested in learning more about optical media testing. Best wishes!
use CD/DVD speed (Score:5, Informative)
http://www.cdspeed2000.com/ [cdspeed2000.com]
You need certain brands of optical drives, but with them and this program (and others), you can see the PI/PO or C1/C2 correction (I can't remember which is for CD and which is for DVD) rates on a per-sector basis on your disc. As the rates rise, the disc is going bad, becoming marginally readable and you can copy the disc before it becomes unreadable.
You can find out which drives to buy at http://cdfreaks.com/ [cdfreaks.com]. The terminology on there for a drive that can do this is a "scanning drive".
I have no idea if you will find that your correction rates are rising over time.
Re:Ask people in the music industry... (Score:5, Informative)
CDs aren't burned for commercial distribution, they're pressed.
Re:It's ok... (Score:5, Informative)
http://dar.linux.free.fr/ [linux.free.fr] try dar. It's like tar but for disks. it also generates PAR files (FEC data) which can help rebuild damaged media.
Comment removed (Score:5, Informative)
Re:Ask people in the music industry... (Score:4, Informative)
Commercial CDs are not burned. They are stamped. [wikipedia.org]
Re:Try the CDFreaks Forums (Score:2, Informative)
Re:But where to buy TY? (Score:4, Informative)
http://www.supermediastore.com/ [supermediastore.com]
That's where I got mine. I haven't had a bad burn yet.
Re:use CD/DVD speed (Score:3, Informative)
I've used it with two different Liteon SATA drives: a Liteon 20A3S and a Liteon 20A1L. Both of these drives (and I believe, Plextors) support scanning for jitter. When you run CDSpeed, the test you want is the Disc Quality tab. Click Advanced and then check the DVD Jitter checkbox. This test will give a good an indication of the quality of the disc.
In my experience, Verbatim Datalife Plus (media code: MCC) are really good discs. CMC Magnetics media vary widely, some are okay, others are garbage.
Re:par2 (Score:4, Informative)
Problem with par2 is that it does not work well with lots of files spread out across directories, and it especially doesn't work recursively. Though par2-ing your image is still probably a pretty good idea.
My partial solution so far is to make a hidden directory and hard link *every* file on the disk into that directory, then run par2 in that "flat" directory. (I put the inode number in the filenames to make sure there aren't any name collisions. If you use the inode number AS the name, you won't get any dupes on multiply linked files)
I haven't got the details worked out yet, though, but iso9660 supports hard links, so the disk burns and verifies well, and (importantly to me) unlike dvdisaster, it's not just a technically usable image, but an actually standard-compliant image.
I'm not quite sure how I'll go about restoring if verify fails, though, since the goal is to keep the same directory structure (presuming it's still mostly intact) and repair the files themselves. Perhaps unionfs over the loopback mounted iso would work. Or just repairing the files and using the a recursive directory listing (also saved on disk) to regenerate the directory structure.
But this would all be less necessary if the standard ECC in the CD format was more generalized: i.e. you could set it to always use the entire disk, no matter how little actual data you had, and just use extra layers of ECC that take advantage of the extra space. It would be good if reading could still be done transparently (like current CD ECC) and report "goodness" of the disk as determined by how much of the parity data was actually needed (so you could keep track of degradation over time, if any, and replace disks if the numbers start increasing)
Re:I highly recommend using Archival Grade Media (Score:5, Informative)
Taiyo Yuden is excellent media, as is most Verbatim media.
To answer a sibling question: no, Taiyo Yuden doesn't make dual layer discs. Verbatim does, but they aren't up to the quality of their single-layer discs.
Re:dvdisaster (Score:5, Informative)
Of course, if the data lost is in the catalog so I can't even find my files, then things get much more complicated. But even so, I've had to use this system a few times (due to damaged DVDs mostly) and it's worked pretty well.
Re:Mod parent up! (Score:3, Informative)
Re:If drive makers were SMART, they would... (Score:5, Informative)
The Exact Audio Copy tool does something like this. (Or, it used to... I kicked the Windows habit a while ago, so I'm not really up to speed)
It only works with Audio CDs to my knowledge, but it will read each section of the disk twice, and if they are identical, it will commit that and move on. If they're not identical, it will read 8 times and look for 4 matches that are identical, and if it finds them, it will commit that. Otherwise, it will read 8 more times and try again, until it's successfully found a match or until it's tried 80 times, at which point it will log the section as corrupt and move to the next sector.
I don't see why it shouldn't be possible to use a similar technique for data disks.
Re:I highly recommend using Archival Grade Media (Score:5, Informative)
I'm not an expert by any stretch, but burnable CDs are all made by using a laser on a reactive film of dye. If the laser sits on a certain spot, it changes the dye a different color. Thus, it imitates the pits from a normally pressed CD/DVD.
The problem is that over time, this dye begins to break down. Your pits stop looking like pits, or non-pits start looking like pits, etc.
At this point, I'd feel better about magnetic retention (on a hard drive) than a burnt CD/DVD, though I'd take a pressed CD/DVD over either (especially if you aren't mucking around with it and scratching it up and whatnot).
I don't think there's a silver bullet for backup and archival at this point. You need to try a lot of different ways, depending on your circumstances. For my home use, I just make sure I have data on redundant servers with mirrored drives on the servers themselves. For the most part, I'm probably good. If my house gets hit by lightning and the surge protectors fail and the harddrives essplode, I'm screwed. If my house burns down, I'm screwed.
But, if a single drive fails, I have a backup on that particular system and I know I need to make sure the data between systems is (mostly) synchronized.
Of course, things I thought were absolutely essential to be backed up for all eternity 3 or 4 years ago no longer means anything at all to me, so I'm not sure this entire process is even worth it for me anymore. I'm sure the next time I go on a power-saving kick I'll end up powering one of the file servers down, or maybe just get a few large harddrives and shove them in my main PC and power them both down.
But I digress. At length.
Re:I highly recommend using Archival Grade Media (Score:3, Informative)
Re:I highly recommend using Archival Grade Media (Score:4, Informative)
I have three methods of backup for the most important files (~100GB worth).
If that were still not good enough, bigger, more 'important' things are going on in the world to have to worry about where my data is.
Re:dvdisaster (Score:3, Informative)
Not true of DVDs, which to my understanding can deteriorate even without damage.
I've bought new-in-box DVDs that were defective, without any visible damage. :(
Comment removed (Score:4, Informative)
Re:par2 (Score:1, Informative)
PAR 1.x worked as you describe, on a per-file basis.
PAR 2.x divides all data across multiple files into blocks. If 75 KB worth of data is corrupt on a group of files including a 2 GB file, a 1 MB file, and a 5 KB file, you only need about 75 KB worth of PAR2 files to correct everything. Of course it is extremely slow to break everything into very small block sizes and generate the appropriate PAR2 files, so you'll probably have 1 MB or 5 MB blocks (or whatever) to deal with instead. In the example, if that 75 KB worth of bad data across three files lies in four separate blocks, and you have 5 MB blocks, it will take 20 MB worth of PAR2s to fix the problem. This dedication of storage space is still favorable to retaining multiple redundant copies of the original data, as long as you can accept the trade-off of time spent in the generation of the parity files.
Re:dvdisaster (Score:3, Informative)
10 cents is pushing it, I'd say more around 13 cents per gigabyte, but DVD media sits happily at 5-6 cents a gigabyte. Of course there are obvious differences, but my mantra is "burn and forget." I don't need to be bothered taking up hard drive space for a movie or iso I may need once a year. I just file the disc away in a large stack and fish it out if I ever need it.
Though as hard drives continue to fall in price and media stagnates (at least this generation), I will probably eventually buy a very large one and just use that. Of course if that one hard drive dies I lose a lot, but if one disc gets scratched, I only lose so much. RAID or backups only add to the per gigabyte cost.
Re:I highly recommend using Archival Grade Media (Score:5, Informative)
You're getting a couple different processes confused - writable-once and re-writable media are two very different things.
ReWritable discs ONLY (CD-RW, DVD+/-RW), use a layer of a metal alloy that undergoes a reversible phase change (crystalline/amorphous) when written (heated and cooled by different amounts at different rates). This phase change produces a very small change in reflectivity, to allow reading.
Recordable (CD-R, DVD-R, DVD+R) media have an organic dye layer that is burned irreversibly. The change in reflectivity produced when lasering that dye layer against a reflective layer is a greater change than with phase-change alloys.
Both Recordable and ReWritable CD media are also hampered by a very thin top protective layer. Microwave a coastered CD-R for a few seconds till it flashes and flakes, and see just how thin and fragile this layer is.
DVD-R/RW media is much more robust - the recordable layer is sandwiched in between two plastic layers, so I'd expect the lifespan of recorded DVDs - even based on the exact same dye - to be considerably longer than recorded CDs.
The question of whether Recordable or ReWritable media has a longer life span is one I haven't seen explored very well at all (though, and I think this was your main point), glass-mastered/stamped CD- and DVD-ROM media certainly do last longer than either.
One odd but explicable trend in Recordable media aging is that the shelf-life of a dics once written is longer than that of a blank disc. Don't get too enthused about stocking up on cheap CD-Rs or DVD-/+Rs at a sale - if you leave them sitting on a shelf too long before burning them, they will go unreliable.
shop much? (Score:3, Informative)
http://www.yuden.co.jp/us/product/pdf/mdvd_e.pdf [yuden.co.jp]
page 5, 4th col... see where it says 8.5GB? see at the top where it says DL?
Re:dvdisaster (Score:3, Informative)
Yeah, that's common enough... but even pressed DVDs are, to my understanding, inherently more fragile than they appear, and highly subject to environmental degradation. Any unsealed microhole and eventually the data layer is toast. :(
As to the inverse bell curve, I see it regularly in OEM computers (any brand) -- fairly high DOA rate, and after that... IMO they're designed to fail** due to always being on the edge of overheated. Those that survive past the designed-in time of death (3 to 5 years) usually live forever.
** Don't think so? stock top-of-the-line Dell, ran hot enough that it tended to lock up... I gave it the same very basic cooling any cheap clone gets, and its internal temp dropped FORTY degrees.