How To Verify CD-R Data Retention Over Time?

Peter (Professor) Fo writes "I've recently had two CD-Rs reported to me as faulty which are just 3 years old. This is worrying — I suspect the failure rate for this batch could be 10%. When researching CD longevity there is old and unreliable information; pious 'how to cosset your discs so they last 100 years' blurb; and endless discussions of what sort of dye to use, don't use cheap media, burn slower (or don't), but not much by way of hard facts besides there's a lot of data loss going on. Does anyone know of a generic utility (win or *nix would suit me) that can map sector readability/error rates of CDs? I'd like to measure decay over time in my environment with my media and my other variables; and I expect others would too."

dvdisaster

You should probably try dvdisaster [dvdisaster.net]. it can test media, and can create (on disk or external) redundancy data, which can be used to recover later.
It's also open source, so you could probably coerce it to export some more information

Re:dvdisaster

Very nice. I use par2 [sourceforge.net] for basically the same purpose. I save about 10% of my DVD capacity, and have a program that creates a directory with md5sums of every file, along with par2 files for all the files, so I can recover from a loss of almost up to 10% of the disk's data.

Of course, if the data lost is in the catalog so I can't even find my files, then things get much more complicated. But even so, I've had to use this system a few times (due to damaged DVDs mostly) and it's worked pretty well.

Re:dvdisaster

"You should probably try dvdisaster..."

Stupid question, but why is the poster still using CDs for data? Hard drives are down to 10 cents per gigabyte, so why would anyone take the time and data risk to still burn information to CDs? I'm slowly moving away from even DVDs.

I highly recommend using Archival Grade Media

Archival Grade Media makes a HUGE difference for backing up important data. It is not very expensive and widely available.

Re:I highly recommend using Archival Grade Media

Also looking for Taiyo Yuden brand helps (or anything made in Japan...that should be TY just rebranded). Every knowledgable board I've looked over on the subject has recommended them, and I have never had a single one of their disks out of probably 200 now be bad from the start. Unfortunately I don't have any 'old' disks to test. I keep most of my data backed up on hard drives.

Re:I highly recommend using Archival Grade Media

Taiyo Yuden is excellent media, as is most Verbatim media.

To answer a sibling question: no, Taiyo Yuden doesn't make dual layer discs. Verbatim does, but they aren't up to the quality of their single-layer discs.

Re:I highly recommend using Archival Grade Media

I'm not an expert by any stretch, but burnable CDs are all made by using a laser on a reactive film of dye. If the laser sits on a certain spot, it changes the dye a different color. Thus, it imitates the pits from a normally pressed CD/DVD.

The problem is that over time, this dye begins to break down. Your pits stop looking like pits, or non-pits start looking like pits, etc.

At this point, I'd feel better about magnetic retention (on a hard drive) than a burnt CD/DVD, though I'd take a pressed CD/DVD over either (especially if you aren't mucking around with it and scratching it up and whatnot).

I don't think there's a silver bullet for backup and archival at this point. You need to try a lot of different ways, depending on your circumstances. For my home use, I just make sure I have data on redundant servers with mirrored drives on the servers themselves. For the most part, I'm probably good. If my house gets hit by lightning and the surge protectors fail and the harddrives essplode, I'm screwed. If my house burns down, I'm screwed.

But, if a single drive fails, I have a backup on that particular system and I know I need to make sure the data between systems is (mostly) synchronized.

Of course, things I thought were absolutely essential to be backed up for all eternity 3 or 4 years ago no longer means anything at all to me, so I'm not sure this entire process is even worth it for me anymore. I'm sure the next time I go on a power-saving kick I'll end up powering one of the file servers down, or maybe just get a few large harddrives and shove them in my main PC and power them both down.

But I digress. At length.

DVD-RAM too

I use dvd-ram [wikipedia.org] to archive important files. Designed for archival type storage, the slower media has a 30 year designed life, the faster media has something like 5 year. Add in the builtin ecc and cheap cost, it is a good way to save my source code and photos.

Re:I highly recommend using Archival Grade Media

IF you go through the time to test them once a year, when not just make new copies of them every year? OR every 3rd year?

par2

Whenever I back up important data, I use par2. If the disc has I/O errors, I just make a full image with dd_rescue (skips past bad blocks, whereas dd will just halt operation) and run "par2 verify" on it. If it's really important, I always verify the integrity no matter what (I've even done it on discs 2 days old, and sometimes, due to the reliability of CD/DVD-R media, it even has errors to repair).

Re:par2

If it's really important, I always verify the integrity no matter what

I do that even if it's not important. I have a script which creates an md5 checksum file for a directory tree and adds it to the directory, and I always run it before burning a CD or DVD. Once burned, I verify the checksum on two different computers.

There have been a few times that the computer that burned the disc successfully verified a new disc, but a different one didn't. When that has happened, I trashed that disc and made a new one.

Sometimes I wonder if a lot of the reports of "deteriorating discs" are actually cases where someone burned a coaster in the first place, and just never happened to try to read or verify the data until years later.

Re:par2

The problem with md5, is that it does nothing to repair broken data. It's great when you download something and find out it's bad so you try again, but for long-term data storage, knowing that your data is corrupt doesn't do you a whole lot of good. Which is why I recommended par2, it can both verify and (more importantly) repair data.

Error-Proofing Data With Reed-Solomon Codes

The following tool allows you to track the failure rate of your media, and allows you to recover the files and replace the cd/dvd when it starts failing. http://hardware.slashdot.org/article.pl?sid=08/08/03/197254 [slashdot.org]

Try the CDFreaks Forums

The obsessed people at CDFreaks can help. Here's a link to their FAQ on CD-R media:

http://club.cdfreaks.com/f33/media-faq-61943/ [cdfreaks.com]

In other places in the cdfreaks forums, you'll find links to tools that can read the C1/C2 error rates. One of the simplest is "readcd", part of the "cdrecord" programs on Linux.

In the DVD world, Lite-On and Plextor both make proprietary programs to read the media-level error rates which only work with their own drives. Lite-On has a Linux version of theirs.

Professional Advice

I work with CD/DVD and related technology as a profession. I analyze, QC, and mass duplicate media by the thousands for extreme and critical field use, every day. My best advice to you is to use Taiyo-Yuden (TY) media, always. I've seen mixed results and bad burns from Mitsui and MAM-A gold, Kodak, and the like. The TY dye type has a proven longer longevity than any other so far. I also suggest burning all of your audio CD's at 16X, this affect what's called single-beam readers. Also, it insures higher integrity of the burn. Burn 16X DVD's at 8X to increase the write integrity.

What others say about is CDCheck is true, use it along with this advice. Use Plextools Pro on a PX-716 drive if you can find one. It seems to be more accurate than Nero tools. Use Plextools to check the C1, C2, and CU rates. If the graph is half-way to the top of the reading, back that disc up. As cheap as media is, I suggest burning more than one copy, storing the image on an external archive hardrive. When burning, don't use overburning. You lose some integrity for error correction.

Store your media in a cool dry place, on it's side. Avoid humidity, light, and heat when you can. Remember, the best analysis tools in the industry are very expensive for individuals. Take a look at CATs if you are interested in learning more about optical media testing. Best wishes!

use CD/DVD speed

http://www.cdspeed2000.com/ [cdspeed2000.com]

You need certain brands of optical drives, but with them and this program (and others), you can see the PI/PO or C1/C2 correction (I can't remember which is for CD and which is for DVD) rates on a per-sector basis on your disc. As the rates rise, the disc is going bad, becoming marginally readable and you can copy the disc before it becomes unreadable.

You can find out which drives to buy at http://cdfreaks.com/ [cdfreaks.com]. The terminology on there for a drive that can do this is a "scanning drive".

I have no idea if you will find that your correction rates are rising over time.

Titanium Plates © if your data matters

Take a page from the book of Church© of© Scientology©®(TM) and engrave your data on Titanium© Plates© and store it in Gold© Vault©. I apologize in advance to the Church© of© Scientology©®(TM) if I didnt use enough Copyright©®(TM) symbols while referring to Them (©?). Please do not sue me. © (TM) ®

Re:The tools are called

dd and diff.

Those tools provide no signal-to-noise ratio (Block Error Ratio, BLER) for physical media errors that the drive is just barely correcting. The point of the request, as I understand it, is to detect how likely a correctable medium is to stay correctable.

Re:The tools are called

Short of an electron scanning microscope, the only way to do it is to hook an oscilloscope directly on to the test points within the drive itself and measure signal levels. This will allow you to measure one or both of: Degradation of the laser optics, degradation of the media. It's anyones guess as to which is which :-)

To make things a little more accurate, you should use several drives to test the media. The drives could benefit from being locked away until such a time as they are needed to repeat the tests. Mix in a few new drives when you do actually make your tests in future as well.

If drive makers were SMART, they would...

Short of an electron scanning microscope, the only way to do it is to hook an oscilloscope directly on to the test points within the drive itself and measure signal levels.

Or the manufacturer of an optical drive could do the SMART thing: provide some sort of self-monitoring, analysis, and reporting tool [wikipedia.org] to let the user see how many errors the drive has corrected per MiB of data. Mobile phones, Wi-Fi cards, and digital TV converter boxes do something like this, showing SNR in "bars" or in percentiles.

Re:If drive makers were SMART, they would...

The Exact Audio Copy tool does something like this. (Or, it used to... I kicked the Windows habit a while ago, so I'm not really up to speed)

It only works with Audio CDs to my knowledge, but it will read each section of the disk twice, and if they are identical, it will commit that and move on. If they're not identical, it will read 8 times and look for 4 matches that are identical, and if it finds them, it will commit that. Otherwise, it will read 8 more times and try again, until it's successfully found a match or until it's tried 80 times, at which point it will log the section as corrupt and move to the next sector.

I don't see why it shouldn't be possible to use a similar technique for data disks.

Re:It's ok...

RAID5 for CDs? Is there anything where I can burn 3 CDs with a 'set' of data. When I want to restore my data I just put in each disk sequentially and then it does some RAID5 magic and spits out my data?

Be a cool project, IMHO.

Re:It's ok...

http://dar.linux.free.fr/ [linux.free.fr] try dar. It's like tar but for disks. it also generates PAR files (FEC data) which can help rebuild damaged media.

Re:Ask people in the music industry...

CDs aren't burned for commercial distribution, they're pressed.

Mod parent up!

I have many gigs of digital photos and I have also more-or-less moved away from optical media for backup and switched to HDD. As the original poster mentioned, most of the "information" you find on the net about archival longevity of optical media is personal anecdotes or pet theories, and good hard data on archival longevity of CD-R or DVD+-R is hard to find. My own personal experience is that name brand discs do have fewer problems than cheap "house brands", but it's hard to quantify or say much beyond that.

Backing up to hard drives has a number of advantages:

1. It's a heck of a lot easier - in most cases of personal data backup, a few 1TB HDDs will hold all the data you need to back up, so there's no need to manage boxes of 100's of discs. I usually back up the same data onto two HDDs, and store one of them in a firesafe. If you're really worried, you can store one of them offsite.

2. Since no media will last forever, you will *always* need to roll your data over to new media every so many years. With HDDs, its *much* easier to roll your data over to new media every 5 or 6 years. Think of transferring two or three HDD's to a new HDD (by the time you roll over the data, the new HDD will probably hold all the data from those two or three older HDDs), compared to re-organizing and re-burning hundreds (or more) of CDs or DVDs.

The bottom line is that if a few HDDs don't hold enough data for your needs, then backing up to optical media will be totally out of the question anyway, and you will probably need to use tape.