Slashdot Log In
How To Verify CD-R Data Retention Over Time?
Posted by
CmdrTaco
on Thu Nov 06, 2008 11:29 AM
from the saving-boobies-for-future-generations dept.
from the saving-boobies-for-future-generations dept.
Peter (Professor) Fo writes "I've recently had two CD-Rs reported to me as faulty which are just 3 years old. This is worrying — I suspect the failure rate for this batch could be 10%. When researching CD longevity there is old and unreliable information; pious 'how to cosset your discs so they last 100 years' blurb; and endless discussions of what sort of dye to use, don't use cheap media, burn slower (or don't), but not much by way of hard facts besides there's a lot of data loss going on.
Does anyone know of a generic utility (win or *nix would suit me) that can map sector readability/error rates of CDs? I'd like to measure decay over time in my environment with my media and my other variables; and I expect others would too."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
dvdisaster (Score:5, Informative)
You should probably try dvdisaster [dvdisaster.net]. it can test media, and can create (on disk or external) redundancy data, which can be used to recover later.
It's also open source, so you could probably coerce it to export some more information
Re:dvdisaster (Score:5, Informative)
Of course, if the data lost is in the catalog so I can't even find my files, then things get much more complicated. But even so, I've had to use this system a few times (due to damaged DVDs mostly) and it's worked pretty well.
Parent
Re:dvdisaster (Score:5, Insightful)
Stupid question, but why is the poster still using CDs for data? Hard drives are down to 10 cents per gigabyte, so why would anyone take the time and data risk to still burn information to CDs? I'm slowly moving away from even DVDs.
Parent
Re: (Score:3, Interesting)
The storage footprint of data tends to be inversely proportional to its importance.
People lose sight of this fact, and because they don't see a good way to backup the 500 gigabytes of data they have, they somehow fail to backup the 20 pages of documents that they need to protect their assets or limit their liabilities.
Don't underestimate the value of a document printed in archival ink, stored in several fireproof locations. Definitely don't underestimate the archival value of, and potential longevity of, h
Re:dvdisaster (Score:5, Insightful)
Great idea!
For a less technological approach, I simply BUY whatever CD or DVD I want to keep.
I thought about doing that, but wasn't able to find any of my source code or family photos on DVD at my local store. Maybe I'm just looking in the wrong place. Where do you shop for yours?
Parent
I highly recommend using Archival Grade Media (Score:5, Insightful)
Re:I highly recommend using Archival Grade Media (Score:5, Interesting)
Parent
Re:I highly recommend using Archival Grade Media (Score:5, Informative)
Taiyo Yuden is excellent media, as is most Verbatim media.
To answer a sibling question: no, Taiyo Yuden doesn't make dual layer discs. Verbatim does, but they aren't up to the quality of their single-layer discs.
Parent
Re:I highly recommend using Archival Grade Media (Score:5, Informative)
I'm not an expert by any stretch, but burnable CDs are all made by using a laser on a reactive film of dye. If the laser sits on a certain spot, it changes the dye a different color. Thus, it imitates the pits from a normally pressed CD/DVD.
The problem is that over time, this dye begins to break down. Your pits stop looking like pits, or non-pits start looking like pits, etc.
At this point, I'd feel better about magnetic retention (on a hard drive) than a burnt CD/DVD, though I'd take a pressed CD/DVD over either (especially if you aren't mucking around with it and scratching it up and whatnot).
I don't think there's a silver bullet for backup and archival at this point. You need to try a lot of different ways, depending on your circumstances. For my home use, I just make sure I have data on redundant servers with mirrored drives on the servers themselves. For the most part, I'm probably good. If my house gets hit by lightning and the surge protectors fail and the harddrives essplode, I'm screwed. If my house burns down, I'm screwed.
But, if a single drive fails, I have a backup on that particular system and I know I need to make sure the data between systems is (mostly) synchronized.
Of course, things I thought were absolutely essential to be backed up for all eternity 3 or 4 years ago no longer means anything at all to me, so I'm not sure this entire process is even worth it for me anymore. I'm sure the next time I go on a power-saving kick I'll end up powering one of the file servers down, or maybe just get a few large harddrives and shove them in my main PC and power them both down.
But I digress. At length.
Parent
Re:I highly recommend using Archival Grade Media (Score:4, Informative)
I have three methods of backup for the most important files (~100GB worth).
If that were still not good enough, bigger, more 'important' things are going on in the world to have to worry about where my data is.
Parent
Re: (Score:3, Interesting)
Re:I highly recommend using Archival Grade Media (Score:5, Informative)
You're getting a couple different processes confused - writable-once and re-writable media are two very different things.
ReWritable discs ONLY (CD-RW, DVD+/-RW), use a layer of a metal alloy that undergoes a reversible phase change (crystalline/amorphous) when written (heated and cooled by different amounts at different rates). This phase change produces a very small change in reflectivity, to allow reading.
Recordable (CD-R, DVD-R, DVD+R) media have an organic dye layer that is burned irreversibly. The change in reflectivity produced when lasering that dye layer against a reflective layer is a greater change than with phase-change alloys.
Both Recordable and ReWritable CD media are also hampered by a very thin top protective layer. Microwave a coastered CD-R for a few seconds till it flashes and flakes, and see just how thin and fragile this layer is.
DVD-R/RW media is much more robust - the recordable layer is sandwiched in between two plastic layers, so I'd expect the lifespan of recorded DVDs - even based on the exact same dye - to be considerably longer than recorded CDs.
The question of whether Recordable or ReWritable media has a longer life span is one I haven't seen explored very well at all (though, and I think this was your main point), glass-mastered/stamped CD- and DVD-ROM media certainly do last longer than either.
One odd but explicable trend in Recordable media aging is that the shelf-life of a dics once written is longer than that of a blank disc. Don't get too enthused about stocking up on cheap CD-Rs or DVD-/+Rs at a sale - if you leave them sitting on a shelf too long before burning them, they will go unreliable.
Parent
Re: (Score:3, Informative)
Re:I highly recommend using Archival Grade Media (Score:4, Interesting)
Exactly. I have 60 archival grade CD-R
s from over 10 years ago that are still readable Yes I have tried them, it is a part of our backup integrity testing every year.
Storage and handling is also very important with them as well.
Parent
Re:I highly recommend using Archival Grade Media (Score:5, Insightful)
Parent
Re: (Score:3, Insightful)
Archival Grade Media makes a HUGE difference for backing up important data. It is not very expensive and widely available.
Do you have any proof that the media makes a difference ? I recommend covering them in honey so that they stay sweet for 100 years.
DVD-RAM too (Score:5, Interesting)
I use dvd-ram [wikipedia.org] to archive important files. Designed for archival type storage, the slower media has a 30 year designed life, the faster media has something like 5 year. Add in the builtin ecc and cheap cost, it is a good way to save my source code and photos.
Parent
par2 (Score:5, Informative)
Re:par2 (Score:5, Interesting)
If it's really important, I always verify the integrity no matter what
I do that even if it's not important. I have a script which creates an md5 checksum file for a directory tree and adds it to the directory, and I always run it before burning a CD or DVD. Once burned, I verify the checksum on two different computers.
There have been a few times that the computer that burned the disc successfully verified a new disc, but a different one didn't. When that has happened, I trashed that disc and made a new one.
Sometimes I wonder if a lot of the reports of "deteriorating discs" are actually cases where someone burned a coaster in the first place, and just never happened to try to read or verify the data until years later.
Parent
Re:par2 (Score:5, Insightful)
Parent
Re:par2 (Score:4, Insightful)
I've used par2 for this purpose, and it's not bad. It doesn't support multiple directories -- you have to create separate parity data for each directory on the CD. It also has no support for restoring any filesystem metadata, only the file names and contents.
I've also used dvdisaster, and I think it has some advantages. It creates a single block of error correction from the entire disc image, so it includes any filesystem information on the disc. It can use existing media in the drive, or an .iso file. The error correction data can then be appended to the .iso file before burning (assuming you've calculated the size correctly.)
I'd really like to see dvdistater's method become a standard feature of CD burning software, with the presumption that most people would want to add error correction to their discs if there's free space. Operating systems could check for this data when reading a CD and automatically use it to detect and correct read errors.
I'd really love to see this system adopted by software companies, music labels and movie studios -- but of course they'd much rather have a shot at selling you another copy of the disc you scratched.
Parent
Re:par2 (Score:4, Informative)
Problem with par2 is that it does not work well with lots of files spread out across directories, and it especially doesn't work recursively. Though par2-ing your image is still probably a pretty good idea.
My partial solution so far is to make a hidden directory and hard link *every* file on the disk into that directory, then run par2 in that "flat" directory. (I put the inode number in the filenames to make sure there aren't any name collisions. If you use the inode number AS the name, you won't get any dupes on multiply linked files)
I haven't got the details worked out yet, though, but iso9660 supports hard links, so the disk burns and verifies well, and (importantly to me) unlike dvdisaster, it's not just a technically usable image, but an actually standard-compliant image.
I'm not quite sure how I'll go about restoring if verify fails, though, since the goal is to keep the same directory structure (presuming it's still mostly intact) and repair the files themselves. Perhaps unionfs over the loopback mounted iso would work. Or just repairing the files and using the a recursive directory listing (also saved on disk) to regenerate the directory structure.
But this would all be less necessary if the standard ECC in the CD format was more generalized: i.e. you could set it to always use the entire disk, no matter how little actual data you had, and just use extra layers of ECC that take advantage of the extra space. It would be good if reading could still be done transparently (like current CD ECC) and report "goodness" of the disk as determined by how much of the parity data was actually needed (so you could keep track of degradation over time, if any, and replace disks if the numbers start increasing)
Parent
Error-Proofing Data With Reed-Solomon Codes (Score:5, Informative)
Save them again with long lasting solutions (Score:4, Informative)
If you consider your data worth it, have enough time and enough money, you should probably re-burn/re-save them to long lasting media.
There was a previous post on askslashdot about this subject.
http://ask.slashdot.org/article.pl?sid=08/08/27/2119252 [slashdot.org]
My suggestion was to use Plasmon "Century-Disc" :
http://slashdot.org/comments.pl?sid=914095&cid=24784787 [slashdot.org]
(even though I have never tried it myself)
Handle with care (Score:4, Funny)
The only 2 reasons I have ever had a CD die.
1. Bad burn.
2. Dropped it/scratched it.
Okay, I really have only had one reason CDs die:
1. I can be somewhat of a dumb-ass.
Try the CDFreaks Forums (Score:5, Informative)
The obsessed people at CDFreaks can help. Here's a link to their FAQ on CD-R media:
http://club.cdfreaks.com/f33/media-faq-61943/ [cdfreaks.com]
In other places in the cdfreaks forums, you'll find links to tools that can read the C1/C2 error rates. One of the simplest is "readcd", part of the "cdrecord" programs on Linux.
In the DVD world, Lite-On and Plextor both make proprietary programs to read the media-level error rates which only work with their own drives. Lite-On has a Linux version of theirs.
Tape (Score:3, Interesting)
Backups? Use tape.
Optical media is inherently shitty.
If you want to get the best out of it:
Buy good media.
Burn at a slower speed.
Verify the data after burning it.
Store it well. A hard case, and a cool, dry location away from the sun are all you really need.
If you want to test the quality of a disc, go ahead and use any of the tools recommended here.
If you want to harden your discs, go ahead and use any of the CRC tools recommended here.
But really, you shouldn't be using optical media as anything other than a cheap delivery medium. If you need to send stuff to people and you need them to have a copy of it indefinitely, tell them to make a damned copy of it, or give them 2 copies, or keep an ISO and send them a copy when theirs fails.
Re: (Score:3, Insightful)
Tape isn't better than optical, necessarily. All backup mediums require testing to ensure reliability, which generally means restoring from backup.
If your data is critical, keep it in several places at the same time, and as closely synced as you can manage. Hard drive, mirrored to another hard drive, backed up to an external hard drive, swap out external drive for another every week (stored off-site), run incremental backups to tape nightly, use an online backup service like rsync.net or Carbonite. Even t
Professional Advice (Score:5, Informative)
What others say about is CDCheck is true, use it along with this advice. Use Plextools Pro on a PX-716 drive if you can find one. It seems to be more accurate than Nero tools. Use Plextools to check the C1, C2, and CU rates. If the graph is half-way to the top of the reading, back that disc up. As cheap as media is, I suggest burning more than one copy, storing the image on an external archive hardrive. When burning, don't use overburning. You lose some integrity for error correction.
Store your media in a cool dry place, on it's side. Avoid humidity, light, and heat when you can. Remember, the best analysis tools in the industry are very expensive for individuals. Take a look at CATs if you are interested in learning more about optical media testing. Best wishes!
Re:But where to buy TY? (Score:4, Informative)
http://www.supermediastore.com/ [supermediastore.com]
That's where I got mine. I haven't had a bad burn yet.
Parent
use CD/DVD speed (Score:5, Informative)
http://www.cdspeed2000.com/ [cdspeed2000.com]
You need certain brands of optical drives, but with them and this program (and others), you can see the PI/PO or C1/C2 correction (I can't remember which is for CD and which is for DVD) rates on a per-sector basis on your disc. As the rates rise, the disc is going bad, becoming marginally readable and you can copy the disc before it becomes unreadable.
You can find out which drives to buy at http://cdfreaks.com/ [cdfreaks.com]. The terminology on there for a drive that can do this is a "scanning drive".
I have no idea if you will find that your correction rates are rising over time.
Re: (Score:3, Informative)
I've used it with two different Liteon SATA drives: a Liteon 20A3S and a Liteon 20A1L. Both of these drives (and I believe, Plextors) support scanning for jitter. When you run CDSpeed, the test you want is the Disc Quality tab. Click Advanced and then check the DVD Jitter checkbox. This test will give a good an indication of the quality of
Titanium Plates © if your data matters (Score:5, Funny)
Different Media (Score:4, Insightful)
Sadly to say, the 100 years nonsense is for pressed CDs like you get music and programs on. Burnable ones last maybe 5-10 years, tops. When you add in literal bit-rot due to fungi and so on that exist and love to eat worm-trails in the media surface itself... I've had CDs go bad in as little as a year or two. I constantly have to re-burn my media every couple of years. Thankfully the media density gets better, so I can toss an entire collection of CD-Rs onto 2-3 Blu-Rays and be done with it.
If you want it to be secure, the only viable solutions seem to be flash media or an old-school hard drive in storage. Thankfully the prices of both are affordable for your critical data. All of my critical data and installers and so on fits in a single 512MB flash drive. Toss that in a safe deposit box and forget about it.
Re:The tools are called (Score:5, Insightful)
dd and diff.
Those tools provide no signal-to-noise ratio (Block Error Ratio, BLER) for physical media errors that the drive is just barely correcting. The point of the request, as I understand it, is to detect how likely a correctable medium is to stay correctable.
Parent
Re:The tools are called (Score:5, Informative)
Short of an electron scanning microscope, the only way to do it is to hook an oscilloscope directly on to the test points within the drive itself and measure signal levels. This will allow you to measure one or both of: Degradation of the laser optics, degradation of the media. It's anyones guess as to which is which :-)
To make things a little more accurate, you should use several drives to test the media. The drives could benefit from being locked away until such a time as they are needed to repeat the tests. Mix in a few new drives when you do actually make your tests in future as well.
Parent
If drive makers were SMART, they would... (Score:5, Insightful)
Short of an electron scanning microscope, the only way to do it is to hook an oscilloscope directly on to the test points within the drive itself and measure signal levels.
Or the manufacturer of an optical drive could do the SMART thing: provide some sort of self-monitoring, analysis, and reporting tool [wikipedia.org] to let the user see how many errors the drive has corrected per MiB of data. Mobile phones, Wi-Fi cards, and digital TV converter boxes do something like this, showing SNR in "bars" or in percentiles.
Parent
Re:If drive makers were SMART, they would... (Score:5, Informative)
The Exact Audio Copy tool does something like this. (Or, it used to... I kicked the Windows habit a while ago, so I'm not really up to speed)
It only works with Audio CDs to my knowledge, but it will read each section of the disk twice, and if they are identical, it will commit that and move on. If they're not identical, it will read 8 times and look for 4 matches that are identical, and if it finds them, it will commit that. Otherwise, it will read 8 more times and try again, until it's successfully found a match or until it's tried 80 times, at which point it will log the section as corrupt and move to the next sector.
I don't see why it shouldn't be possible to use a similar technique for data disks.
Parent
Re:The tools are called (Score:4, Informative)
If you want something fast that will tell you if there is any trouble reading the disk go here [e-systems.ro], or if you want something more complex that'll go by sector go here [orconhosting.net.nz].
Oh and in the future if the software needed can be Windows software,allow me to suggest Freeware World Team [all4you.dk]. FWT have hands down the best freeware search engine I have ever seen. You simply type into the search box what you want the software to do and they'll find you a piece of freeware that does it. I use it here at the shop all the time when myself or a customer has a job that needs filling. No spyware,no trialware,just great freeware with a truly great search engine to find it with.
Parent
SMART over time (Score:3, Interesting)
The request asks no such thing, he simply wants to measure decay rate.
The method of dd+diff can tell only whether the disc has decayed or has not decayed: 1 bit of information. Something that can read C1/C2 error rates, like the program Wanker mentioned [slashdot.org], gives much more information that can be used to give a better idea of how much decay has happened before it becomes unreadable. Plotting this over time gives (ta-da) the rate of decay.
Re:not possible (Score:4, Insightful)
Parent
Re:It's ok... (Score:5, Interesting)
RAID5 for CDs? Is there anything where I can burn 3 CDs with a 'set' of data. When I want to restore my data I just put in each disk sequentially and then it does some RAID5 magic and spits out my data?
Be a cool project, IMHO.
Parent
Re:It's ok... (Score:5, Informative)
http://dar.linux.free.fr/ [linux.free.fr] try dar. It's like tar but for disks. it also generates PAR files (FEC data) which can help rebuild damaged media.
Parent
Re:It's ok... (Score:4, Interesting)
This has worked well for me. I had a situation where a RAID 5 lost two drives and my backup had some corruption in a 100 gig video project. Pulled out my 3 year old DVD's and recovered the data fine (and yes there were one or two of chunks that had problems but recovered from the redundancy).
Just my method and 2 cents.
--
So who is hotter? Ali or Ali's Sister?
Parent
Re: (Score:3, Funny)
Re:Ask people in the music industry... (Score:5, Informative)
CDs aren't burned for commercial distribution, they're pressed.
Parent
Re:Ask people in the music industry... (Score:4, Informative)
Commercial CDs are not burned. They are stamped. [wikipedia.org]
Parent
Mod parent up! (Score:5, Insightful)
I have many gigs of digital photos and I have also more-or-less moved away from optical media for backup and switched to HDD. As the original poster mentioned, most of the "information" you find on the net about archival longevity of optical media is personal anecdotes or pet theories, and good hard data on archival longevity of CD-R or DVD+-R is hard to find. My own personal experience is that name brand discs do have fewer problems than cheap "house brands", but it's hard to quantify or say much beyond that.
Backing up to hard drives has a number of advantages:
1. It's a heck of a lot easier - in most cases of personal data backup, a few 1TB HDDs will hold all the data you need to back up, so there's no need to manage boxes of 100's of discs. I usually back up the same data onto two HDDs, and store one of them in a firesafe. If you're really worried, you can store one of them offsite.
2. Since no media will last forever, you will *always* need to roll your data over to new media every so many years. With HDDs, its *much* easier to roll your data over to new media every 5 or 6 years. Think of transferring two or three HDD's to a new HDD (by the time you roll over the data, the new HDD will probably hold all the data from those two or three older HDDs), compared to re-organizing and re-burning hundreds (or more) of CDs or DVDs.
The bottom line is that if a few HDDs don't hold enough data for your needs, then backing up to optical media will be totally out of the question anyway, and you will probably need to use tape.
Parent
Re: (Score:3, Informative)
Re:Nero (Score:4, Funny)
"American parts
Parent