How To Verify CD-R Data Retention Over Time? 303
Peter (Professor) Fo writes "I've recently had two CD-Rs reported to me as faulty which are just 3 years old. This is worrying — I suspect the failure rate for this batch could be 10%. When researching CD longevity there is old and unreliable information; pious 'how to cosset your discs so they last 100 years' blurb; and endless discussions of what sort of dye to use, don't use cheap media, burn slower (or don't), but not much by way of hard facts besides there's a lot of data loss going on.
Does anyone know of a generic utility (win or *nix would suit me) that can map sector readability/error rates of CDs? I'd like to measure decay over time in my environment with my media and my other variables; and I expect others would too."
I highly recommend using Archival Grade Media (Score:5, Insightful)
It's ok... (Score:1, Insightful)
just hash the cd and you are golden. Like my apples during the fall.
Re:The tools are called (Score:5, Insightful)
dd and diff.
Those tools provide no signal-to-noise ratio (Block Error Ratio, BLER) for physical media errors that the drive is just barely correcting. The point of the request, as I understand it, is to detect how likely a correctable medium is to stay correctable.
Re:not possible (Score:4, Insightful)
Mod parent up! (Score:5, Insightful)
I have many gigs of digital photos and I have also more-or-less moved away from optical media for backup and switched to HDD. As the original poster mentioned, most of the "information" you find on the net about archival longevity of optical media is personal anecdotes or pet theories, and good hard data on archival longevity of CD-R or DVD+-R is hard to find. My own personal experience is that name brand discs do have fewer problems than cheap "house brands", but it's hard to quantify or say much beyond that.
Backing up to hard drives has a number of advantages:
1. It's a heck of a lot easier - in most cases of personal data backup, a few 1TB HDDs will hold all the data you need to back up, so there's no need to manage boxes of 100's of discs. I usually back up the same data onto two HDDs, and store one of them in a firesafe. If you're really worried, you can store one of them offsite.
2. Since no media will last forever, you will *always* need to roll your data over to new media every so many years. With HDDs, its *much* easier to roll your data over to new media every 5 or 6 years. Think of transferring two or three HDD's to a new HDD (by the time you roll over the data, the new HDD will probably hold all the data from those two or three older HDDs), compared to re-organizing and re-burning hundreds (or more) of CDs or DVDs.
The bottom line is that if a few HDDs don't hold enough data for your needs, then backing up to optical media will be totally out of the question anyway, and you will probably need to use tape.
If drive makers were SMART, they would... (Score:5, Insightful)
Short of an electron scanning microscope, the only way to do it is to hook an oscilloscope directly on to the test points within the drive itself and measure signal levels.
Or the manufacturer of an optical drive could do the SMART thing: provide some sort of self-monitoring, analysis, and reporting tool [wikipedia.org] to let the user see how many errors the drive has corrected per MiB of data. Mobile phones, Wi-Fi cards, and digital TV converter boxes do something like this, showing SNR in "bars" or in percentiles.
A hard-drive remains the most reliable (Score:2, Insightful)
I think that the most reliable way to backup your data is to use a USB hard drive changed every 1/2 years.
Actually I use two : one at my parents in Europe, one at my place in Canada.
A hard drive is much more reliable than any CDR/DVDR, and if your data is important it's worth it.
Just my two cents.
David
Re:par2 (Score:5, Insightful)
Re:Tape (Score:3, Insightful)
Tape isn't better than optical, necessarily. All backup mediums require testing to ensure reliability, which generally means restoring from backup.
If your data is critical, keep it in several places at the same time, and as closely synced as you can manage. Hard drive, mirrored to another hard drive, backed up to an external hard drive, swap out external drive for another every week (stored off-site), run incremental backups to tape nightly, use an online backup service like rsync.net or Carbonite. Even then you're not assured of complete data integrity, but it's not too shabby and not horrifically expensive.
If your jobs can be archived onto CD/DVDs, then doing that is one more safety net. If you don't want to go through a long, drawn-out process of running CRC checks or whatever on the optical media, simply burning it twice on two different computers is an option. If that's not possible, burning it twice on two different known-good brands is better than nothing.
Re:par2 (Score:4, Insightful)
I've used par2 for this purpose, and it's not bad. It doesn't support multiple directories -- you have to create separate parity data for each directory on the CD. It also has no support for restoring any filesystem metadata, only the file names and contents.
I've also used dvdisaster, and I think it has some advantages. It creates a single block of error correction from the entire disc image, so it includes any filesystem information on the disc. It can use existing media in the drive, or an .iso file. The error correction data can then be appended to the .iso file before burning (assuming you've calculated the size correctly.)
I'd really like to see dvdistater's method become a standard feature of CD burning software, with the presumption that most people would want to add error correction to their discs if there's free space. Operating systems could check for this data when reading a CD and automatically use it to detect and correct read errors.
I'd really love to see this system adopted by software companies, music labels and movie studios -- but of course they'd much rather have a shot at selling you another copy of the disc you scratched.
Different Media (Score:4, Insightful)
Sadly to say, the 100 years nonsense is for pressed CDs like you get music and programs on. Burnable ones last maybe 5-10 years, tops. When you add in literal bit-rot due to fungi and so on that exist and love to eat worm-trails in the media surface itself... I've had CDs go bad in as little as a year or two. I constantly have to re-burn my media every couple of years. Thankfully the media density gets better, so I can toss an entire collection of CD-Rs onto 2-3 Blu-Rays and be done with it.
If you want it to be secure, the only viable solutions seem to be flash media or an old-school hard drive in storage. Thankfully the prices of both are affordable for your critical data. All of my critical data and installers and so on fits in a single 512MB flash drive. Toss that in a safe deposit box and forget about it.
Re:I highly recommend using Archival Grade Media (Score:3, Insightful)
Archival Grade Media makes a HUGE difference for backing up important data. It is not very expensive and widely available.
Do you have any proof that the media makes a difference ? I recommend covering them in honey so that they stay sweet for 100 years.
Re:I highly recommend using Archival Grade Media (Score:5, Insightful)
Re:dvdisaster (Score:5, Insightful)
Stupid question, but why is the poster still using CDs for data? Hard drives are down to 10 cents per gigabyte, so why would anyone take the time and data risk to still burn information to CDs? I'm slowly moving away from even DVDs.
Good idea but (Score:3, Insightful)
Re:dvdisaster (Score:5, Insightful)
Great idea!
For a less technological approach, I simply BUY whatever CD or DVD I want to keep.
I thought about doing that, but wasn't able to find any of my source code or family photos on DVD at my local store. Maybe I'm just looking in the wrong place. Where do you shop for yours?
Re:dvdisaster (Score:3, Insightful)
One reason to still use CD/DVD is to easily keep different copies of a file with reduced risk of ALL data getting lost at once.
A HD may easily become a single point of failure.
Example:
Say you have 2 drives.
Drive A - Current data
Drive B - Backup data with multiple versions.
If drive B dies you potentially loose all your backups at ONCE.
For very important files a combo of HD and CD/DVD (or a cloud service) probably produces the best protection with relatively ease of maintenance.
Re:dvdisaster (Score:3, Insightful)
Are you serious? CDs are a useful way to distribute a large group of files through a method other than e-mail (or "the cloud," though I kinda hate that term). It's especially helpful if the information needs to be available for a while... That is, rather than storing the files somewhere on a server where things can get lost or moved, sometimes having a physical CD is just a better option.
By your logic, if I wanted to send 400MB of files to, say, 50 people, you'd have me buy 50 hard drives? Or even 50 USB flash drives?
The Internet may be making medium-sized file transfers (in the 50MB-to-5GB range) a lot easier, but sometimes an indirect transfer by CD or DVD is still a better option. Sure, the Internet transfer method is only going to get easier over time, and CDs will eventually go the way of floppies. But that time is a lot farther away than you seem to think.
Re:dvdisaster (Score:3, Insightful)
RAID IS NOT BACKUP.
Raid doesn't protect you from rm -fr. Backups do