Ask Slashdot: Best File System For the Ages? 475
New submitter Kormoran writes: After many, many years of internet, I have accumulated terabyte HDDs full of software, photos, videos, eBooks, articles, PDFs, music, etc. that I'd like to save forever. The problem is, my HDDs are fine, but some files are corrupting. Some videos show missing keyframes and some photos are ill-colored. RAID systems can protect online data (to a degree), but what about offline storage? Is there a software solution, like a file system or a file format, specifically tailored to avoid this kind of bit rot?
Stone tablet and chisel (Score:5, Funny)
I prefer to chisel the 0s and 1s into a stone tablet. Very secure, no bit rot.
Re: (Score:3)
What about bit fungus?
Re: (Score:2)
Why in simple hell is this not modded +1, Funny?
Re: (Score:2, Informative)
Damn beat me to it!
Stone is secure!
Dude, if your hard drives were fine, your files wouldn't be corrupted. Keep RAID backups if you want a solution. The file system doesn't make a Fing difference.
Re: Stone tablet and chisel (Score:5, Funny)
I give you the 15
*drops one*
10 commandments...
Re: (Score:3)
You mean the two commandments, for those who still use Base-10 and not binary.
When the Jews asked Jesus which of the commandments was the most important, he condensed them into Love the Lord thy God with all thy heart and Love thy neighbor as thyself. If you obey these two commandments you won't break the spirit of the traditional 10 given to Moses.
Re:Stone tablet and chisel (Score:5, Funny)
Yes, weathering. That is why casting in bronze is vastly superior to mere chiseling in stone.
bit rot (Score:5, Informative)
zfs
Re:bit rot (Score:5, Insightful)
It's pretty sad that in this day and age, only one person has highlighted the relevance of ZFS here, and they're an AC. Someone mod parent up. RAID is borderline necessary if you don't have multiple backups, (to recover from in the event of random corruption caused by gamma rays from outer space or a butterfly flapping their wings on another continent or whatever) but so far as I know, only ZFS has built-in checksumming to detect/prevent the data corruption in the first place.
Re:bit rot (Score:5, Informative)
No, RAID Is not sufficient to prevent bit-rot. In fact, RAID can accelerate it. You see, using a redundant mode like 1, 5, 6, most controllers (software and hardware) will only read enough disks to get the data, 1 drive in the case of RAID1, N-1 for RAID5 and N-2 for RAID6 (the non-parity ones, to save a parity calculation). But the drives can return bit errors - it's rare, but it does happen (there's a undetectable fault error rate, something along the lines of 1 in 10^20 bytes read or so will have an undetected error). And this the RAID controller will happily return to you since it didn't check the redundant drives to verify correctness. And it's possible it gets written back corrupted, thus causing corruption.
You really need something like ZFS which puts a checksum on every file and verifies it, so if it does get an error it can resolve it.
Re:bit rot (Score:5, Informative)
You really need something like ZFS which puts a checksum on every file and verifies it, so if it does get an error it can resolve it.
ZFS also has its own flavors of RAID 1/5/6.
ZFS on Linux has software RAID. (Score:5, Informative)
Quote: "ZFS is capable of many different RAID levels, all while delivering performance thatâ(TM)s comparable to that of hardware RAID controllers."
That sounds good to me. I want to avoid hardware RAID because, when hardware RAID controllers fail, they are often difficult to replace.
Re:bit rot (Score:5, Insightful)
(there's a undetectable fault error rate, something along the lines of 1 in 10^20 bytes read or so will have an undetected error)
I just want to call this out because it's so important. That number, 10^20, sounds big, but considering the size of modern drives it's really not.
Randomly picking the WD 8TB Red NAS drive (WD60EFRX), which is designed for consume RAID as an example:
The spec sheet [wdc.com] says the URE (unrecoverable read error) rate is at worst 1 x 10^14 per bits read. However, that drive holds 8 x 10^12 bytes! If you were to read every single byte there is about a 64% chance that at least 1 bit is read incorrectly.
(8 x 8 (bits per byte) x 10^12) / (1 x 10^14) = 64,000,000,000,000 / 100,000,000,000,000 = 0.64
Correct my math if I'm wrong, but this should make anyone think twice about using any kind of RAID as a "backup" solution. If you have a disk fail you have a better than 50/50 chance of introducing corrupt data during the rebuild process!
Frankly, ZFS-style checksumming is the future of files systems. It has to be for any data you care about.
Re:bit rot (Score:5, Funny)
(there's a undetectable fault error rate, something along the lines of 1 in 10^20 bytes read or so will have an undetected error)
I just want to call this out because it's so important. That number, 10^20, sounds big, but considering the size of modern drives it's really not.
Vhrist, you guys. Why so p[aranoid? FAT has been workking just fine since day one, and there's not reason to beliveve it won't keep workingn that way for
Slash rot (Score:5, Insightful)
Concur. File corruption due to "age" will not occur without hard read errors. Also, "ill-coloured photos" likely would not be ill-coloured in the case of actual data corruption, but would have whole blocks of hash in them. The user claims to have multiple terabyte sized hard drives - hard drives in this size category userd for archival storage are simply not old enough to be suffering data corruption due to age. The only hard drives suffering so are MFM hard drives that likely the poster wouldn't have a clue how to even interface into a current computer. Hard drives used for archival data storage will likely not age degrade before the interface standard they are based on becomes obsolete. Thus, a perfectly reasonable archival data storage strategy is to simply copy data from one hard drive to a newer (likely much larger and faster) drive when the next generation interface becomes standard, and before the previous generation is totally obsolete. For example, one can still get PATA + SATA USB adapters, SATA + M.2 adapters, etc.
If the user who submitted this question is actually experiencing a problem at all, suggest that PEBCAK. Better explanation is the poster is not actually experiencing current problems at all, but is simply trying to sound important with inflated claims of reams of data and that Slashdot has been had.
Further, no person with Slashdot posting authority should have been ignorant of any of the issues in this question that make its legitimacy questionable at best, and certainly not Slashdot worthy in any circumstance.
Re:Slash rot (Score:5, Insightful)
Re: (Score:3)
Re:bit rot (Score:5, Informative)
Whose to say zfs will be around in a few decades?
The real solution here is relatively frequent backups, multiple copies in different filesystem and physical formats (ie. flash, hard drive, optical). Over time you just keep moving your file store to the new mediums. I have files that are over twenty five years old now, some of them coming from DOS and Windows 3.1, others from my old original Slackware 3 installs. Along the way some of those files have been on CD-Rs, DVDs, early USB thumb drives, various hard drives running everything from FAT, FAT32, ReiserFS, HPFS, NTFS, ext2 and ext3. And I'll keep on doing that until I drop dead, and I'll leave it up to my family to decide whether they want to keep any of the documents, pictures, music files, videos and so on that I've been collecting.
At no point do I ever assume a mere file system sitting on one physical and/or logical volume is ever going to do the job of keeping my files available over the long haul. RAID and file systems in all their glory are not intended for that. Multiple physical copies at multiple locations on multiple types of media, that's the only real way to assure your files remain accessible and safe over time.
Re: (Score:3)
Whose to say zfs will be around in a few decades?
Why wouldn't it be? The only things that could wipeout all implementations of a widely used format like ZFS would be nuclear war or an ELE asteroid strike. In either event, reading disk drives would be the least of your problems.
Re: bit rot (Score:3)
Multiple copies may be one solution, but it introduces another problem that doesn't have an elegant solution... you need a tool that can verify the integrity of your data (across the multiple copies). How do you choose which one is "correct" when you migrate and copy to a new system? In addition, how are you sure that any given copy is actually complete? What if you want to permanently delete a file from your archive?
I mitigated some of these problems for my photo library by using version control softwar
Re: (Score:3)
Re: (Score:3)
Re: (Score:2)
ZFS was build to be run on big server, with lot of ram, with battery protected raids. On whatever OS it runs, it tried to use lot more resources than other Filesystems, specially if you enable dedupe and compression. It is a good filesystem, but lot of people think that it is a good general filesystem for all users, where it is not. It can be used, but it is not light, it works best in dedicated fileservers
Re: (Score:2)
Btrfs and ZFS have metadata and data checksum support.
XFS has only metadata checksum support.
Re: (Score:3)
I have been using XFS for many years and have found it to be quite reliable and have been able to recover data when the underlying data store got corrupted. It's also quite mature in Linux and relatively fast. My last experience with BTRFS was a failure (several years ago) due to it being incredibly slow when there were thousands of small files in directories. Once ZFS is stable in the Linux kernel I'll give it a try.
Re: (Score:3, Insightful)
Tell me about a usable linux distribution that has a fully working zfs implementation.
I should have an answer for you shortly. Say, in half a decade or so, give or take.
Re:bit rot (Score:4, Informative)
Re: (Score:2)
As of Ubuntu 16.04 it's in Ubuntu's main repository. You don't even have to install another repository.
If you're not Ubuntu you should be intelligent enough to add a repository to what ever distro you do use.
Re: (Score:2)
To take a lazy example, Ubuntu 16.04 advertised its ZFS support.
Re: (Score:2)
zfs
ZFS is a pretty good solution. Multiple NAS ZFS systems [freenas.org] with snapshots and replication are even better.
I personally like XFS in production (including LVM), but ZFS is hard to beat if bitrot is your #1 concern.
Re: (Score:2)
Like all hardware, disk drives have two states - failed and going to fail. Bitrot will also occur with long term storage, whether you notice or not.
A self-healing file system with substantial redundancy capabilities like ZFS is the obvious answer.
However, there are many ways to configure ZFS, and some configurations have better redundancy than others. A misconfigured system would be worse than useless because of the false sense of security. Exactly how many terabytes of data you have also matters for cre
Re: (Score:2)
We're just gonna have to wait for the great features in ZFS to be re-implemented in some other filesystem, free of Oracle's clutches.
Re: (Score:2)
storage v noses on earth
Eh?
Re:bit rot (Score:5, Informative)
He said a filesystem for the ages. While it has wonderful features, ZFS isn't even a filesystem for this age, let along ages to come. FAT32 and ISOFS are your best bets for being readable 20 years from now.
Bear in mind that your hard disk checksums each block and returns an error if the block is uncorrectable upon read rather than give you bad data. So, if you're getting bit rot at all then you have a hardware problem.
With or without a hardware problem you want to be able to recover your data. The answer is par2, such as parchive or QuickPar. Par2 uses a Reed-Solomon code to take a set of source files and produce a set of recovery files such that the original files can be checked for correctness and up to N original files can be corrected where N is the number of recovery files created.
And that's your answer. A filesystem like FAT32 or ISOFS that's likely to still be implemented in future OSes and a recovery files which let you rebuild anything that suffers from bit rot.
Re:bit rot (Score:5, Informative)
"ZFS isn't even a filesystem for this age" - WTF does that even mean?
It means that even back when FAT was a johnny come lately it already had greater market penetration than ZFS. With decades behind it and broad market penetration today, there's good reason to believe it won't vanish with the advent of the next development in filesystem architecture. ZFS is likely to be a blip on the radar, a pause before the next innovation. Not what you want for an archival format.
Bit-rot is an issue inherent to any storage medium
Bit rot, aka corrupted data, is not inherent to correctly operating hardware. As implemented, you'll see tens of thousands of unreadable blocks on a hard disk before you see a single one in which data has been undetectably corrupted. Every single sector gets a checksum in hardware and if the checksum does not pass you get the famous Abort Retry Ignore. For most storage you get Forward Error Correction coding so that some number of bit errors can be corrected on read before having to throw an error.
When you see bit rot, the storage media is usually not at fault. More often the data passes through faulty non-parity ram, a noisy memory bus or an overheated controller and gets corrupted on its way to storage rather than getting corrupted at rest on the storage. It died when you used an overclocked piece of garbage to copy it from an old hard disk to a newer, bigger one.
Re:bit rot (Score:4, Informative)
Bit-rot is an issue inherent to any storage medium
Here's a quick article which explains how hard disks use error correcting codes so that the user-level experience is no bit rot but rather many many read failures before even a single block of undetectably corrupted data. Next time you can know what you're talking about.
http://www.pcguide.com/ref/hdd... [pcguide.com]
Terabytes over decades on NTFS (Score:3, Interesting)
Re: (Score:2)
Not a single example in 30TB over 20 years? I think you should check again.
Re: (Score:2)
I tried downloading an old attachment (6-7 years ago now) from my gmail account but the attachment is corrupted. No matter how many times I download it or to what computer, it's corrupted. I wonder what Google is using?
Re: (Score:2)
I tried downloading an old attachment (6-7 years ago now) from my gmail account but the attachment is corrupted. No matter how many times I download it or to what computer, it's corrupted. I wonder what Google is using?
What type of file is it? It might be a media format the player software no longer recognises (find an older player). Or if it is an exe it might be a 16 or 32 bit exe that won't run in a 64 bit environment. (find an older operating system). If it's not confidential, could you post a link so we can try it?
Re: (Score:2)
...'ve got somewhere between 20-30 TB that has been accumulating for more than 20 years on NTFS...
Given what appears to be Microsoft's strategy slowing morphing away from [consumer] OS's, I'd be reluctant to need to rely on Microsoft for anything long-term.
Re:Terabytes over decades on NTFS (Score:5, Insightful)
Schrodinger's bit rot. If you never look in the box again after putting the cat in it, you can pretend it lived forever.
HAHA! (Score:2)
Thanks for that one!
Clay pots in the desert (Score:2)
Re: (Score:2)
LOL this... Had someone a while back want data stability for a millenia, including the system to read the data. The conclusion we came to was carve it in marble or in fired ceramic, including the instructions for building the data reader in plain text.
HDDs are NOT fine (Score:3)
If the bits on your drive are changing while the drive is offline, that isn't a filesystem issue. A filesystem issue would be if your OS wrote the wrong information to the drive, but that can't happen with an offline drive.
Re:Filesystems with CRCs... (Score:2)
It looks like there are (at least) two with CRC: zfs and btrfs. Here's info for btrfs CRCs: https://en.wikipedia.org/wiki/... [wikipedia.org]
You'd still need a backup or RAID solution to replace a bad black.
Re: (Score:2)
It looks like there are (at least) two with CRC: zfs and btrfs. Here's info for btrfs CRCs: https://en.wikipedia.org/wiki/ [wikipedia.org]... [wikipedia.org]
You'd still need a backup or RAID solution to replace a bad black.
If only Slashdot posts had CRC or something like that, the posts wauld say what the poster intended.
RAID (Score:4, Informative)
Still RAID is a good choice for your redundancy of choice.
Or paper: http://ollydbg.de/Paperbak/#1 [ollydbg.de]
Re: (Score:2)
Not all RAIDs are equal (Score:2)
Re: (Score:2)
How about DNA? (Score:2)
Re: (Score:2)
"Our colleagues from ETH Zurich did a test and found that the half life of DNA after a chemical treatment can be 4000 years in room temperature, much better than my CDs!"
Re: (Score:2)
yeah, but now you can't update wikipedia til you have a kid. And if mom and dad have conflicting edits, watch out...
Error correction codes. PAR2, btrfs, partitions,VM (Score:5, Informative)
The magic phrase to Google is "error correction codes" (ECC).
PAR2 uses Reed-Solomon error correction. parchive is the ECC file format specification, for Linux you will want PyPar or par2tbb, and on Windows you use a GUI called QuickPar.
Btrfs can be set to use ECC on a single disk.
You can slice a single disk into partitions and then use RAID1 or LVM mirroring, or RAID5 or RAID6. LVM can alao be useful to divide (and combine) any number of drives into any number of volumes, then you can RAID across the volumes.
If you Google "ecc disk", "ecc backup", or "ecc archive" you'll find other options, with details about each option.
Re:Error correction codes. PAR2, btrfs, partitions (Score:5, Informative)
QuickPar on Windows is long-obsolete. MultiPar [vector.co.jp] is the more modern variant.
Re: (Score:3)
I agree on PAR2, simply because it's a file you can easily copy around, take backup off and so on. From a 1GB file I have ~3000 source blocks and ~30 recovery blocks, so I can recover from a lot of bit flips or failed 4kb sectors for a 1% size gain. If it's a photo set I usually make sure I can recover at least one completely missing photo. The nice thing is that it's sufficiently overkill you can probably go through several hardware generations without checking/repairing before you accumulate an unrecovera
ext4 (Score:2)
ZFS and lots of redundancy (Score:5, Informative)
ZFS will guard against bit rot. That's not enough. RAID isn't enough. You need redundancy outside your home or office. Cloud maybe expensive for the amount of data you have, but Amazon S3 maybe the most affordable in that range. You could get S3 for maybe $15-20 a month if you have a terabyte of data. If that's cost prohibitive, rotate external drives regularly and keep one at work. You'll lose very little data since you're archiving things.
Re: (Score:2)
ZFS will guard against bit rot. That's not enough. RAID isn't enough. You need redundancy outside your home or office. Cloud maybe expensive for the amount of data you have, but Amazon S3 maybe the most affordable in that range. You could get S3 for maybe $15-20 a month if you have a terabyte of data. If that's cost prohibitive, rotate external drives regularly and keep one at work. You'll lose very little data since you're archiving things.
AWS S3 pricing is $0.023/GB or $23/TB/month.
But for infrequently accessed data, AWS Glacier offers the same durability of S3 for only $0.004/GB or $4/TB/month. There's an infrequent access tier in between those two for $12.50/TB/month.
Volume discounts kick in above 50TB.
Re: (Score:2)
But for infrequently accessed data, AWS Glacier offers the same durability of S3 for only $0.004/GB or $4/TB/month. There's an infrequent access tier in between those two for $12.50/TB/month.
Volume discounts kick in above 50TB.
Online.net's C14 service [online.net] is even cheaper, at EUR 0.002/GB/month plus EUR 0.01/GB for "operations" (such as creating an archive from the temporary staging area, manually verifying archives on demand, or recovering an archive), and offers the same 99.999999999% durability as Glacier. No bandwidth costs and no complicated retrieval speed costs like Glacier, and you can use rsync to upload to the staging area. Naturally, they perform behind-the-scenes error checking and repair, but the manually-selected verifi
Any Linux FS (Score:3, Interesting)
I'd go for any Linux file system because Linux is the platform that evolves the least. It's still in the 90s so in 2037 it will still be current.
(Watch out of the hater storm! Here they come!)
But it's kinda true if you omit the snideness of the first statement. Because it's maintained by the user base, it's less likely to "devolve" into something incompatible due to market pressure. I, myself, would go for an Apple file system but Apple isn't so keep in keeping the Mac current and it doesn't bode well for the future. There might be a great change in the horizon.
"some photos are ill-colored" (Score:4, Informative)
Re: (Score:2)
That's a well known problem to photographers, photos colors are affected over time. Keep the photo negatives in a safe place!
That struck me as odd too. If the colours in digital photos or movies don't look right, I would try to display them with different software. It's more likely that the software that displays is reading and interpreting the format of the file differently than bit-rot would only affect the colour pallette and not make the whole file unreadable.
Re: (Score:2)
That struck me as odd too. If the colours in digital photos or movies don't look right, I would try to display them with different software. It's more likely that the software that displays is reading and interpreting the format of the file differently than bit-rot would only affect the colour pallette and not make the whole file unreadable.
Or the OP is using a different monitor. It doesn't matter if the new monitor better or worse than the old one. If it is different and the photos are adjusted for the old monitor, it will look "off".
Backblaze: SMART metrics of imminent failure (Score:2)
Backblaze made a report of what SMART drives they see indicating imminent drive failure: https://www.backblaze.com/blog... [backblaze.com]
Lots of parity (Score:2)
No media is perfect. There's just varying likelyhood of error rates over time, depending on the quality of the media. Without knowing ahead of time whether a specific piece of media is going to fail, the question needs to change from "How do I keep it from getting corrupted" to "How do I mitigate eventual corruption?"
And the question basically boils down to one answer: redundency.
Off the top of my head, I can think of three things you can do, and these are not mutually exclusive.
1. Multiple copies of dat
ZFS (Score:2)
"Is there a software solution, like a file system or a file format, specifically tailored to avoid this kind of bit rot?"
Yes, ZFS is specifically tailored for this. Configure a zpool running RAID-Z2 with a hot spare or RAID-Z3. Half a dozen 6TB or 8TB disks should suffice.
Set it to auto-scrub regularly. Send logs and warnings to your email, and pay attention to them. (This is the hard part). Especially pay attention if they stop arriving. (This is even harder).
I have used Nexenta for some time, but the free
Online (Score:2)
'Forever' is a long time.
'Offline' is difficult to deal with long-term (i am thinking decades to centuries) such is the nature of technology and the lack of any real history we have of digital data management,
Personally I would say the best bet is keeping your data 'live' online to some extent, it is the only real way to monitor and control the inevitable decay.
Basically your data's lifespan is related to how long you can convince someone to care for it for you.
Different objectives mean different solutions (Score:2)
Pick your poison:
- Tape: inexpensive and slow, require frequent testing (backup we do, it's restoration the problem!), usually unreadable after 6 to 12 months or less (that's in production people).
- WORM: more expensive than tape and just as slow, work well in the medium term (meaning 10 years top).
- XFS NAS: faster than the above, require good hardware and a bit more work than either tape or worm. Don't forget to setup replication to multiple systems. May suffer from bitrot in the long term (checksumming/h
Re: (Score:2)
usually unreadable after 6 to 12 months or less
What kind of crappy tapes do you you use? We've restored DLT tapes after 7 years in Iron Mountain.
this doesn't make any sense (Score:2)
Use permanent storage. (Score:2)
HDDs will die. If you want something that will last for many decades or even centuries without getting corrupted then you need to stop using a volatile filesystem. The best option is to go with write once media. The best option I know is M-DISC.
M-DISC's design is intended to provide greater archival media longevity.[3][4] Millenniata claims that properly stored M-DISC DVD recordings will last 1000 years.[5] While the exact properties of M-DISC are a trade secret,[6] the patents protecting the M-DISC technology assert that the data layer is a "glassy carbon" and that the material is substantially inert to oxidation and has a melting point between 200 and 1000 C.[7][8] -- Wikipedia
Re: (Score:3)
HDDs will die. If you want something that will last for many decades or even centuries without getting corrupted then you need to stop using a volatile filesystem. The best option is to go with write once media. The best option I know is M-DISC.
M-DISC's design is intended to provide greater archival media longevity.[3][4] Millenniata claims that properly stored M-DISC DVD recordings will last 1000 years.[5] While the exact properties of M-DISC are a trade secret,[6] the patents protecting the M-DISC technology assert that the data layer is a "glassy carbon" and that the material is substantially inert to oxidation and has a melting point between 200 and 1000 C.[7][8] -- Wikipedia
Did you even bother reading the wiki you linked to or did you just copy and paste the first paragraph ?
"However, according to the French National Laboratory of Metrology and Testing at 90 C and 85% humidity the DVD+R with inorganic recording layer such as M-DISC show no longer lifetimes than conventional DVD±R.[11]"
Snapraid (Score:3)
ZFS is nice I use it it makes assumptions about sane gear that are not safe on desktop grade hardware. BTRFS I also use works great. But for your specific use case snapraid is the thing to use. By that use case things that never change a big pile of files you keep adding to. Mind you your going to have to replace drives over time.
What you might need (Score:2)
A archival optical format. M-DISC DVDs and Blu-ray are theoretically able to retain data for 1000 years. And DVD uses some error correcting codes already, Reed-Solomon I believe.
An SSD is a bad choice for archival, in some cases MLC Flash can decay and accumulate errors in 3 months while unpowered [extremetech.com].
For a file system that is likely to be understood in the distance future, ISO 9660 with no file larger than 2 GiB should do the trick.
Packing your data into a custom archive file format that has more sophisticated
Here's how I'd do it. (Score:2)
1. Add lots of redundancy in the form of PAR2 files.
2. Store the whole lot as a tar format, dumped to the drive as a block device. This format is so simple that a future programmer will have no trouble reverse-engineering it, even if all documentation has somehow been lost, and there are no key structures which will render the whole thing impossible to read if lost. Just to be sure, the first thing going on there is a copy of the tar format specification.
3. Include also a copy of the par2 software for sever
Delete shit (Score:2, Insightful)
Seriously, minimalism is underrated. There is such a thing as too much useless data. It's hard to catalog, it's hard to track, and if you sat down and sorted out what you actually could still use, most of it is probably worthless or you'd never find the time to use ever again. You might ask "well it's still worth storing IN CASE I ever find a use for it", but that's a typical data-hoarder sentiment that is unsustainable. You can't just keep buying media to store everything and never delete, it's a managemen
Re: (Score:2)
I knew someone in college with over 9,000 1.2M 5.25" floppies worth of
If he was smart, a least half of them were used for backup.
RAID if you must, but cloud is better (Score:2)
Just RAID it (preferably mirroring)store multiple redundant copies, physically separated. Either use a checksumming filesystem (i.e. zfs) or make your own checksums so you can recognize bitrot.
But you'll never know when things have degraded beyond recovery, .
Unless you're prepared to regularly validate that the data is still readable, you'd be better off storing the data at any major cloud vendor and let *them* verify integrity over time. Or better, mirror the data across multiple cloud providers.
My most im
Reed-Solomon Erasure Coding (Score:2)
How about getting rid of it? (Score:5, Insightful)
You've got terabytes of information you will never access again. How about just getting rid of most of it? Pick some subset you want to keep and then buy 3 HDDs and create triple copies of it Repeat this every year and you'll probably not lose any of the information.
Re: (Score:2, Insightful)
I completely agree. Someday you will die. Maybe .01% of what you have stored will be valuable to your posterity (including photos and videos). After two generations - nobody will care except for a picture or two.
Too bad you have now dumped Terabytes of uncurrated junk on them that they will now have to spend days of time checking for anything useful. Do your posterity a favor and save the useful stuff and junk the rest.
By the way, if you don't, this will repeat with your children, and childrens children
Re: (Score:3)
It's worse than that. A friend of mine is in the estate sale business. He and his partner have been doing it, along with sidelines in collectible art and furniture, for close to 30 years, and cater to a who's-who list of local old money families.
Unless you are an extremely serious collector of high value objects, about half your stuff will sell for 10 cents on the dollar and the rest will go to the landfill. Vintage silver service? Valued at the melt value of the silver.
I helped him move stuff to the du
I think this as well.. but then, I think again... (Score:3)
First off, people in this thread seem to think that all the information people are saving is about them.
I have lots of data as well, but most if it is about my kids and family. My kids are starting to hit their teens, and we still go back and watch old family videos. Most are short, under 2 min. They capture points of times in our lives that we won't get back. Our old house, in a different state... friends we had then, neighbors. Not like full documentaries on them, but fading memories. It's not impor
BTRFS another option (Score:2, Informative)
in addition to ZFS, BTRFS also handles bitrot. I'm running a 4 disk BTRFS RAID 10 in my closet, mounting to a development machine on my desk via NFS, it's been working fine for about a year, and I scheduled a scrub a couple times a month whose purpose is exactly this, to catch and correct bitrot. It does so by using a CRC32 check, and if it detects a problem on one slice it overwrites that slice from the data on the good slice.
Also I have offline and offsite backups of very important items.
When using BTRFS
OFFLINE Storage, with FS Access (Score:5, Informative)
That a job for Linear Tape FileSystem
https://en.wikipedia.org/wiki/... [wikipedia.org]
Tape is (still) the best medium for Long Term Storge. Over the years tape (or more likely, the engineers) has agresively incorporated in the standards things like FEC codes (from reed-solomon to more exotic ones nowadays).
And since 2010, with LTFS, you can aceess the files with the convenience of a normal filesystem (but bear in mind, access is slow as hell).
Back up your data to tape (more than one set), and send it to specialized offline storage facilities (cimate controlled: ie. temperature/humidity/dust/light control) from different providers, in diferente geographical areas.
Since now there is only one true-tape standard (LTO-7 released in 2015, the tape business has been shrinking, so the proliferation os standards seems to be over now), so, if you use that today, chances are you will still find equipment to read it 50 years from now. Nonetheless, keep a few (as in two or more) SYSTEMS (Computer+Drive+SW) set up so that you can re-read. A cheapo micro formfactor mobo with an Atom Pocessor (but NOT the Atom C2000series PLEASE), linux, a 1Gbps nic and a tape drive should be more than enough. ....
Now, for Online, as other posters have said, ZFS WITH ECC memory (and therefore, a very expensive Xeon, or AMD server type mobo) and JBOD will do the trick.
SnapRaid (Score:2)
Its not the only solution of its type, but it is imo the best:
http://www.snapraid.it/ [snapraid.it]
It is perfect for your kind of situation - long term, reliable, efficient storage of lots of data that seldom changes. Think of it as offline RAID backup, it works like RAID, but it computes parity during your backup operations "offline"..
The beauty of it, imo, is that is is not file system dependent. It works with NTFS, EXT2, HFS, whatever. It works on Linux, Windows, Macs, whatever. You don't need special controllers, and
Paper Tape (Score:2)
Paper Tape - As long as you don't damage it, it will never suffer data loss.
Check your RAM (Score:3)
A lot of bit rot is actually caused by faulty RAM.
When data is moved around, it has to go through RAM, and even smart filesystems like ZFS may not help you there. Servers usually have ECC memory for that reason and ZFS explicitly recommends it.
Letting go is the answer (Score:3)
Re: (Score:2)
Tape suffers from bit rot.
And tape standards themselves also suffer form obsolescence. QIC-80 format, anyone?
Re: (Score:2)
Tape drives will store your stuff for upwards of 10 years, up to 30 if you store them really well. They're also available in large sizes and is pretty cheap (about a cent per GB).
And if you believe any of that, I have a very interesting investment offer for you...
Re: (Score:2)
You obviously know nothing of enterprise tape drives.
Re: (Score:2)
That's easy: use an external USB controller. You can still buy cheap PATA-USB interfaces, and of course SATA and M.2.
USB has been around 20 years, and it could be another 20 before we lose USB 2.0 / 3.0 compatibility.
Re: (Score:3)
USB has been around 20 years, and it could be another 20 before we lose USB 2.0 / 3.0 compatibility.
Before that we had FireWire 400/800 and SCSI I/II/III. Won't be long before Apple obsoletes USB 1/2/3 for something with a much smaller connector.
Re: (Score:2)
Is this even possible long term? What would have happened if you stored all of your information on PATA drives 10 years ago, its rare to find a motherboard with PATA on it now, yes there are converters and 3rd party PCI cards, but those are eventually going to dry up too.
Now, say you choose SATA, what happens when M2 becomes the defacto standard? So, why dont you choose M2? What happens when M2 is phased out?
It is not just the file system and the data you need to think about, its the physical hardware too. With the rate things change in hardware, and connecting that hardware to other hardware, its unrealistic that you could expect to be able to use your current storage media in 10 years, let alone 20, 30 or 40 years.
This is the problem with maintaining your own hardware, and a really useful use case for cloud storage, so long as you can trust the provider to keep the hardware up to date while your files stay clean, private and available.
Re: (Score:3)
This is the problem with maintaining your own hardware, and a really useful use case for cloud storage, so long as you can trust the provider to keep the hardware up to date while your files stay clean, private and available.
If you want to keep your data private, get it off the Internet. No cloud provider can guarantee your data will stay private, much less clean and available.
Re: (Score:3)
I've had a theseus' ZFS pool that I started years ago on a set of PATA drives. RAID-Z2 on OpenSolaris. It's since moved to SATA drives, been expanded a few times, moved from Debian to FreeBSD to now FreeNAS.
Setup a pool with the level of redundancy you need and as technology changes use a system compatible with the old and new tech and just replace drives as needed.