Best Way to Back Up Photos and Video? 642
jsalbre writes "I do a lot of digital video work, and my wife is a professional photographer. With raw DV from the video camera using up 11GB/hr, and raw images from the digital SLR using 7MB I'm quickly using up a lot of space. I currently back up all my important files each night from one harddrive to another, but I now have over 200GB of irreplaceable data (more than just DV and photos, but those make up the largest chunk) and I'm having to exclude the "less important" irreplaceable files as my backups have started failing. Several people have suggested backing up vital unchanging files to DVD (video, images,) and continue backing up frequently accessed files to harddrive, but with recent studies showing that optical media doesn't last very long I don't want to come back in a few years and find that all my backups are useless. Not to mention that some of my DV files are larger than even a dual-layer DVD, and it would be near impossible to automate backup to DVD. How do other Slashdotters back up their important data? I'd appreciate distinction between methods for frequently accessed files and for infrequently accessed files. Any suggestions will be highly appreciated!"
Re: Backups (Score:3, Interesting)
Why not make two optical backups. Store at least one in a fireproof safe. For the massive files, you might have to invest in one or two hot swappable drives you can use as 'tapes', storing one in your safe. Mirroring might help.
My methods. (Score:5, Interesting)
I have a 'cheap' system (sub 500) that acts as my data server. It houses 3 DVDrom drives, and a DVDRW drive, as well 1 200 GB drive. (the processor speed and ram really aren't too important, but for curiousity, it's an athlon 2000+ with 512 meg of ram). It runs gentoo, and I essentially pull the files to burn to DVD over the network weekly, and I keep the stuff I don't access alot on DVD, and the stuff I do access alot on HD -- but I primarily use the HD for holding images waiting to be burned.
USB HD (Score:5, Interesting)
Re: Backups (Score:5, Interesting)
First that comes to mind is Tape backup. They store huge about of data, and are very cheap these days, and have been proven to last for a while. Keep a good backup schedule, and keep one copy of the tapes offsite.
Secondly, I'd do optical. Optical's cheaper, but it's also not as long lasting, and takes longer to make the actual back up.
Thirdly, I'd do RAID. Mirror all the files onto a second set of hard drives. If you really want to get paranoid, mirror onto two sets of drives, and once a week swap out a copy of mirrored drives from a fireproof location.
If your data is truely irreplacable, then this is a good regiment. But it's also very expensive.. so you'll have to make up your mind.
One thing I've never understood... (Score:5, Interesting)
Something about that seems horribly backward.
That said, Exabyte still rocks my socks
Re: Backups (Score:3, Interesting)
Wouldn't it be ironic if paper backups were to become the way of the future.
permastor (Score:3, Interesting)
http://www.permastor-us.com/ [permastor-us.com]
RAID+LVM snapshots (Score:3, Interesting)
I use RAID to defend against hardware failures trashing my data, and I use logical volume management snapshots to protect against most user errors.
Neither is perfect. Some hardware failure modes could theoretically kill two or more of my four hard drives at once, which would destroy my data. Large power surges are the most likely danger, so I use a high-quality surge protector. I consider the remaining dangers unlikely enough to accept the risk.
Snapshots are also imperfect. When you create a Linux LVM snapshot volume, you have to specify how much storage is allocated to it. If changes on the source volume exceed that snapshot capacity, the snapshot stops storing the deltas and the snapshot becomes effectively useless. However, the most likely way that I might screw up and trash my data is by deleting large numbers of files. Since deleting files only updates the blocks that store the directory and inode data, not the contents of the files, a relatively small snapshot partition would hold the changes from deletion of all the files on the source. Now, if I were to accidentally run "shred" on bunches of files... I'd be screwed. I choose to accept that risk, too.
Although the RAID+LVM combo doesn't do quite as good a job as "real" backups, its failings are pretty minor, and unlikely, and it's advantage is huge: I don't have to think about it. I don't have to mess with lots of removable media and I don't have to remember to do backups.
The one thing I still worry about is some sort of catastrophe that destroys my whole system. Suppose my house burns down, for example. I'd lose it all. So I still need to find some way to get offsite copies of the most important stuff.
Disasters (Score:5, Interesting)
Re:Parity Files (Score:2, Interesting)
Re:Don't know if this will help but... (Score:3, Interesting)
Another thought I have is that since computer hardware tends to either fail right away or last a while, it may not be a bad idea to stick potential backup harddrives in a system for a month or so to weed out the bad ones.
Re: Backups (Score:3, Interesting)
Probably because mini-DV holds about 13GB and an LTO has a capacity of 400GB. Get a 4 tape autochanger and you've got 1.2TB, or about 92 Mini-DV tapes.
RTFA -- some discs do work pretty well! (Score:2, Interesting)
Also, keep your discs in the dark at controlled temp and humidity if possible.
Re:Tape Backup? (Score:1, Interesting)
DAT is better, but not perfect since the head moves like a VCR.
If you buy nicer tape drives like DLT, you can read an entire tape without a failure. The DLT tapes we have are bonded by IBM to last 20 years if stored properly. If 16 years from now (we wrote them 4 years ago) the tapes fail, IBM will give us $10k per bad tape. Try to get that guarantee for the optical crap that's sold today.
Do what Google does (Score:1, Interesting)
1. Buy the cheapest $/GB ratio disks you can find (250GB today)
2. Attach as many of them as you can to any machine(s) you own.
3. Don't use any kind of raid. Google uses GFS (replicating file system), but you can do a poor man's GFS using rsync.
My setup: DELL SC400 with 9 hard drives hanging off of it: 3 IDE inside, 4 SATA + 2 USB outside. The SATA drives run on a separate ATX power supply. USB have their own power. 7 of the drives are 250GB. I organize my data in sets (photos, video, music, books, software, projects, notebook backups, etc). I wrapped a script around rsync which allows me to assign a replication factor to each of the sets separately (depending on importance and storage requirements), but most sets are replicated 3 times. There is one master copy which I mount remotely. The others are replicated.
Advantages: each disk is independent (you don't rely on raid hardware or presence of any other disks to get to your data); you can grow it incrementally; SIMPLICITY (fewer things to break); you can dedicate one or two disk as emergency copy (in case of fire just grab two extenral USB disks)
Disadvantages: slighly more expensive than more cost effective RAID 5, but at under $0.50/GB it hardly matters.
*Eats hat* (Score:3, Interesting)
I take back what I said before. I could have sworn it was the other way around.
Never mind.
Re:this is actually a BIG question (Score:5, Interesting)
In the digital world, there's currently no such thing as an archive. There are backups that last for quite some time, but I seriously doubt any of them will last forever. The only reason any of these backups last so long is because the people creating them put some serious effort into keeping the data safe - and even then, what's to say it's not going to fail tomorrow?
You're right about the 21st century becoming a second dark age. Half the time, it proves extremely difficult to find web-published articles from two years ago, never mind what someone was putting on the web 15 years ago. Servers come and go as those involved become disinterested with the media they created. But, the difference between a print magazine going belly up and a dotcom media source going belly up is that the printed magazine will still exist while the data from the dotcom will likely never be accessible to the public again.
In the case of personal media, digital is a disaster. My grandparents still have stacks of photos documenting their entire lives, as do my parents, as do my parents for me. However, my photo collection currently suffers a gap which will never be recovered, specifically 1997-2000. During those years, I used a digital camera, and I left the photos on a working hard drive for safe keeping - alas, when I went to retrieve some files off of the drive when I wanted to go back and read a paper, I discovered the drive had committed suicide in a year without use. Yeah, that sucks.
Currently, the best way to back up data is RAID - and that's not even backing the data up, it's just making it more persistent. When you move to another machine, move all of the data to the new RAID. Repeat forever. To be extra safe, have a backup RAID just in case the first one suffers from a catastrophe.
Why is digital media troublesome? Books rarely render themselves unreadable while sitting on shelves, and are likewise rarely destroyed when dropped. Carving something into rock requires a bitchin' act of god to get rid of. But the deleting of a file, or the death of a hard drive, can wipe vast amounts of history out of existence, both in a personal and societal sense. Without an ability to permanently archive digital data, none of the data from the digital age will exist in the future.
Re:this is actually a BIG question (Score:2, Interesting)
You wrote:
Currently, the best way to back up data is RAID - and that's not even backing the data up, it's just making it more persistent. When you move to another machine, move all of the data to the new RAID. Repeat forever. To be extra safe, have a backup RAID just in case the first one suffers from a catastrophe.
Persistence is a good word. However, even a massive RAID with perfection data recovery and error correction isn't going to work. Why? Fashion, Moore's Law, and politics.
Fashion? Mac v. Windows.
Moore's Law? More data on smaller drives doesn't gurantee compatibility of the data on the new OS that can run the new memory - this goes to my earlier point on file format and OS compatibility.
Politics? RI/MPAA's favourite TLA: DRM.
Imagine it's 2505. The metal's been mined out of the planet so much that it's actually cheaper to mine the landfill mounds. the metal is often oxidised, unless it's buried deep - then it's in better shape. In 2505, the population of San Francisco is about 50,000 - the great die off of the mid 21st (yes, this means you and me - probably from the bird flu, starvation, nuclear war, or some asshole who figures out how to make AIDS airbourne) and early 22nd century cut human numbers way back, especially when the oil cost more to drill out of the ground than it could be sold for.
Our garbage picker, we'll call her "Maria" is about 150 down in the tunnel, pulling out aluminium cans, plastic bits of detritus, and lo and behold: a CD in PERFECT condition!!!! Still in its shrinkwrap (which has disintegrated a bit - it's browned and stuck to the CD case, but still there...) she takes it with her at lunch to show to her boss. The artist is a female, blond hair, with clearly enhanced breasts (this makes Maria roll her eyes as she mutters "what the FUCK were these people thinking? Oh - nemmind - they WEREN'T, and that's why this planet is such a fucking wasteland...) and she is dressed in tight clothing made of petroleum distillates. Her name is Britney Spears, and this record came out in 2017, at the height of the oil panic, during the Chinese-American cold war (the USA lost and was forced to devolve into smaller states viz. the Soviet Union, under its debt of 35 trillion dollars).
Her boss, Vaskez, says "Sure Maria - bring it to the Museum - see if they have any use for it."
So she hops on her bicycle and scoots over to the Museum. they are pleased with her find. The Museum tech, Jemmy, set up a solar panel and let it charge for a few hours (it's old and the batteries aren't/can't be made anymore) and then go into the recesses of the basement and fish out a CD player. They blow the dust off, and see that this old boombox was built in 1999. Not too bad! supposedly, it still works. They pop it in, and it fails to boot. Why?
DRM. The CD sees the old player as an "illegal device" and refuses to load. Because the CD wasn't a "classic" CD, but an enhanced DRM CD. So much for Britney. Jemmy notices Britney's inflated breasts and thinks "mmmmmm...breasts..."
Maria goes back, dejected.
THAT, my friend, is the Future of Digital Media. Crap to be dug out of a landfill in 500 years, that no one will properly comprehend, much less be able to experience.
Maria later found a VHS tape, but the acetate backing of the tape had long since rotted away. She saw an old tape play once when she was a child - it wsa the last working tape and the only example of actual media from the early 21st century left in country of Pacifica. The entire museum's electrical budget for a week was blown on a 3 hour extravaganza to watch a movie called Star Wars. It was in English, which no one there understood. they understood the story, and found it "typical" and "mythic" and not really worth the expense. The tape self destructed as it was played. They had no way to record the contents - electronic devices such as that no longer existed.
It's ALL going away. Soon.
RS
Re:To expand (Score:2, Interesting)
How to Backup and/or Archive huge amounts of data. (Score:2, Interesting)
Secondly I am a little disappointed in the answers you have gotten so far. Both in seriousness and in quality.
And as a remembrance to the first point, whenever you read TSM in my reply below, keep in mind there might be other software or solutions that could be of help...but I think myself TSM might be the best solution for you.
At first a small setting of the environment: you and your wife work with large files, and with a fair amount of total data. So in the end, you need high quality software and hardware that enables you to move all your data on all your computers offsite and/or to newer storage hardware. This hardware will be costly, but as you and your wife are professionals, you should take into account any extra costs for backing up and archiving as occupational costs.
If you do not have the amount of money needed...then you will always end up with solutions that are not answering all the needs you pointed out.
TSM will manage the data of multiple computers. These will need a network connection to the TSM server, which might be located on any one of your computers (it doesn't uses much CPU, but it will eat up 1 GB of memory though). So likely you could use any available old/spare computer you likely possess, as a DV professional I expect you to buy faster computers on a regular basis.
You likely want to keep track of your backups which is about storing different versions (that is, when editing a file you create a new version) of the same file over a certain period. A restore should likely be possible for say some version of some time ago. And sometimes you would like to be able to group some files together for backup and versioning, as the one without the other maybe pointless. And whenever a files becomes obsolete (say after ten versions or any old version after a defined period), it should be removed from your precious and expensive storage
You also definitely want to archive which is storing a partical version of a file for a certain period (maybe forever).
This is all possible with TSM and the right hardware. Have TSM perform backups regularly (scheduled daily or moreoften if you want) or manually started in between whenever you need one. You can chose what files to backup, and what not. You can also create archives with TSM, which will be stored for any period you define.
But then it becomes serious: what will you be using for hardware? With TSM you need a storage library which can store all versions in your backups and archives online, so you can access it without hassles. And likely you want to use removable media so TSM can make duplicates of your data which then may be brought offsite (any vault or shelf in another location).
I think you have two possibilities: either a tape library or a optical library. There is really not much against optical libraries if only that tapes like LTO3 may store 400 GB uncompressed (800 GB compressed), which exceeds any optical device by far. But do not compare filesize to device storage as TSM will split any large file over any amount of opticals or tapes if needed.
Lastly, you and your wife are a DV/photo professionals, not specialists into business continuity. So you likely want a one-stop solution that is easy to work with, uses TSM and a library, and is pre-setup. You then might take a look at www.storserver.com. They have solutions with disk-based storage only, or additional tpae libraries etc. etc.
There might be others I do not know of, please consider an IBM Business Partner for more help in this.
In the end: such storage is much more expensive compared to USB-HDs. But these won't last forever, do not keep track of different versions in your backup, manage your archives and it won't keep all your backup and archive versions online.
Still, you could use maybe a single USB disk for each of the seven days for backup, and use additional USB-HDs for archives. As soon as your daily data exceeds the USB-HD capacity, you come into trouble again.
Clarify requirements (Score:1, Interesting)
1. house burning down
2. disk crashing
3. PC being stolen
4. accidentally deleting files
etc.
Personally, I think RAID-1 is great for (2) and use rsync backup to an offsite machine to guard against (1) and (3). I don't care much about (4). But I'm dealing with much smaller amounts of data than the questionner.
Re:Parity Files (Score:3, Interesting)
Re:Why is that backwards? (Score:4, Interesting)
Anyway the big problem with optical is that you can only store 4.7 gigs on a DVDR, which is nothing to this guy. HDD's and Tape are the only possible solutions for this guys problem. I'd go with two HD's on firewire or USB2.0 and storing atleast one of them off site at the end of the day. Tape can be ok too but what is the seek time like on todays tech? If he is looking for one clip is he going to have to ff through the whole tape?
The answer to me seems like some form of software raid setup for write once only to external HD's.
Re:Th old fasion way (Score:3, Interesting)
Don't laugh. That's exactly what used to happen in the early, early days of film in the US. For copyright protection, films had to be printed out frame by frame and deposited in the Library of Congress.
In more than one instance, the original film was lost but the paper prints survived - so people just rephotographed the paper prints to make a new copy of the film.
http://memory.loc.gov/ammem/edhtml/edppr.html [loc.gov]
Re: Backups (Score:3, Interesting)
All this I wrote in bash in one afternoon.
Optical backup (Score:3, Interesting)
And burn a second set of DVD's for actual use, so you don't have to break the seal on the others.
Also, as history shows, storage media will continue to grow in size and decline in price. In five years, he will probably want to re-archive everything, anyway, to condense it down.
Re:To expand (Score:3, Interesting)
Actually, I Frame only MPEG-2. Not uncompressed. Uncompressed is 270 Mbit/s
Re:To expand (Score:3, Interesting)
DV/miniDV has a bitrate of 25Mbps
DVCPRo has a bitrate of 50Mbps
HD DV (if you can find one) has a starting rate of 100mbps