Best Way to Back Up Photos and Video? 642
jsalbre writes "I do a lot of digital video work, and my wife is a professional photographer. With raw DV from the video camera using up 11GB/hr, and raw images from the digital SLR using 7MB I'm quickly using up a lot of space. I currently back up all my important files each night from one harddrive to another, but I now have over 200GB of irreplaceable data (more than just DV and photos, but those make up the largest chunk) and I'm having to exclude the "less important" irreplaceable files as my backups have started failing. Several people have suggested backing up vital unchanging files to DVD (video, images,) and continue backing up frequently accessed files to harddrive, but with recent studies showing that optical media doesn't last very long I don't want to come back in a few years and find that all my backups are useless. Not to mention that some of my DV files are larger than even a dual-layer DVD, and it would be near impossible to automate backup to DVD. How do other Slashdotters back up their important data? I'd appreciate distinction between methods for frequently accessed files and for infrequently accessed files. Any suggestions will be highly appreciated!"
Compression (Score:2, Insightful)
Don't know if this will help but... (Score:3, Insightful)
"duh" indeed (Score:5, Insightful)
"Several people have suggested backing up vital unchanging files to DVD (video, images,) and continue backing up frequently accessed files to harddrive"
They've already considered hard drives. Since he's dismissed hard drives and seemingly all forms of optical media, the only thing that I can think of for this article getting posted is that the submitter *really* wants Slashdot to tell him that "Yes, it's ok to mortgage the house to buy that new Network Appliance SAN you've been drooling over."
Re:Cost? (Score:1, Insightful)
Backup makes another copy that you can take to another location, and is protected against your own "rm" command.
Re:raid is not a backup methodology (Score:2, Insightful)
Re:what the article is pointing to -- (Score:4, Insightful)
Agreed. I realise tapes aren't the new hotness, but they're the most reliable, and they have good storage capacity. In addition, I'd consider a larger capacity storage server. Together this stuff may not be as cheap as tossing everything on DVDs, but apparently this is for people who work in digital media for a living. From that perspective, its worth investing in your profession.
Perhaps better than slashdot, they're bound to have a huge network of friends in the profession who have already crossed this bridge. It couldn't hurt to ask how people specificaly in these professions manage their media storage.
Re:"duh" indeed (Score:3, Insightful)
The question he really needs to answer is... what is worth more? Losing your data? Or spending $5,000 on a NAS server or a RAID machine?
Optical Reliability (Score:3, Insightful)
The NIST report didn't say optical media were inheritantly unreliable at all; it said that there were big differences in media quality, and that storage conditions were important.
Personally I think hard drives are the pits for data reliability. The drives are good for MAYBE 3 years, subject to all sorts of electrical failures, and even if you have a RAID you still can lose the whole thing due to a {virus,controller,power supply,filesystem,usererror}.
I use redundant MAM-A gold stabilized CD-Rs for my data which were the most stable option in the NIST report. That works great for everything I have including digital photos.
DV might be a pain with CD-R so I would probably start with staggered redundant sliver DVD-Rs until I saw some more data on the lifetime of this media.
No way would I consider hard drives an acceptable archival solution.
Needs (Score:5, Insightful)
This is the pot calling the kettle black, though. Is there a support group out there?
Parity Files (Score:3, Insightful)
Re: Backups (Score:4, Insightful)
So what you'd suggest is that he downloads the video from the MiniDV tape to the computer, then archives it onto backup tapes. Why not just keep the original MiniDV?
Re: Backups (Score:5, Insightful)
First, the new data may have been processed (edits, color correction, etc).
Second, the backup media may be better rated for long term storage. I'm not familiar with MiniDV, the stuff I work with is all DLT and HCART2 under Veritas Netbackup, at 200GB raw/400GB compressed per tape.
Third, it may be helpful to have the indexing done for him by a good backup program.
However, as I say, I work with Netbackup. To say it's pricey is an understatement... but it's changed my views on what's a "workable" backup system to only liking enterprise grade stuff.
Back up only the important stuff. (Score:4, Insightful)
When I backup my stills onto dvd I use jpeg 2000, its lossy but really not that bad once the image is in a good state.. I did some tests in college on jpeg/jpeg2000 vs tiff (uncompressed) of the smae image to see how much is lost. Not a lot it turns out. I love uncompressed images, but the loss when storing as jpeg isn't so great to matter unless you do a lot more manipulation. I'm also still shooting film which can always be rescanned at a later date.
However, you shouldn't backup all the DV (raw video) you dump on the computer. The original tap e can act as the backup. its still on the tape even after you dump it into the computer. Label it and set the right protect notch. Voili, instant backup footage.
I'm assuming you edit this down and give the client a dvd/video. Just keep a DVD copy for yourself. Thats all they can really ask you for. If they come back at a later date, because the dvd is bad try yours. If that doesn't work you have to go back to the tape and redit and recharge.
this is actually a BIG question (Score:5, Insightful)
I have been in the same position the Author discussed, and I have come to ONLY negative conclusions. In a few words, and I hate to say this, but buddy:
WE'RE FUCKED.
Digital is a loser's proposition. backing up to analogue or even digital data on analogic substrates (such as DV tape) fail. Simply nad purely.
The *only* thing that comes close is some kind of RAID, and those, even with the plummeting price of storage, are still too expensive given the needs.
Also, a RAID assumes a continuity of several things that are not likely to be continuous:
With Video:
Framerate, number of lines, colour depth, aspect ratio, file format, compression format, Operating system compatibility, etc etc etc. All of these things are variables.
With Audio:
sample rate, compression format, bit depth, file format, etc.
Basically all of it points to very bad places.
I am fairly well convinced that our age will simply disappear. They will find our garbage, the few books not pressed on acidic paper, our paintings (fat lot of good the abstract stuff will mean to them) and drawings, that's about it. the rest will just be shiny little bits of crap in the landfill.
Since we will have used up all the dense energy forms, they will be appalled at the energy requirements just to get the few remaining museum piece devices to work. Archiving the 21st century will be impossible. To the 25th century, the 21st century will be seen as a dark age - not only for the holocaust of the die caused by the failure of the petroleum based economy, but from the simple fact that very little of the information formats we are totally geared into will survive, including this note on /.
His problem of saving personal video is just the tip ofthe iceberg. His problem is the problem of our very civilisation, writ small.
That's why I am abandoning video, and going back to painting. In 500 years, my painting CAN survive. the video simply won't.
RS
Re:"duh" indeed (Score:2, Insightful)
Re: Backups (Score:3, Insightful)
But most of the time, it's a digital camera, where it's flash ram it's recording to. For digital video, it's most likely in need of some editing which is the whole reason to bring it into the computer anyways, which is when you need to start the backup proceedure.
Re:Tape Backup? (Score:2, Insightful)
I use Dantz Retrospect. I have two completely separate "sets" of backups. Each set gets an incremental update on alternating Fridays. When you start a set, you will get a full backup (in my case 5 DVD+R's). Incremental updates then keep getting written to a disk until the disk is full or you get paranoid and throw it in the safe.
This means that all files including a back-log of revisions are available on two completely different sets of disks. Even if both sets become defective, the software can rescue whatever files it can from the functioning media.
Every 6 months or a year, I close out the set and move them off-site. Their only purpose now is in case an entire newer set fails or in case I want a really old revision.
DVDs live at home in my fire-proof safe, at work, and eventually with a friend or family member elsewhere.
This is extremely convenient and meets the need. Any time I am nervous about the state of the discs I can toss them in and have the entire set tested for integrity.
I get really nervous about people backing up to additional hard drives or even hard drives on other systems. If your fear is simple mechanical failure, then a RAID would have been your answer. You must be anticipating malicious or accidental erasure of your content. Well if the system you store all of this on is vulnerable to these attacks, what makes you think a second drive or even a second system will be immune? Get it on a removeable medium and get it out of electrons' reach.
Re: Backups (Score:2, Insightful)
Example - I don't work for Cabelas but I order alot of shit from them so thats where I went to off the bat.
"Stack-On Fire Resistant Personal Safe
Perfect for storing valuables, documents and more. Both of these fire resistant safes are ETL verified to manufacturer's fire protection specifications. Up to 1,700 F. for one hour with the interior temperature remaining below 350 F. Solid steel, pry resistant door with four-number combination and key lock provides greater security. Holes for mounting to the floor are pre-drilled. Fastening hardware is included."
http://www.championsafe.com/tech/fire.asp [championsafe.com]
You're going to laugh, but... (Score:4, Insightful)
These are the only ones I can trust to be around in 100 years or more.
*All* digital images get written to CD-Rs are are stored in a commercial document-control facility. But the ones I really want to keep get written to film.
Re:To a second hard drive? (Score:4, Insightful)
It is to survive a hardware failure &/or increase speed. It
is not meant as a backup device to archive data. Accidentally
delete a file and its gone from your RAID. Accidentally
overwrite a file with same name, the original is toast. Lose
two drives in your RAID array, good chance your data is gone.
As others have said, optical may not be as reliable as once
thought, and is not practical for large files. While tape
suffers from drive obsolescence, the media aging rate is
fairly well known and less random than optical and can be
planned for.
Re:for video back-ups... (Score:4, Insightful)
Re:raid (Score:3, Insightful)
This common platitude doesn't hold water. While you can't delete
The rest of your post exactly the point I was making... nobody's arguing that RAID is useless... just that it isn't an effective backup mechanism, and you need to supplement it with something else.
Re:this is actually a BIG question (Score:5, Insightful)
This may be hard to believe (and I probably sound smug) but there was a time a while ago when the camera hadn't been invented yet, and nobody had any photos at all. We still seemed to survive as a race/civilisation though.
I'd mod you both down as alarmist and uninformed. (Score:5, Insightful)
Books are not such a perminant media as you might think. They wear out, and can be destoryed. A good example is the Mayan Codices. Records seem to indicate there were thousands, however Spanish priests burned them as "works of the devil" during the European conquest of the Americas. Today only 4 remain.
Digital data can be so perminant because it is so easily copied. Perminance of data does not come form trying to make a single, eternal copy, but from having many copies all over the world. Digital data can be copied for essentially zero cost very easily. Thus it's easy to give it a great deal of robustness. Also, as new formats come out, you simply copy and convert the data. I have data on my harddrive today that orignally existed on 5.25" floppy for the Apple II. It has simply been copied and converted a number of times.
Finally, it's not like book are going away. On the contrary we publish millions of works a year amounting to billions of books.
You seem to have a false sense of perminance, as though in the past things were archived forever. That's not the case, actually, most data was lost, that's one of teh reasons we have such an incomplete picutre of history. You don't even know all that was lost, because the record of it even existing, if there was one, is also lost. What has survived is by chance, or by effort, not because we had some wonderful archival system.
You don't have to have something on an immutable, indestructable medium for it to survive. The Nordic Legends weren't written down for centuries, yet today we still have them. They were passed down, as an oral traditon for generations. There was no perminance to them other than stories in people's minds, yet they've durvived thousands of years.
DV tape is cheap (Score:4, Insightful)
Alternately, all modern NLEs have 'export to tape' functions. Just record your final product back out to your DV deck or DV camera and make a master archive on tape.
Re: Modding for stupid reasons (Score:3, Insightful)
You like to talk about the bible but realise the immense effort that went in to each copy prior to the printing press. It was an amazing amount of effort to copy all of it and attempt to do it without error. These days, you spend 1/10th the effort and you'll have something far more perminant.
If I have DV that's important to me, $50 and a bit of time will allow me to archive it in several stable formats that I can then place in controlled settings.
Generally, however shit that's important isn't that high priority. I have "important" data, in that I don't want to lose it. However it's not important enough to make any serious backup attempt of. Having it on 2 seperate servers with redundant drives is good enough for me. The world won't stop turning if it's lost.
This concept that because there is data that has little perminance somehow manes all data, espically important data, has no perminance is fucking stupid. Ya, your home movie of your kid's 8th birthday party that no one cares about and that you'll never watch on DV tape will probably not survive. Important data, like weather information, will because it's copied thousands of times over to stable formats and stored.
You seem to forget that, much like
Re: Backups (Score:3, Insightful)
No.
They are a minimally acceptable backup media for short-term storage.
Consider the fact that with tapes, you really just have to worry about tape errors. If the tape drive fails, you can use another.
With hard drives, you have to worry not only about errors on the drive, but about hardware failures in the electronics as well.
In 10 years, that hard drive will probably be dead no matter what you do. But a properly stored tape backup would still be around.
Re:Why is that backwards? (Score:2, Insightful)
> sitting on the shelf in a dry enviorment it should last
> pretty much forever.
There's considerable debate of the "last pretty much forever."
How long can bearings sit before freezing up?
Fluid bearings suffer from evaporation, for example.
Hard drive platters have a very thin lubricant layer, if I recollect. How long does that last before chemical degradation?
Devices cannot be presumed to "last forever" just because they are not being used. In fact, just sitting and not being used may actually shorten their life.
(While it is not a fair comparison, try starting a vehicle that's been in storage for a couple of years - seals dry out, mechanisms gum up, and so forth.)
So, the jury is out on how long a hard drive sitting on a shelf will last.
How to back up all that data forever? (Score:3, Insightful)
Re: Backups - some comments (Score:3, Insightful)
2. I'm all about using your friend's internet connection to do this. Furthermore, in response to someone else - if you think your friend is spilling Cheerios on it.. a) get better friends and b) get MORE friends/backups. I'll take redundancy over perfection any day.
3. RAID is great in those situations where your intra-backup loss (ie, from a day) is very great.
I agree with you that a lot of people recommend RAID 10, but I think they are quite wrong OR they are using crap systems - my lengthy explanation follows.
Good RAID controllers use battery-backed write cache - that means they "accept" your write immediately and use a battery to actually put it on the harddrive LATER, even if the power goes out. This is a HUGE speed improvement for multiple small write situations, even with just ONE disk. I ignore this effect in the below discussion.
I'm going to assume a system where you have two similar drives on different buses on the same machine. I'm also going to assume that you're HD I/O bound (ie, the harddrive platters/heads are what's causing the slowdown, not your CPU) I'm also going to assume you do more reading than writing - at least more files if not more bits (which is pretty typical)
--- First, why RAID 0 is stupid (unless you're using very large files AND not using them at the same time) I'm going to compare RAID0 to just putting different stuff on different drives (for instance, OS/swap/apps on drive 0 and data on drive 1.) I'm calling this setup "noRAID"
RAID 0 is straight striping - it writes half of every file to each disk. This means that the _write_ time (time from the time it starts to the time it finished writing) is twice as fast, but the _seek_ time (time to get the head to the right place to write) is exactly the same as a single disk. For writing very large files this is almost twice as fast. For writing smaller files it is not faster at all because the seek time (time to find where to write it) totally overwhelms the time to actually do the writing. For reading the same thing is true. The "bulk" of reading a file is exactly twice as fast but the seek is not changed at all. So most of the time it really isn't faster except for really big files.
The short answer is that RAID0 is stupid because it has no benefits when seeking.
Compare this to just using 2 drives: if you try to read or write simultaneous small files that are on different disks, noRAID is absolutely _twice as fast_ If you try to write a single very large file RAID0 approaches being twice as fast as the write time becomes much larger than the seek time. Of course, the weak point in this argument is that sometimes you want two things on the same disk - then noRAID is only the same speed as RAID0 for small files. So noRAID doesn't average being actually twice as fast.
In addition, RAID0 is half as redundant because either disk failing destroys everything.
--- Second, why RAID1 is good.
RAID 1 is straight mirroring. On a modern RAID system (like Linux's SoftRAID) this gives you performance that - compared to a single disk - is exactly identical on write to a single disk (Assuming your CPU can always keep up) For multiple file reading, though, it peforms better than any other setup, even _better than noRAID_ because it only needs to read from 1 disk and it reads from whichever disk has a head in a convenient spot to do THAT read.
It doesn't have the disadvantage of noRAID, because it ALWAYS has a copy of the data it needs on the
Re: Backups (Score:2, Insightful)
If what you are trying to protect is truly a backup, and not an archive (that is, it is a spare copy, not the only copy), then it is not necessary to protect both the original and the backup from fire (or flood, etc.). It is necessary only to ensure that the hazard doesn't affect both at the same time.
If you want to store both in the same room, or even in the same house, then indeed some sort of fireproof safe would be needed. But it you can store one offsite, then no fireproofing is needed. There's defnitely no need for a safe-deposit box at a bank!
Once I simply rented an additional locker at my athletic club (in the hallway, where they were cheaper, not in the humid locker room). Another time I stored my home backups at the office, and vice versa.
At the offsite location, you may need theft protection. Hiding is the best way, but there are many safes that are very good at this, even if they're not fireproof.
Another reason for not using a safe-deposit box is that you want the storage location to be easily accessible so you will use it often. You don't want your backups sitting on the hall table for two weeks waiting for your next trip to the bank!
Re:Needs (Score:1, Insightful)
2.) maybe you're collecting information now for the day in the future when you have tools to readily cross-reference and find this data for you as useful resources. (e.g. coredata/spotlight, beagle, etc.)
external storage (Score:2, Insightful)