Best Way to Back Up Photos and Video? 642
jsalbre writes "I do a lot of digital video work, and my wife is a professional photographer. With raw DV from the video camera using up 11GB/hr, and raw images from the digital SLR using 7MB I'm quickly using up a lot of space. I currently back up all my important files each night from one harddrive to another, but I now have over 200GB of irreplaceable data (more than just DV and photos, but those make up the largest chunk) and I'm having to exclude the "less important" irreplaceable files as my backups have started failing. Several people have suggested backing up vital unchanging files to DVD (video, images,) and continue backing up frequently accessed files to harddrive, but with recent studies showing that optical media doesn't last very long I don't want to come back in a few years and find that all my backups are useless. Not to mention that some of my DV files are larger than even a dual-layer DVD, and it would be near impossible to automate backup to DVD. How do other Slashdotters back up their important data? I'd appreciate distinction between methods for frequently accessed files and for infrequently accessed files. Any suggestions will be highly appreciated!"
what the article is pointing to -- (Score:4, Informative)
Tape Backup? (Score:2, Informative)
And supplement that with LaCie external firewire drives.
Tape... (Score:4, Informative)
And as long as you store the tape properly, it should last a long time.
for video back-ups... (Score:5, Informative)
RAID 5 (Score:1, Informative)
Long-Term and Short-Term needs (Score:3, Informative)
First, there's the need to keep things around long-term. Second, there's the need to have things protected from disaster in the short term.
I once used an external firewire HD for backup, and was reminded of the importance of burning things as well when that HD went tango-uniform on me, destroying months of work.
I'd suggest looking into some sort of RAID - even just a simple mirror - for the short-term protection. That way you don't have quite as much a single point of failure that can wipe out your data, so you can do backups more because you need the space than because you need to sleep well at night.
As for the backups, optical discs are very convenient, but magnetic tape might have a longer lifetime depending on environmental conditions, and although I've seen CD-R comparisons [pcbuyersguide.com], I've yet to see something similar for DVDs.
There are times where a high-capacity removable hard disk looks very attractive. Shades of the old Bernoulli's or whatever.
(This may not be first post, though there were none when I started. Maybe I'll have to settle for first useful post.)
Re:Tape Backup? (Score:5, Informative)
Treat optical media like magnetic media (store in cool dry place) and use high-quality media and you'll get far better results than tape.
Add in the speed at which tape drives become obsolete and tapes hard to obtain, while CD's are still readable. And I've found optical to be a superior archive medium.
If you examine the study cited you'll notice that the study is for optical media in harsh conditions. Additionally they specifically state "It is demonstrated here that CD-R and DVD-R media
can be very stable (sample S4 for CD-R and sample D2 for DVD-R). Results suggest that these media types will ensure data is available for several tens of years and therefore may be suitable for archival uses."
Re: Backups (Score:5, Informative)
One thing good about paper & film is they withstand decades of storage vs. years of normal magnetic storage. Photos and films from the late 1800's/early 1900's are still around whereas you're really gambling with current storage media.
8.4 TB Tape Backup (Score:1, Informative)
If you want to save some money, an AIT 4 tape drive system works quite well. AIT 4 tapes can hold up to 520GB. A used AIT drive can be had for less than $200 on ebay and can eaily backup 260GB
Re:Compression (Score:3, Informative)
Re:Tape Backup? (Score:3, Informative)
Re: Backups (Score:5, Informative)
Re:somewhat obvious solutions (Score:2, Informative)
1. Human Stupidity, one mistaken format of the raid instead of that USB drive and poof.
2. Localized disasters, Flood, Lightning, Tornadoes, Blizzards, and Fire are all things that will can trash a raid.
3. Human malice, theft, vandalizm, hackers, viruses, worms and the like. Offline storage is less suceptable to these issues.
Storm
Archive versus Backup: Know the difference! (Score:3, Informative)
Your computer's own hard drives should keep only what you are actively working on. Get the rest of the stuff out of your way.
Buy GOOD DVDs ... burn all the files you are not actively working with to these - two separate DVDs for each archive, of two different brands. Check for file integrity, label them well and store them in a convenient, off-site location, cool and dark. Delete the originals from the working drive. Check the archive disks fairly often for degradation and re-burn as needed. They are no more labile than negatives and videotape.
For the large files, buy removable drive bays and holders, and copy them onto large hard drives. REMOVE the drives and store them with the DVDs.
On your working system, continue to back up the data for the active projects. Consider getting a RAID 5 system for data integrity, because if you back up data from one drive to another you risk overwriting a good copy with a bad copy.
To expand (Score:5, Informative)
Re:Tape Backup? (Score:2, Informative)
Do this with premium panasonic tapes, and they should last for a few years at least. Copy to new tapes in 3 years. Put the tapes in a sealed container with some form of desicant in the air tight container to absorb excess moisture.
this should pretty much ensure the DV tapes will last pretty much until there is a better way to store data.
Considering how cheap DV tapes are (for 13GB of video) its the way to go.
Re: Backups (Score:5, Informative)
In everything I've read, the moral definitely seems to be harddrives, lots of harddrives, for price performance. I'm assuming you have a reasonable LAN or can set one up.
Here's the setup I haven't finished implementing yet: PLEASE give me any comments about it to help me improve my setup.
1. Setup a file server using at least one big, inexpensive disk. (This can also be a desktop as long as it can reasonably serve files.) This is your "USE" server.
2. Separate you files (on a per-directory basis) into categories based on how frequently they are changed. The important consideration is: 'If a file is changed/deleted from USE how long should I wait delete a file in the backup' Personally, I only need two categories. "current" = a month or so depending on disk space and "archive" = never (family pics, videos, etc.)
That means that if I delete something in my "current" tree _AND_ I don't notice for a month, my backups will delete it and it's gone forever.
3. Setup a 'backup server' using at least one inexpensive hard disk. Set your backup server to login to your USE server and sync your files.
It should be able to do both "full" (copy everything) and "incremental versioning" = "IV" (if something is changed, keep BOTH copies, marking them appropriately) backups. Neither of these kinds of backups should ever eliminate any information automatically - they should just add information.
4) For me, I'd run:
1) An IV backup of "archive" every night.
2) A full backup of "current" every week.
3) An IV backup of "current" every night.
4) A job that deleted the oldest backups of current every week.
Notice that I'm _never_ running a full backup of "archive" but I'm also _never_ deleting the backup.
Notes:
rsync or rsync over ssh is my preference for doing this kind of backup. It works very nicely, but I'm too tired to get it right just this minute so I'm leaving IV/full backup commands as exercises for other
cron is fine for setting it up automatically.
wget has similar functionality to rsync for a website and you don't need any privileges.
I think most of
Do make sure you log the output of your syncing software. Also make sure you monitor disk usage. If you want to be fancy, it could keep all of the full-backups of "current" until space is short (with a reasonable margin) and then always delete as many of the oldest ones as it needs to to make enough room. This means your number of snapshots will vary with disk space - some people think that's evil.
This system scales reasonably well - for more size add more harddrives per server and/or more servers. For redundancy add more backups per live copy. As long as you can keep it organized and your network handles it, there's also no reason a USE server can't be served by two backup servers or a backup server can't also serve several smaller workstations - or any combination thereof.
Do not add multiple harddrives to a backup server for redundancy. These servers are essentially free and you get much more redundancy (and some scalability) if you use two backup servers. With a setup like this, any server should only have one copy (excepting multiple versions of the same tree)
You could just do a full backup of current every night or whatever, and you could have many possibly more complicated "current" backup schemes. But for me the total size of "current" is massively smaller than "archive" so it's really not important. Remember, having more of these isn't more redundant - they're all on the same drive.
This backup server should generally run no services except possibly ssh and certainly shouldn'
Re:RAID 5 (Score:2, Informative)
Rsync and Dirvish for disk-to-disk backup (Score:5, Informative)
Rsync ( http://rsync.samba.org/ [samba.org] is really great for backup of Unix-like systems. The ability to hardlink identical files allows me to store hundreds of daily full images of 100GB of sources to a single target 250GB hard disk. Rsync is very smart about moving only changed data over the network, resulting in speedups of 10x to 100x. This allows me to do full backup on my offsite colo without using a lot of bandwidth. Note that Rsync is great for Mac/Unix/Linux, but it does sometimes have problems with windoze clients. But then, so do I ...
Dirvish (originally written by jw schultz) is a Perl wrapper around Rsync. It facilitates the scheduling and management of Rsync based backups. We have a fairly active mailing list and contributions from around the world (open source is so cool!).
Backups should be safe against:
Backups should be automatic (or they will not get done) and cheap (hard disks are cheaper than tape, and much cheaper when you use hard linking). Rsync stores the data in a file system closely approximating the original, which facilitates restores.
If a cheap electrolytic filter capacitor dries out in your power supply, and the 5V output decides to start making a 15V squarewave instead, everything in your computer case will get fried. Including every one of the RAID disks. External USB enclosures (or airgaps!) protect against host and power supply failure.
If I was really paranoid about protecting my data, I would run a long ethernet cable to a nerdly neighbor a few houses away, and put a second dirvish server there. While I do rotate my drives into ziplok bags in a fire-resistant safe, the maximum credible accident (a furnace explosion) would tear open the firesafe. If I was paranoid and rich, I would use a high bandwidth VPN connection to a big disk in a colo machine in a different city.
The best backup is server-pull, frequent, automated backup onto multiple R/W media in multiple places, and frequent checking of that data. The closer you can approximate this, the more secure your data will be.
Keith
What we do... (Score:2, Informative)
Every now and then these are moved to a RAID5 array.
Any totally critical stuff is also backed up onto DVDR.
Not optimum, but unless you want to spend a ton of money......
r.
Re: Backups (Score:4, Informative)
First of all, disks are *so cheap* these days, hard drives are a more than acceptable backup medium. As disks tend to be identical in size and construction if you buy in batches, disk-to-disk backup is quite the good system, just as long as you don't always keep the disks in the same location (aka, not even on the same controller! *gasp*)
Secondly, you went into a lot of specifics that I didn't care to; a lot of backup systems are custom tailored to the situation.. so while this kind of system might work great for you, I doubt if it would work so well in this case, especially. Digital media tends to be very non-compressible, very volatile media. That being said, operations like MD5 are very crucial to insure the data from one location matches another, which means even more precautious MD5 storing measures. You're also dealing with larger files which means rdiff is almost entirely out of the question.. I could go on and on about different, application specific schemes, but I feel I did good enough with suggesting three different mediums and to have at least two copies of two of them, preferably in 4 different locations.
Re:To a second hard drive? (Score:3, Informative)
The price per gigabyte of tape is much lower than hard disks. The large capacity drives are expensive, but in the long run, the higher media cost for hard drives will be even more expensive.
MM
Re: Backups (Score:2, Informative)
Deletion, however, is not the same thing as corruption.
KFG
Re: Backups (media safe) (Score:2, Informative)
Also, I wouldn't really worry much over fancy locks. I remember going into a safe shop where the owner pointed at a menacing 3 ft tall safe with the whole mechanism ripped apart. He said it was an easy 15 minute job with a crowbar. I hang the keys to my safes in plain sight. I'd rather a thief pop it open and decide it's not worth hauling 100 lbs for simple home videos or pre-school art projects.
My method:
1. Raid 1
2. Nightly auto rdiff-backup
3. Occasional manual rdiff-backup, elsewhere
4. DVD-R DL
Semi-solution... (Score:2, Informative)
You can get a "decent" film recorder for 35 mm film for under $1000, but if you want better results you'll need to spend more...
Here are a few you can check out if you like:
http://www.ctcsouth.com/ [ctcsouth.com]
There are companies that will transfer digital video to analogue film stock for you, but they are REALLY expensive. If the video is really important, you can check them out. Example: http://www.cinebyte.com/ [cinebyte.com]
Re:this is actually a BIG question (Score:1, Informative)
There was a period of time when 19th-century printing took off with periodicals printed on very cheap high-acid paper. The majority of those have self-destructed because they were made to be cheap and most people didn't care enough/known enough about archivability.
There are lots of high-cost highly reliable archival technologies out there; but as their costs are so much more than the cheap-and-temporary media, they fall out of popular use...
Re: Backups (Score:3, Informative)
Re:for video back-ups... (Score:3, Informative)
Re: Backups (Score:2, Informative)
> remaining below 350 F.
Fine for paper (Fahrenheit 451).
Not so fine for magnetic tape (125 F) or CD/DVD (248 F) media, both would be damaged long before 350 F.
There are a number of data media rated fire resistant safes that will keep under 125 F for an hour for a 1800 F fire.
From
http://www.kodak.com/global/en/service/tib/tib430
The glass transition temperature for polycarbonate is approximately 140 degrees Celsius. If the temperature gets within 20 degrees Celsius of the glass transition temperature, there is a likelihood of significant disc deformation.
From
http://vsg.cape.com/~pbaum/magtape.htm [cape.com]
Other than a fire, the real danger of high temperatures (above 80 degrees Fahrenheit) is an increase in tape pack tightness caused by wound in debris, tape distortion caused by this pressure, and possible layer to layer adhesion. Print-through is increased by approximately 1.4 dB for every 10 degrees Celsius (18 degrees Fahrenheit).
Re:permastor (Score:3, Informative)
If you check it out, I offer systems from 80GB to 800GB and this is the effective capacity, not the "marketing" capacity (before RAID utilization). For the 80GB system the $/GB is $5/GB and it scales down to $2/GB for the 800GB system.
The systems provide the same type of reliability technologies that Enterprises use to guard their data, RAID for Hardware Failure, Point in Time Copy to create snapshots for Application Data Corruption or Pilot error and backup to CD/DVD or even a remote Data Mirror capability (standard on the OfficeStor systems and will soon be an option for the HomeStor systems) for Disaster Recovery.
Thanks!
Re: Backups (Score:3, Informative)
Re:Parity Files (Score:3, Informative)
Depends one which parity program you use. Best bet is to put all of your data in the root folder (zip it up if you have directory trees to preserve) and make a set of parity data using QuickPar. I usually fill 5-15% of the disc with parity, netting me about 4Gb of storage per DVD+/-R.
Now as long as the inner ring of the DVD/CD isn't too badly scratched (the ToC area), you can always recover this data. If the ToC area gets trashed, you'll need a special drive to recognize the disc (consumer DVD drives will refuse to load the disc).
If the directory gets hosed, you can rip the disc to an ISO file using ISOBuster, copy the image and rename it as PAR2. Then fire up QuickPar. QuickPar will scan the PAR2 file and the ISO file and find all of its parity blocks and the matching data blocks. It will then reconstruct the original files.
I've pulled more then one disc back from the brink using this method. Takes a while, but worth the peace of mind.
The big advantage of parity files is that it gives you a larger recovery window. Low-level errors are hardly ever noticeable by the end-user until it's too late. The built-in ECC on the disc fixes the issue on the fly, but doesn't alert the user that the media is failing. So when the ECC fails, the user is left with no option to recover. However, if you have parity files, once you notice that the ECC is failing, you still have a shot at using the parity files to recover your data.