Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
Data Storage Media

Best Way to Back Up Photos and Video? 642

Posted by timothy
from the pray-to-cthulhu dept.
jsalbre writes "I do a lot of digital video work, and my wife is a professional photographer. With raw DV from the video camera using up 11GB/hr, and raw images from the digital SLR using 7MB I'm quickly using up a lot of space. I currently back up all my important files each night from one harddrive to another, but I now have over 200GB of irreplaceable data (more than just DV and photos, but those make up the largest chunk) and I'm having to exclude the "less important" irreplaceable files as my backups have started failing. Several people have suggested backing up vital unchanging files to DVD (video, images,) and continue backing up frequently accessed files to harddrive, but with recent studies showing that optical media doesn't last very long I don't want to come back in a few years and find that all my backups are useless. Not to mention that some of my DV files are larger than even a dual-layer DVD, and it would be near impossible to automate backup to DVD. How do other Slashdotters back up their important data? I'd appreciate distinction between methods for frequently accessed files and for infrequently accessed files. Any suggestions will be highly appreciated!"
This discussion has been archived. No new comments can be posted.

Best Way to Back Up Photos and Video?

Comments Filter:
  • Re: Backups (Score:3, Interesting)

    by hendridm (302246) on Saturday June 25, 2005 @09:31PM (#12911785) Homepage
    with recent studies showing that optical media doesn't last very long I don't want to come back in a few years and find that all my backups are useless ... How do other Slashdotters back up their important data?

    Why not make two optical backups. Store at least one in a fireproof safe. For the massive files, you might have to invest in one or two hot swappable drives you can use as 'tapes', storing one in your safe. Mirroring might help.

    • Re: Backups (Score:5, Interesting)

      by ciroknight (601098) on Saturday June 25, 2005 @09:36PM (#12911817)
      I concurr. For digital media, I would definitely do two of at least two different back up strategies.

      First that comes to mind is Tape backup. They store huge about of data, and are very cheap these days, and have been proven to last for a while. Keep a good backup schedule, and keep one copy of the tapes offsite.

      Secondly, I'd do optical. Optical's cheaper, but it's also not as long lasting, and takes longer to make the actual back up.

      Thirdly, I'd do RAID. Mirror all the files onto a second set of hard drives. If you really want to get paranoid, mirror onto two sets of drives, and once a week swap out a copy of mirrored drives from a fireproof location.

      If your data is truely irreplacable, then this is a good regiment. But it's also very expensive.. so you'll have to make up your mind.
      • Re: Backups (Score:4, Insightful)

        by ozmanjusri (601766) <aussie_bob@hotmai[ ]om ['l.c' in gap]> on Saturday June 25, 2005 @10:26PM (#12912051) Journal
        First that comes to mind is Tape backup.

        So what you'd suggest is that he downloads the video from the MiniDV tape to the computer, then archives it onto backup tapes. Why not just keep the original MiniDV?
        • Re: Backups (Score:5, Insightful)

          by RollingThunder (88952) on Saturday June 25, 2005 @10:43PM (#12912115)
          Could be a number of reasons to do the seemingly illogical double-hop method.

          First, the new data may have been processed (edits, color correction, etc).

          Second, the backup media may be better rated for long term storage. I'm not familiar with MiniDV, the stuff I work with is all DLT and HCART2 under Veritas Netbackup, at 200GB raw/400GB compressed per tape.

          Third, it may be helpful to have the indexing done for him by a good backup program.

          However, as I say, I work with Netbackup. To say it's pricey is an understatement... but it's changed my views on what's a "workable" backup system to only liking enterprise grade stuff.
        • Re: Backups (Score:3, Interesting)

          by magarity (164372)
          Why not just keep the original MiniDV?

          Probably because mini-DV holds about 13GB and an LTO has a capacity of 400GB. Get a 4 tape autochanger and you've got 1.2TB, or about 92 Mini-DV tapes.
        • Re: Backups (Score:3, Insightful)

          by ciroknight (601098)
          If he recorded on MiniDV, then GREAT, there's never a reason to tape over the master copy of anything!!

          But most of the time, it's a digital camera, where it's flash ram it's recording to. For digital video, it's most likely in need of some editing which is the whole reason to bring it into the computer anyways, which is when you need to start the backup proceedure.
      • Re: Backups (Score:5, Informative)

        by arete (170676) <areteslashdot2@x i g . net> on Saturday June 25, 2005 @10:43PM (#12912114) Homepage
        While I love raid, RAID is not a backup - raid is about availability and consistency. So if you delete one item in a RAID it is SUPPOSED to be lost to the entire array.

        In everything I've read, the moral definitely seems to be harddrives, lots of harddrives, for price performance. I'm assuming you have a reasonable LAN or can set one up.

        Here's the setup I haven't finished implementing yet: PLEASE give me any comments about it to help me improve my setup.

        1. Setup a file server using at least one big, inexpensive disk. (This can also be a desktop as long as it can reasonably serve files.) This is your "USE" server.

        2. Separate you files (on a per-directory basis) into categories based on how frequently they are changed. The important consideration is: 'If a file is changed/deleted from USE how long should I wait delete a file in the backup' Personally, I only need two categories. "current" = a month or so depending on disk space and "archive" = never (family pics, videos, etc.)

        That means that if I delete something in my "current" tree _AND_ I don't notice for a month, my backups will delete it and it's gone forever.

        3. Setup a 'backup server' using at least one inexpensive hard disk. Set your backup server to login to your USE server and sync your files.

        It should be able to do both "full" (copy everything) and "incremental versioning" = "IV" (if something is changed, keep BOTH copies, marking them appropriately) backups. Neither of these kinds of backups should ever eliminate any information automatically - they should just add information.

        4) For me, I'd run:
        1) An IV backup of "archive" every night.
        2) A full backup of "current" every week.
        3) An IV backup of "current" every night.
        4) A job that deleted the oldest backups of current every week.

        Notice that I'm _never_ running a full backup of "archive" but I'm also _never_ deleting the backup.

        Notes:
        rsync or rsync over ssh is my preference for doing this kind of backup. It works very nicely, but I'm too tired to get it right just this minute so I'm leaving IV/full backup commands as exercises for other /. readers, but it's two 1-line scripts and I've seen them on here before :)

        cron is fine for setting it up automatically.

        wget has similar functionality to rsync for a website and you don't need any privileges.

        I think most of /etc belongs in "current"

        Do make sure you log the output of your syncing software. Also make sure you monitor disk usage. If you want to be fancy, it could keep all of the full-backups of "current" until space is short (with a reasonable margin) and then always delete as many of the oldest ones as it needs to to make enough room. This means your number of snapshots will vary with disk space - some people think that's evil.

        This system scales reasonably well - for more size add more harddrives per server and/or more servers. For redundancy add more backups per live copy. As long as you can keep it organized and your network handles it, there's also no reason a USE server can't be served by two backup servers or a backup server can't also serve several smaller workstations - or any combination thereof.

        Do not add multiple harddrives to a backup server for redundancy. These servers are essentially free and you get much more redundancy (and some scalability) if you use two backup servers. With a setup like this, any server should only have one copy (excepting multiple versions of the same tree)

        You could just do a full backup of current every night or whatever, and you could have many possibly more complicated "current" backup schemes. But for me the total size of "current" is massively smaller than "archive" so it's really not important. Remember, having more of these isn't more redundant - they're all on the same drive.

        This backup server should generally run no services except possibly ssh and certainly shouldn'
        • Re: Backups (Score:4, Informative)

          by ciroknight (601098) on Saturday June 25, 2005 @11:47PM (#12912346)
          While RAID was originally meant for data security, availability, and consistancy, it has a lot of other applications that weren't in it's original design.

          First of all, disks are *so cheap* these days, hard drives are a more than acceptable backup medium. As disks tend to be identical in size and construction if you buy in batches, disk-to-disk backup is quite the good system, just as long as you don't always keep the disks in the same location (aka, not even on the same controller! *gasp*)

          Secondly, you went into a lot of specifics that I didn't care to; a lot of backup systems are custom tailored to the situation.. so while this kind of system might work great for you, I doubt if it would work so well in this case, especially. Digital media tends to be very non-compressible, very volatile media. That being said, operations like MD5 are very crucial to insure the data from one location matches another, which means even more precautious MD5 storing measures. You're also dealing with larger files which means rdiff is almost entirely out of the question.. I could go on and on about different, application specific schemes, but I feel I did good enough with suggesting three different mediums and to have at least two copies of two of them, preferably in 4 different locations.
          • Re: Backups (Score:3, Insightful)

            hard drives are a more than acceptable backup medium

            No.

            They are a minimally acceptable backup media for short-term storage.

            Consider the fact that with tapes, you really just have to worry about tape errors. If the tape drive fails, you can use another.

            With hard drives, you have to worry not only about errors on the drive, but about hardware failures in the electronics as well.

            In 10 years, that hard drive will probably be dead no matter what you do. But a properly stored tape backup would still

        • Re: Backups (Score:3, Informative)

          by OverlordQ (264228)
          sounds like you need BackupPC [sf.net]
        • Re: Backups (Score:3, Interesting)

          by Mr2cents (323101)
          That's more or less the setup I'm using too. I have the big router/home server with a shared 120GB harddisk, and a backup server controlled by a timer. At 2AM the timer turns on the computer and it makes it's backup using rdiff-backup. As it only copies differences in changed files, I don't really worry about the backup size, at the moment it's merely 1.6GB. After the backup, it generates a log in html and copies it back to the main server.

          All this I wrote in bash in one afternoon.
    • Re: Backups (Score:5, Informative)

      by gui_tarzan2000 (625775) on Saturday June 25, 2005 @09:47PM (#12911877)
      Fire proof safes aren't all heat proof. I was told by our insurance company recently that the FP safes and cabinets are only rated that for durable goods such as paper (which can withstand a lot of heat in some situations), metal, etc. but CD/DVD substrates will melt or distort rather quickly because they're so thin. I'd guess the expensive ones would be ok but I wouldn't count on that for the cheaper ones. I'd sooner put them in a safe deposit box at a bank where the vaults are much safer in most scenarios.

      One thing good about paper & film is they withstand decades of storage vs. years of normal magnetic storage. Photos and films from the late 1800's/early 1900's are still around whereas you're really gambling with current storage media.

  • by MSDos-486 (779223) on Saturday June 25, 2005 @09:31PM (#12911789)
    File -> Print
    • Longevity measured in millenium:
      Number one choice: vellum
      Number two choice: papyrus

      OR
      Longevity measured in decades:
      Number one choice: DVD
      Number two choice: EIT-3 (tape)
      Number three choice: RAID-5 (hard disk)
      Number four choice: RAID-5 NAS (disk)
      Number five choice: RAIT-5 (EIT-3 tape)

      Since you indicated a need for a long term solution, but didn't mention price range, why not consider redundent RAID-5 NAS, which could be platform agnostic?
      • Recent studies have shown that DVD that's not pressed, but written with a consumer-grade DVD recorder does not have especially good longevity, just as a warning. Commercially pressed DVDs do have excellent life, but same as home-written CDs, it's not that great otherwise.

        The specific media makes a big difference as well. Look up the NIST study about it. Interesting reading.
  • by Oceanplexian (807998) on Saturday June 25, 2005 @09:31PM (#12911790) Homepage
    i think you might want to take a look at tape drives
    • by nametaken (610866) on Saturday June 25, 2005 @09:58PM (#12911924)

      Agreed. I realise tapes aren't the new hotness, but they're the most reliable, and they have good storage capacity. In addition, I'd consider a larger capacity storage server. Together this stuff may not be as cheap as tossing everything on DVDs, but apparently this is for people who work in digital media for a living. From that perspective, its worth investing in your profession.

      Perhaps better than slashdot, they're bound to have a huge network of friends in the profession who have already crossed this bridge. It couldn't hurt to ask how people specificaly in these professions manage their media storage.
    • To expand (Score:5, Informative)

      by Ironsides (739422) on Saturday June 25, 2005 @10:25PM (#12912047) Homepage Journal
      At work we have to archive Broadcast Quality TV Shows (Yes, I work in a TV station). These are 50 Mbit/s for the video plus at least another 4 Mbit/s for the audio. Needless to say this takes up a lot of space. For this, we use LTO-1 tapes that store 100GByte per tage uncompressed (compression gets us zilch with the video and audio). The tapes have error correction that we pay attention to. If there are getting to be too many errors we replace the tape and have the info copied to the new tape. Since we have so many shows, we are moving to LTO-3 tapes that store 400GB per tape. The LTO tapes are expensive. However, as long as you do not do constant reading from them and use them as a true archinve they should be fine. For massive redundancy, put the same files on two different tapes. Also, the reader/writer is a little expensive, but you only need one. Also, LOT-3 drives can read/write to LTO-1 tapes (only as 100GB, not 400GB). Write speed is pretty good to, being above 14Mbyte/s. Shelf life in a temperature/humidity controlled environment is pretty long. A bank vault should be pretty good as well.
  • Tape Backup? (Score:2, Informative)

    by Tibor the Hun (143056)
    Would tape backup work for you? It's archival quality, but you get what you pay for...

    And supplement that with LaCie external firewire drives.

    • Re:Tape Backup? (Score:5, Informative)

      by Kevinv (21462) <.ten.neraahnav. .ta. .nivek.> on Saturday June 25, 2005 @09:43PM (#12911856) Homepage
      since when is tape archival quality? It's barely backup quality. I've had way more properly stored tapes fail than I have properly stored optical media.

      Treat optical media like magnetic media (store in cool dry place) and use high-quality media and you'll get far better results than tape.

      Add in the speed at which tape drives become obsolete and tapes hard to obtain, while CD's are still readable. And I've found optical to be a superior archive medium.

      If you examine the study cited you'll notice that the study is for optical media in harsh conditions. Additionally they specifically state "It is demonstrated here that CD-R and DVD-R media
      can be very stable (sample S4 for CD-R and sample D2 for DVD-R). Results suggest that these media types will ensure data is available for several tens of years and therefore may be suitable for archival uses."
      • Re:Tape Backup? (Score:3, Informative)

        by Detritus (11846)
        Tape is reliable, if you spend enough money on the hardware. You just have to decide, do you want a reliable tape drive or a shiny new car?
  • Tape... (Score:4, Informative)

    by StillAnonymous (595680) on Saturday June 25, 2005 @09:32PM (#12911793)
    Well, it's expensive, but maybe you can find a deal on an LTO2 or SAIT tape drive on Ebay. These babies boast 200GB and 500GB of native storage respectively . The transfer rates are nothing to sneeze at either.

    And as long as you store the tape properly, it should last a long time.
  • Compression (Score:2, Insightful)

    Have you considered compressing your video using one of the many codecs available? DivX is quite popular, and RMVB offers some of the best quality:size ratio I've seen. I understand how nice it is to be able to store raw mpeg for later use but is it really necessary for your purposes?
    • Re:Compression (Score:3, Informative)

      by PsiPsiStar (95676)
      But what if he wants to reuse the clips. I think he needs a lossless format like HUFFYUV, and that's not going to compress much.
  • I keep my originals (video, images) somewhere on DVD and use a smaller, screen optimized version to show to family and friends.
  • by BobWeiner (83404) on Saturday June 25, 2005 @09:34PM (#12911800) Homepage Journal
    I typically take the edited footage and back it up to another miniDV tape from the computer (using my miniDV camcorder). I then lock the tape to prevent accidental erasure and store the tapes offsite. For photos, I'm taking my chances and burning them off to DVD. I also periodically make digital prints and send them to my parents and sister, who live in two separate locations. Worst comes to worst, at least they have a hard copy available should I lose the original digital version that I have on my computer.

    • by Anonymous Coward on Sunday June 26, 2005 @01:25AM (#12912701)
      Someone needs to mod this one up to 5 just from the video angle alone. I know a ton about video production, and I'm constinently amazed about how many people don't know they can back up their full res video files back to MiniDV tape via something typically referred to as "Print to Tape." You can print your edited footage back to MiniDV via Firewire just like you downloaded them off of the camera to begin with. This saves the FULL RES video files, not some crappy MPEG2 DVD dumbed down low bitrate crap -- the actual NTSC video. Not a complete backup solution, but when it comes to video, nothing beats this because you are getting 30-40 gigs worth of video backup for the price of a single MiniDV tape.
    • Be careful how you do this. I was doing this under iMovie a while back - I condensed several clips from various tapes to one DV. I noticed that if you suddenly suck up a bunch of CPU (Like say, trying to launch an app so you can do something else while you are waiting) iMovie will drop frames going out to the camcorder. (Ouch, those frames would have been lost forever if I hadn't noticed!)
  • You don't really say just how much those irreplaceable files are worth to you. A lot of people have things are irreplaceable until they find out how much it will actually cost them to back it up properly. Then suddenly little Timmy's first steps don't look so hot.

    Go pick yourself up a xRAID or the like and back all of your files up to a nice RAID 5 system. Once a year or so do a dump to optical media but just add additional space as necessary.
  • by PrivateDonut (802017) <chris5377@mailcan . c om> on Saturday June 25, 2005 @09:35PM (#12911806)
    (i have no experience in the matter) you should place all the un-changing files on a hard-drive which will then sit in a draw and is only plugged in when required. I have been lead to believe that this will reduce the likelyhood of harddrive failure to close to 0. Then you can setup a RAID type setup for you changing files.
    • I don't know if it would be a good idea to do that. There is the possibility of the harddrive seizing up or something along those lines if it just sits there for years without being powered up. I would turn on the harddrive every once and a while just to be sure. With that said though, I have quite a few old 1GB and under drives that have been sitting around, some of them for years. So far, whenever I grabbed one for some project, everyone of them has still worked just fine.

      Another thought I have is th
  • My methods. (Score:5, Interesting)

    by sglider (648795) on Saturday June 25, 2005 @09:35PM (#12911807) Homepage Journal
    Let's face it, one method won't fit all, so I hope your search proves fruitful. That said, here's what I do.

    I have a 'cheap' system (sub 500) that acts as my data server. It houses 3 DVDrom drives, and a DVDRW drive, as well 1 200 GB drive. (the processor speed and ram really aren't too important, but for curiousity, it's an athlon 2000+ with 512 meg of ram). It runs gentoo, and I essentially pull the files to burn to DVD over the network weekly, and I keep the stuff I don't access alot on DVD, and the stuff I do access alot on HD -- but I primarily use the HD for holding images waiting to be burned.
  • USB HD (Score:5, Interesting)

    by lseltzer (311306) on Saturday June 25, 2005 @09:35PM (#12911811)
    To a hard disk in a USB enclosure. Better yet, but more expensive, to a NAS box.
    • Anyone have recommendations for a cheap NAS? I would think that they should eventually be down as cheap as $20 (i.e. as cheap as a USB enclosure), but they seem to be much, much higher.
    • by jerryasher (151512) on Sunday June 26, 2005 @01:11AM (#12912655)
      For the past five or six years, I've been taking my data, applying steganography techniques to encrypt it into the background of porn images, and then distributing those images via usenet and a few porn sites I've whipped together (ok, ok, the bangbus videos.)

      At any time when I need to recover the data, I just use google to find someone with a copy of my data, download, decrypt, and voila!

      This is my cheapskate's Network Storage Device!
  • I have old full height 5.25 hard drives from the IBM XT days that STILL BOOT and still have viable data on them over 20 years later..

    200gb drives can be had for under $100 on ebay.
    Load them up, remove them and store them in a fire proof safe..

    Problem solved..

  • Memory (Score:5, Funny)

    by vjmurphy (190266) on Saturday June 25, 2005 @09:37PM (#12911826) Homepage
    "How do other Slashdotters back up their important data?"

    I memorize it.
    • Re:Memory (Score:3, Funny)

      by antifoidulus (807088)
      11101010111010101011101010101010101001010100011111 00101010101010101010101010010101011111100100110111 00001110000010000100001010101001010101010101010100 10111000010001000001011110001000100010000101000010 000010101000101....uh holdon..don't tell me...jeez I know it's on the tip of my tongue.....god what was that...ugh I always forget these little OH 1...it was 1...001012000100010000011001010100101010101000
    • "How do other Slashdotters back up their important data?"

      I encode it into articles that I place on usenet, and then let google act as my archiver.

  • by Shag (3737) on Saturday June 25, 2005 @09:37PM (#12911830) Homepage
    There are a couple different factors in play here.

    First, there's the need to keep things around long-term. Second, there's the need to have things protected from disaster in the short term.

    I once used an external firewire HD for backup, and was reminded of the importance of burning things as well when that HD went tango-uniform on me, destroying months of work.

    I'd suggest looking into some sort of RAID - even just a simple mirror - for the short-term protection. That way you don't have quite as much a single point of failure that can wipe out your data, so you can do backups more because you need the space than because you need to sleep well at night.

    As for the backups, optical discs are very convenient, but magnetic tape might have a longer lifetime depending on environmental conditions, and although I've seen CD-R comparisons [pcbuyersguide.com], I've yet to see something similar for DVDs.

    There are times where a high-capacity removable hard disk looks very attractive. Shades of the old Bernoulli's or whatever.

    (This may not be first post, though there were none when I started. Maybe I'll have to settle for first useful post.)

  • I was just looking into this, and I decided that the best idea was probably Streamload [streamload.com].

    They offer unlimited storage but you pay to download more than a certain amount a month--but if you have hardware failure that leads you to really need it, you probably won't mind paying, or spacing out your downloads.

  • Why not get a multi-terrabyte raid array? Pricey to get started, but it'll keep your data reasonably safe.

    You also could use a tape backup. Any of the results from here [pcmall.com] could do the trick. At work, we use one of the 200/400GB tape drives for backup and are only using about 10-15% of the space (and that's for a dozen servers). We haven't had to test the lifespan of one of these, but tapes typically have an excellent lifespan compared to hard drives or optical media.
    • There are a few good reasons to skip raid as a sole backup.

      1. Human Stupidity, one mistaken format of the raid instead of that USB drive and poof.

      2. Localized disasters, Flood, Lightning, Tornadoes, Blizzards, and Fire are all things that will can trash a raid.

      3. Human malice, theft, vandalizm, hackers, viruses, worms and the like. Offline storage is less suceptable to these issues.

      Storm

  • by kf6auf (719514)
    My recommendation is to put a nice RAID array in a different computer. I suggest RAID 5 because it is more efficient ("wastes" less space on redundancy and still gets the job done) than RAID 1 but they have roughly equal reliability. The advantage is that it's an entirely separate computer and you even have redundant backups. Hell, you could even put it off-site, but then the connection speeds drop a bit. If you use a RAID array on your local machine you may be tempted to treat the RAID like it is itsel
  • a lot of people have been recommending hard drives, which i think are great for frequently used files and frequent backups. the problem is that they fail to get your data off site.

    you probably want to occasionally back up to something that you can store in a lock box away from your house. i guess you could do this with disk drives, but i'd rather use tape. more reliable than optical media and plenty of capacity.
  • by tenzig_112 (213387) on Saturday June 25, 2005 @09:49PM (#12911883) Homepage
    (other than the continual confusion of "backup" and "archive") is that the same people who talk about how unreliable CDR/DVDR discs are for longterm archival purposes seem to be the same ones who advocate buying a portable firewire drive for every project and putting it on a shelf until the client calls with changes.

    Something about that seems horribly backward.

    That said, Exabyte still rocks my socks
  • For the MiniDV video, just save the MiniDV tapes.. don't re-use them. What you don't need online access to, you've got on tape.

    The digital images should be more manageable on their own. Buy a couple redundant backup hard drives. or, save them to DVDs, etc.
  • by exp(pi*sqrt(163)) (613870) on Saturday June 25, 2005 @09:55PM (#12911913) Journal
    Rename all of the files so they have filenames like "Teen_Lesbian_fff_Hot!Hot!Hot!.avi". Now make them available through your favorite p2p service. Even better, prepend these files with short snippets of pr0n. You'll find that years later you can kick up just about any p2p client and you'll find your files are still available.
  • permastor (Score:3, Interesting)

    by joequser (215949) on Saturday June 25, 2005 @09:56PM (#12911918)
    A friend of mine recently started a small business to address exactly this need. His product is a Linux based RAID box that plugs in to a home network, and supplies reliable storage via samba.

    http://www.permastor-us.com/ [permastor-us.com]
  • I suggest cardboard punchcards. With the right care, those will be in your archive for years to come and have virtual no chance of failure.
  • RAID+LVM snapshots (Score:3, Interesting)

    by swillden (191260) * <shawn-ds@willden.org> on Saturday June 25, 2005 @09:57PM (#12911922) Homepage Journal

    I use RAID to defend against hardware failures trashing my data, and I use logical volume management snapshots to protect against most user errors.

    Neither is perfect. Some hardware failure modes could theoretically kill two or more of my four hard drives at once, which would destroy my data. Large power surges are the most likely danger, so I use a high-quality surge protector. I consider the remaining dangers unlikely enough to accept the risk.

    Snapshots are also imperfect. When you create a Linux LVM snapshot volume, you have to specify how much storage is allocated to it. If changes on the source volume exceed that snapshot capacity, the snapshot stops storing the deltas and the snapshot becomes effectively useless. However, the most likely way that I might screw up and trash my data is by deleting large numbers of files. Since deleting files only updates the blocks that store the directory and inode data, not the contents of the files, a relatively small snapshot partition would hold the changes from deletion of all the files on the source. Now, if I were to accidentally run "shred" on bunches of files... I'd be screwed. I choose to accept that risk, too.

    Although the RAID+LVM combo doesn't do quite as good a job as "real" backups, its failings are pretty minor, and unlikely, and it's advantage is huge: I don't have to think about it. I don't have to mess with lots of removable media and I don't have to remember to do backups.

    The one thing I still worry about is some sort of catastrophe that destroys my whole system. Suppose my house burns down, for example. I'd lose it all. So I still need to find some way to get offsite copies of the most important stuff.

  • We were using 200gig external USB drives for backup and were wondering why they would all eventually start failing. It turns out the manufacturer (Maxtor) recommended against using these drive for backup??? Sounds like they're saying that the drives aren't reliable so what good are they? We switched to drives that were "certified" for backup/archival purposes.
  • I have a mac mini which also deals with backup. Even when hooking up
    a few fire wire drives, the thing is still small and quiet. Then rsync
    the data over regularly with a script like

    #!/bin/sh
    rsync -avzuP -e ssh --delete /home/user/dvds macmini:/Volumes/external/video

    and call this by cron. This works reliably also with large files like
    vmware workstations or dvd backups with several gig file size. Having the backup
    over the network allows having the backup machine in a separate place
    which limits the risk (for
  • by Rocky Mudbutt (22622) on Saturday June 25, 2005 @10:02PM (#12911952) Homepage
    "Real men don't use backups, they post their stuff on a public ftp server and let the rest of the world make copies." - Linus Torvalds
  • Here's what I do, YMMV. I have a RAID-1 array that I back up my 'important' stuff to (I'm switching to a 4x300GB RAID-5 soon). Every 3 - 4 years I replace the drives. With the natural increase in storage space per dollar, I am able to fit everything from my old array with tons of space left over for the new stuff.

    Of course, like everyone else you'll have to look at the content you create and decide if it's worth the money required to back it up. Establish a cost per gigabyte for each solution, and decide
  • Disasters (Score:5, Interesting)

    by Detritus (11846) on Saturday June 25, 2005 @10:11PM (#12911989) Homepage
    I see a lot of suggestions. What happens when:
    • Fire and attendant smoke/water damage.
    • Flooding.
    • Lightning strike.
    • Severe electrical fault.
    • Burglary.
    • Catastrophic power supply failure.
    • Disk controller fails in interesting way.
  • Just get hundreds of gigs of disk (probably you'll need at least 1,000), then use rsync to mirror them. I've used this system for some years, with an amount of data on the order of what you're talking about.

    I have multiple mirrors, and I rotate between the mirrors, so that on a given day, I have a backup which is 1 day old, 2-4 days old, and a week or so old. On top of that, I periodically take mirrors out of circulation, say, every month or so.

    Yes, I have alot of hard drives. However, drives are cheap
    • by klic (739114) on Saturday June 25, 2005 @11:19PM (#12912226) Homepage
      I host dirvish ( http://www.dirvish.org/ [dirvish.org] ), a backup application for Linux/Unix, using Rsync and Perl. Like Chuck Messenger, I rotate the target drives. You can only trust an air-gap between your backed-up data and a hostile world.

      Rsync ( http://rsync.samba.org/ [samba.org] is really great for backup of Unix-like systems. The ability to hardlink identical files allows me to store hundreds of daily full images of 100GB of sources to a single target 250GB hard disk. Rsync is very smart about moving only changed data over the network, resulting in speedups of 10x to 100x. This allows me to do full backup on my offsite colo without using a lot of bandwidth. Note that Rsync is great for Mac/Unix/Linux, but it does sometimes have problems with windoze clients. But then, so do I ...

      Dirvish (originally written by jw schultz) is a Perl wrapper around Rsync. It facilitates the scheduling and management of Rsync based backups. We have a fairly active mailing list and contributions from around the world (open source is so cool!).

      Backups should be safe against:

      • Failed hard drives
      • Stupid mistakes
      • Enemy action
      • Fire, flood, and theft
      • Host and power supply failure
      • Unauthorized access

      Backups should be automatic (or they will not get done) and cheap (hard disks are cheaper than tape, and much cheaper when you use hard linking). Rsync stores the data in a file system closely approximating the original, which facilitates restores.

      If a cheap electrolytic filter capacitor dries out in your power supply, and the 5V output decides to start making a 15V squarewave instead, everything in your computer case will get fried. Including every one of the RAID disks. External USB enclosures (or airgaps!) protect against host and power supply failure.

      If I was really paranoid about protecting my data, I would run a long ethernet cable to a nerdly neighbor a few houses away, and put a second dirvish server there. While I do rotate my drives into ziplok bags in a fire-resistant safe, the maximum credible accident (a furnace explosion) would tear open the firesafe. If I was paranoid and rich, I would use a high bandwidth VPN connection to a big disk in a colo machine in a different city.

      The best backup is server-pull, frequent, automated backup onto multiple R/W media in multiple places, and frequent checking of that data. The closer you can approximate this, the more secure your data will be.

      Keith

  • by the eric conspiracy (20178) on Saturday June 25, 2005 @10:12PM (#12911996)

    The NIST report didn't say optical media were inheritantly unreliable at all; it said that there were big differences in media quality, and that storage conditions were important.

    Personally I think hard drives are the pits for data reliability. The drives are good for MAYBE 3 years, subject to all sorts of electrical failures, and even if you have a RAID you still can lose the whole thing due to a {virus,controller,power supply,filesystem,usererror}.

    I use redundant MAM-A gold stabilized CD-Rs for my data which were the most stable option in the NIST report. That works great for everything I have including digital photos.

    DV might be a pain with CD-R so I would probably start with staggered redundant sliver DVD-Rs until I saw some more data on the lifetime of this media.

    No way would I consider hard drives an acceptable archival solution.

  • Needs (Score:5, Insightful)

    by drooling-dog (189103) on Saturday June 25, 2005 @10:15PM (#12912002)
    You're storing raw DV and 7MB imagery? Face it: You're probably a data hoarder. You'll never look at this stuff again, because you'll always be too busy collecting & storing the new stuff. Maybe you need to step back and re-evaluate your real needs.

    This is the pot calling the kettle black, though. Is there a support group out there?

  • Whatever medium you choose, you could combine with a parity archive tool [sourceforge.net] to recover from minor media errors.
  • by Tsu Dho Nimh (663417) <abacaxi@NoSpaM.hotmail.com> on Saturday June 25, 2005 @10:19PM (#12912025)
    Archive (make a copy and delete from the work area) is for files you might want someday. Backup is for files you KNOW you will need tomorrow or next week.

    Your computer's own hard drives should keep only what you are actively working on. Get the rest of the stuff out of your way.

    Buy GOOD DVDs ... burn all the files you are not actively working with to these - two separate DVDs for each archive, of two different brands. Check for file integrity, label them well and store them in a convenient, off-site location, cool and dark. Delete the originals from the working drive. Check the archive disks fairly often for degradation and re-burn as needed. They are no more labile than negatives and videotape.

    For the large files, buy removable drive bays and holders, and copy them onto large hard drives. REMOVE the drives and store them with the DVDs.

    On your working system, continue to back up the data for the active projects. Consider getting a RAID 5 system for data integrity, because if you back up data from one drive to another you risk overwriting a good copy with a bad copy.

  • Parity Files (Score:3, Insightful)

    by RedXIII (81965) on Saturday June 25, 2005 @10:19PM (#12912026)
    I keep reading comments about how CDs/DVDs are unreliable. Here's a great trick i use to make sure my data is safe: i always include 50-100mb of parity files on each DVD. The disc would need to be REALLY messed up to be unrecoverable.
    • Re:Parity Files (Score:3, Interesting)

      by m50d (797211)
      Not really, because you only need for the FAT to be unreadable (CDs/DVDs don't normally have multiple copies like hard disks often do) and then the data will still be there but you'll have a helluva time trying to access it.
      • Re:Parity Files (Score:3, Informative)

        by WuphonsReach (684551)
        Not really, because you only need for the FAT to be unreadable (CDs/DVDs don't normally have multiple copies like hard disks often do) and then the data will still be there but you'll have a helluva time trying to access it.

        Depends one which parity program you use. Best bet is to put all of your data in the root folder (zip it up if you have directory trees to preserve) and make a set of parity data using QuickPar. I usually fill 5-15% of the disc with parity, netting me about 4Gb of storage per DVD+/
  • by acomj (20611) on Saturday June 25, 2005 @10:44PM (#12912118) Homepage
    This might sound obvious.. I'm a photographer.. I have a 2 bay firewire drive set I use for "HD" backup of my photos and video. Its 700 gig. I also burn DVDs.

    When I backup my stills onto dvd I use jpeg 2000, its lossy but really not that bad once the image is in a good state.. I did some tests in college on jpeg/jpeg2000 vs tiff (uncompressed) of the smae image to see how much is lost. Not a lot it turns out. I love uncompressed images, but the loss when storing as jpeg isn't so great to matter unless you do a lot more manipulation. I'm also still shooting film which can always be rescanned at a later date.

    However, you shouldn't backup all the DV (raw video) you dump on the computer. The original tap e can act as the backup. its still on the tape even after you dump it into the computer. Label it and set the right protect notch. Voili, instant backup footage.

    I'm assuming you edit this down and give the client a dvd/video. Just keep a DVD copy for yourself. Thats all they can really ask you for. If they come back at a later date, because the dvd is bad try yours. If that doesn't work you have to go back to the tape and redit and recharge.

  • by Ralph Spoilsport (673134) on Saturday June 25, 2005 @11:13PM (#12912210) Journal
    And one that I have railed about for many years.

    I have been in the same position the Author discussed, and I have come to ONLY negative conclusions. In a few words, and I hate to say this, but buddy:

    WE'RE FUCKED.

    Digital is a loser's proposition. backing up to analogue or even digital data on analogic substrates (such as DV tape) fail. Simply nad purely.

    The *only* thing that comes close is some kind of RAID, and those, even with the plummeting price of storage, are still too expensive given the needs.

    Also, a RAID assumes a continuity of several things that are not likely to be continuous:

    With Video:
    Framerate, number of lines, colour depth, aspect ratio, file format, compression format, Operating system compatibility, etc etc etc. All of these things are variables.

    With Audio:
    sample rate, compression format, bit depth, file format, etc.

    Basically all of it points to very bad places.

    I am fairly well convinced that our age will simply disappear. They will find our garbage, the few books not pressed on acidic paper, our paintings (fat lot of good the abstract stuff will mean to them) and drawings, that's about it. the rest will just be shiny little bits of crap in the landfill.

    Since we will have used up all the dense energy forms, they will be appalled at the energy requirements just to get the few remaining museum piece devices to work. Archiving the 21st century will be impossible. To the 25th century, the 21st century will be seen as a dark age - not only for the holocaust of the die caused by the failure of the petroleum based economy, but from the simple fact that very little of the information formats we are totally geared into will survive, including this note on /.

    His problem of saving personal video is just the tip ofthe iceberg. His problem is the problem of our very civilisation, writ small.

    That's why I am abandoning video, and going back to painting. In 500 years, my painting CAN survive. the video simply won't.

    RS

    • by agent oranje (169160) on Sunday June 26, 2005 @12:47AM (#12912580) Journal
      Mod parent up "brilliant."

      In the digital world, there's currently no such thing as an archive. There are backups that last for quite some time, but I seriously doubt any of them will last forever. The only reason any of these backups last so long is because the people creating them put some serious effort into keeping the data safe - and even then, what's to say it's not going to fail tomorrow?

      You're right about the 21st century becoming a second dark age. Half the time, it proves extremely difficult to find web-published articles from two years ago, never mind what someone was putting on the web 15 years ago. Servers come and go as those involved become disinterested with the media they created. But, the difference between a print magazine going belly up and a dotcom media source going belly up is that the printed magazine will still exist while the data from the dotcom will likely never be accessible to the public again.

      In the case of personal media, digital is a disaster. My grandparents still have stacks of photos documenting their entire lives, as do my parents, as do my parents for me. However, my photo collection currently suffers a gap which will never be recovered, specifically 1997-2000. During those years, I used a digital camera, and I left the photos on a working hard drive for safe keeping - alas, when I went to retrieve some files off of the drive when I wanted to go back and read a paper, I discovered the drive had committed suicide in a year without use. Yeah, that sucks.

      Currently, the best way to back up data is RAID - and that's not even backing the data up, it's just making it more persistent. When you move to another machine, move all of the data to the new RAID. Repeat forever. To be extra safe, have a backup RAID just in case the first one suffers from a catastrophe.

      Why is digital media troublesome? Books rarely render themselves unreadable while sitting on shelves, and are likewise rarely destroyed when dropped. Carving something into rock requires a bitchin' act of god to get rid of. But the deleting of a file, or the death of a hard drive, can wipe vast amounts of history out of existence, both in a personal and societal sense. Without an ability to permanently archive digital data, none of the data from the digital age will exist in the future.
      • by Tim Browse (9263) on Sunday June 26, 2005 @02:40AM (#12912870)
        In the case of personal media, digital is a disaster. My grandparents still have stacks of photos documenting their entire lives, as do my parents, as do my parents for me. However, my photo collection currently suffers a gap which will never be recovered, specifically 1997-2000. During those years, I used a digital camera, and I left the photos on a working hard drive for safe keeping - alas, when I went to retrieve some files off of the drive when I wanted to go back and read a paper, I discovered the drive had committed suicide in a year without use. Yeah, that sucks.

        This may be hard to believe (and I probably sound smug) but there was a time a while ago when the camera hadn't been invented yet, and nobody had any photos at all. We still seemed to survive as a race/civilisation though.

      • by Sycraft-fu (314770) on Sunday June 26, 2005 @02:48AM (#12912894)
        Infomation now is much more perminant than it was in the past, and digital has improved this a great deal. The amount of information we generate these days is enormous, far more than ever before the digital age. Thus it's not supprising much of it gets destroyed. For that matter, most of it isn't worth saving anyhow.

        Books are not such a perminant media as you might think. They wear out, and can be destoryed. A good example is the Mayan Codices. Records seem to indicate there were thousands, however Spanish priests burned them as "works of the devil" during the European conquest of the Americas. Today only 4 remain.

        Digital data can be so perminant because it is so easily copied. Perminance of data does not come form trying to make a single, eternal copy, but from having many copies all over the world. Digital data can be copied for essentially zero cost very easily. Thus it's easy to give it a great deal of robustness. Also, as new formats come out, you simply copy and convert the data. I have data on my harddrive today that orignally existed on 5.25" floppy for the Apple II. It has simply been copied and converted a number of times.

        Finally, it's not like book are going away. On the contrary we publish millions of works a year amounting to billions of books.

        You seem to have a false sense of perminance, as though in the past things were archived forever. That's not the case, actually, most data was lost, that's one of teh reasons we have such an incomplete picutre of history. You don't even know all that was lost, because the record of it even existing, if there was one, is also lost. What has survived is by chance, or by effort, not because we had some wonderful archival system.

        You don't have to have something on an immutable, indestructable medium for it to survive. The Nordic Legends weren't written down for centuries, yet today we still have them. They were passed down, as an oral traditon for generations. There was no perminance to them other than stories in people's minds, yet they've durvived thousands of years.
  • by callipygian-showsyst (631222) on Saturday June 25, 2005 @11:53PM (#12912372) Homepage
    I'm a professional photographer. Every year, I send my best/most valuable photos away to a lab that does color separations (C, M, Y, K) and saves them on black and white film, as well as making a Kodachrome 64 slide.

    These are the only ones I can trust to be around in 100 years or more.

    *All* digital images get written to CD-Rs are are stored in a commercial document-control facility. But the ones I really want to keep get written to film.

  • DV tape is cheap (Score:4, Insightful)

    by ickypoo (568859) on Sunday June 26, 2005 @03:57AM (#12913098)
    The video files that your NLE uses are exact duplicates of the data your camera writes to DV tape. Take a hint from that and just save your DV tapes. All modern NLEs work with EDLs (edit decision lists), so save your session files, overlays, transition parameters, etc to a CDR and push the lock-tab on your master tapes. Keep your tapes labelled and organized so you wont have a problem finding them again. It's trivial to recreate your project at that point, and it thankfully isn't MPEG-compressed on a video DVD.

    Alternately, all modern NLEs have 'export to tape' functions. Just record your final product back out to your DV deck or DV camera and make a master archive on tape.
  • by trance9 (10504) on Sunday June 26, 2005 @12:11PM (#12914706) Homepage Journal
    Lots of gmail accounts. Lots, lots of them.

An inclined plane is a slope up. -- Willard Espy, "An Almanac of Words at Play"

Working...