Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Data Storage Software

Automated CD/DVD Archival? 75

Posted by Cliff from the a-complete-backup-system dept.
An anonymous reader asks: "Our department used to use a Cedar Technologies Desktop CD-R Publisher for fully automated backup of data (~2 CDs per day) controlled by a Linux PC. The publisher just broke and we are looking into a new backup solution to automatically burn and print CDs or DVDs. Solutions for CD/DVD duplication are available for Windows and Mac (for example: Primera and Rimage [which acquired Cedar in 2000]) but not for Linux. While a Mac would be OK, none of the manufacturers seems to offer scriptability or a command line interface which is essential for our task. Tape and HD backup are not an option - the data is already mirrored on RAIDs. Has anyone set up a similar archival system using Linux?"
This discussion has been archived. No new comments can be posted.

Automated CD/DVD Archival? More

Automated CD/DVD Archival?

Comments Filter:
  • You can take it to home/beach in the weekends.

  • One word: SCSI. (Score:2, Informative)

    by yppasswd ( 538509 )
    Just go with a SCSI jukebox; it should work fine on Linux
    with "mt", "mtx" and some shell woodoo; ours did.
    You might perhaps check if your vendor supports standard
    SCSI commands, though.

  • And? (Score:2)

    by bconway ( 63464 ) *
    Tape and HD backup are not an option - the data is already mirrored on RAIDs.

    What does one have to do with the other? Why are you insisting on CD backups when there are superior solutions available? It sounds like you're intent on duplicating a poor solution instead of examining the problem as a whole and finding the best solution for your needs.

    • Xor? (Score:5, Insightful)

      by yppasswd ( 538509 ) on Tuesday March 15, 2005 @08:46AM (#11942591)
      If he says "Tape and HD backup are not an option", well,
      I'd think he has considered all the other solutions.
      There are cases where CD archiving is the only solution.
      For example:
      - Legal requirements of read-only media. My case.
      - If archives must be guaranteed readable by common
      hardware (and I mean COMMON, try buying a tape reader
      in your favorite supermarket...)

      Sure, backup on CD is a pain, but this was not
      his question.
      • I don't know about you, but I usually am not sent to the supermarket by my company to buy a dds4 drive. We've got vendors for our equipment. :)

      • Re:Xor? (Score:4, Insightful)

        by bluGill ( 862 ) on Tuesday March 15, 2005 @11:06AM (#11943645)

        Several companies make write-once tape drives. We use Sony AIT drives at work, but there is no reason to assume the others won't work. (I know DLT has a write-once mode)

        Tape drives are common hardware for enterprize backup, more common that CD/DVDs. CD/DVD is consumer media, I it easy to show in court that they are not up to the other standards that anyone who must save data must maintain.

        • Re:Xor? (Score:3, Insightful)

          by superpulpsicle ( 533373 )
          The only reason why tape drives are still common for enterprise backups, is because disk backup solutions were not affordable before.

          Writing 40GB at 5MB a second is how tape manufacturers market to you. What they never tell you is how long it takes you to restore.

      • Re:Xor? (Score:3, Informative)

        by seigniory ( 89942 )
        HP has a line of WORM Ultrium tapes that satisfy your read-only requirement and provide all the performance and durability of standard tape backup.

        http://h18006.www1.hp.com/products/storageworks/ wo rmdps/related.html
      • Sure, backup on CD is a pain, but this was not
        his question.

        I don't think it's unreasonable to ask for clarification. The stated reason for Tape and HD not being options didn't make any sense.

        If you offer someone a cup of coffee and they respond "No thank you, I'm allergic to tea", it's only normal to ask for more information.

        It's most likely that the submitter has valid reasons for not being able to use HD or Tape backup, but s/he was extremely vague about why.

        LK
    • CD/DVDs sure seem like an ideal solution for making cheap (due to good frequency) backups for off-site storage. Excellent if the backups must be accrued and stored for longer than a week or two.

    • Re:And? (Score:4, Informative)

      by Kz ( 4332 ) on Tuesday March 15, 2005 @09:28AM (#11942896) Homepage
      As pointed by another poster, there are situations where CD/DVD are superior to either Tape/HD.

      but the existance of RAID mirrors have nothing to do with that.

      a mirror protects you from hard disk failure, not against data corruption, for that you need a different thing.

      also, the original poster uses 'backup' and 'archive' words. these are _totally_ different things! I think he means: i need archive, i had archive with CD/DVSs, i don't want tape/HD backup. but it seems he hadn't checked up the possibility of an HD archive.

      I've recently replaced a wall full of DVDs (around 6000 discs) with a near-line HD archive system, at just 1.56$/GB total cost!

      of course, for offsite backup of the archive, it still burns DVDs
      • I think DVDs are a hell of a lot more reliable than HDs (and I doubt you got good-quality HDs for that price). IDE hard drives in my experience have a lifetime of about 1 year before they develop bad sectors (and corrupt your data). Recordable media lasts for many years when properly stored (no large temperature/humidity swings, no light exposure, etc).

        • Re:And? (Score:2, Informative)

          by DustMagnet ( 453493 )
          I think DVDs are a hell of a lot more reliable than HDs

          I find DVDs are much more reliable, once written. I have yet to find a long-term reliable DVD or CD writing solution. Yes, I can buy high quality drives and high quality disks, but I still get spindles with write problems. At home I just use cheap disks and find I'm saving money by tossing out spindles that don't work. It bothers me that sometimes I get no errors on writing, but get errors on reading.

          Anyway, I have yet to find a reliable way to

        • Re:And? (Score:2)

          by Kz ( 4332 )
          optical discs are really realliable when properly stored, unfortunately that means not being used. Here in Lima we get more than 90% humidity most of the year, so even in well ventilated, A/C indoors, optical discs degrade quickly.

          the main advantage of HDs is that it's easy to check data integrity with checksums. of course, HDs do die, that's why now all those DVDs are the backup of the archive. And now that they're not used except when accidents happen, they can be stored properly.

  • Google (Score:2, Informative)

    by Pan T. Hose ( 707794 )
    You need some simple script driving mkisofs and cdrecord. When you Google for cdrecord mkisofs backup [justfuckinggoogleit.com], the first result is scdbackup [webframe.org]:

    Simplified Backup on CD or DVD for Linux

    Purpose:
    backup large amounts of data on one or more CDs or DVDs,
    simple (therefore no excuse not to do the backup),
    no special tool needed for reading the backup

    Sounds exactly like what you need. There are many more tools like that. Good luck.


    • If you read the friggin' article sumary, you would see that what is needed is an automatic CD-R loader, not just the software to burn CDs.

  • Let me tell you about the Rimage systems (Score:5, Informative)

    by everyplace ( 527571 ) on Tuesday March 15, 2005 @08:42AM (#11942561) Homepage
    Recently at work I had to recommend an automated cd printing / burning system, and I went with the Rimage 2000i. We're a mac-only design shop in nyc, and needed the machine not for backup, but more for one-offs with automated labels, in a machine that was networkable.

    One of the things on the Rimage website that's kind-of misleading (at least it was to me) was that it NEEDS a windows pc in order to share the rimage machine with other machines, like a mac. But once it's setup, the machine works wonders.

    What's interesting about this machine though, is that despite the ridiculous setup hurdles, after it all works they provide a fairly decent way of writing your own scripts to control the machine.

    The entire device uses xml files in order to handle job requests, and the client they ship it with is actually just a beast of a java app. But the xml files are used for the imaging orders, the production reports, everything. They also have a fairly extensive sdk that allows you to do pretty much anything.

    I had an unfortunately difficult time setting this thing up, but the tech support (while their english was a little lacking) were actually incredibly knowledgeable. One of the things they told me was that almost no-one who buys this device uses the provided client. It is designed to be integrated into custom work solutions, so for you this might actually be appropriate.

    If you're looking for a solid dvd archival device / printer that has an autoloading function and is fully scriptable, the Rimage 2000i (or any of their devices higher end than that one) could work.
    • Sorry, I forgot one point. The fact that the entire production ordering system operates solely on xml file handling (to me) means that this is really OS independent if you write the front-end for it. The only hurdle really is a) writing your own frontend for linux and b) accepting that you need a windows box to serve the device from. I'm currently experimenting with making a web-based front end for this, but its too early to say whether or not I'll have success.

  • Needlessly elaborate solution (Score:3, Interesting)

    by AndrossUT ( 721573 ) on Tuesday March 15, 2005 @08:48AM (#11942603) Homepage Journal
    take an old linux box with a burner. Set up an automated backup burn... and use a LEGO Mindstorm setup to pull out the disc, slap an autmatically printed label on, and put a new, blank disc in. Sure it's not the most cost effective or efficient way, but it would be damn cool looking.

  • amanda (Score:2)

    by martin ( 1336 )
    will backup to CD's no problem. www.amanda.org
    • however, they're looking for an automated setup - not just a burning program.

      they want the cd to pop out, and then some label machine label it and put it away.
      • all possible with the tape changer program..

        • Notwithstanding the OP's implied reliance on their RAID array as their backup strategy being a fatally flawed approach, and tape backups for that RAID a potentially recommended course of action because archival grade discs cost more per gigabyte than tape for starters, recommending they use tape when they have obviously thought of it and rejected the approach might not be fulfilling certain business requirements they didn't share with us. For all we know, they already have an overwhelming, substantial inves

          • Re:amanda (Score:3, Informative)

            by Noksagt ( 69097 )

            recommending they use tape when they have obviously thought of it and rejected the approach might not be fulfilling certain business requirements they didn't share with us.

            He didn't recommend tape. He suggested AMANDA with optical media.

            For your amanda suggestion to be complete, you need to offer a solution that also automatically labels the tape cartridges.

            AMANDA supports barcode readers. It can also automatically print labels. If you have the hardware to automatically apply the labels to the media, you

        • maybe yes.

          but a program doesn't become magically hardware that takes the cd out and writes on it.

          (and he states that he doesn't want tapes, doesn't state the reason though)
  • Hire a monkey. Or a college student if you want cheaper.

  • Linux/Primera (Score:3, Informative)

    by wonkavader ( 605434 ) on Tuesday March 15, 2005 @09:21AM (#11942841)
    I must say I have been horrified by my experience with a Primera Bravo box. Not because it's bad -- it's really great -- but because there are no linux drivers, and the Windows methods are absurdly awful.

    To burn a disk, you go into a GUI and mockup (or just load) an image to print on the disk.

    Then, you print it to file -- something.prn

    Then you go into another GUI and set up the task, picking an ISO image, and the image file you just made, click here, click there, then burn.

    That works just fine for 40 of the same disk, but if you want a different image on each (different date or different text, or the ISO filesize) you need to make each change manually (or with tags) and then print to file and then set up each task.

    Man.

    In unix/linux, or with command-line tools for windows, even, that would be:

    create_postscript_with_substitutions [inputs] > printerfile.prn
    burn_image_and_print isofile printerfile.prn

    Done. You'd be able to do everything this guy wants and more with 10 seconds of typing. You'd be able to automate processes. And it's not hard. Primera's been selling this stuff for years, and yet, no Linux support, and no command-line support.

    If this had Linux support, or even DOS command-lines, I'd recommend it to everyone I meet. As is, it's an anchor.

  • Splitting Discs (Score:4, Interesting)

    by Wiwi Jumbo ( 105640 ) on Tuesday March 15, 2005 @10:01AM (#11943096) Homepage Journal
    I'd like to do (somewhat) automated backup's as well... tho more on a homesize scale...

    I'm trying to back up my pictures and my mp3's on a regular basis, but at 15 Gig's for one and 56 Gig's for the other even putting them on DVD can be a pain.

    Every app I've used (at least on XP) can automatically "split" the files over multiple discs but they all use their own format for it making recovering the file difficult if not impossible with out the original program.

    Is there anything that will split on the file structure so I can just read the files like they were burned normally?

    Right now I keep adding and removing files from the "to burn" list and try and get as close to each disc limit and then do the same for the next disc... makes me put off my backups for longer then I should....

  • Ack Thpt (Score:5, Insightful)

    by binaryspiral ( 784263 ) on Tuesday March 15, 2005 @10:07AM (#11943139)
    Burned DVDs and CDrs have piss poor shelf life for archival material. RAID arrays can disappear at the snap of a raid card's whim (yes - it happened to me last week).

    Please - re-evaluate your solution before you actually need to recover data or get (gasp) audited and need every single file to work 100% for the auditors.

    Recordable optical media sold retail are not up to the standards of archival requirements of most governing bodies (like the SEC). There are optical WORM drives that are used by medical data centers and hold only 30GB on a huge platter. And most of those are getting retired for other methods.

    (this is not a plug, it's just what works best for us)
    There are many tape solutions like Exabyte's VXA-3 with 160GB native storage space on an $80 tape. Granted $/GB is higher than a DVD-R - the tape will not let you down. The tape is equal to ~35 DVDs and writes at 500MB+/minute.

    We have an Exabyte autoloader with 10 tapes on a weekly rotation - and it was as close to heaven when we needed to restore a server. We also backup >400GB of data weekly from a few of our database servers - and need it to be there when something fails.

    The entire rig will set you back about $3000 including tapes. This will give you over a TB of backup space. And the tapes are archival ready.

    For your use - 1-2CDs/Day each tape will last you about 3-4 months. But it will also allow you to rotate your backups off site and give you much better utilization of space and much higher chances of recovering this data years down the road.

    Please reconsider your backup solutions... if it's worth saving at all - it's worth being able to get it back later.
    • Granted $/GB is higher than a DVD-R - the tape will not let you down.
      And the tape can be ERASED and REWRITTEN, which may reduce media costs considerably. Most tape media is cheaper per gig than most DVD-RWs.
    • writes at 500MB+/minute

      Sorry, I think you meant 500mbit/s, not 500MB/s.
      • No, I meant 500 Megabytes per minute. Some research [exabyte.com] may have helped you respond.

        VXA says you'll get 43GB per hour, but in a real life you'll see ~500MB per minute which works out to about 30GB/hour. With that speed, we have backups dumped on to a 1TB NAS then written to tape. It allows the backups to flow as fast and smooth as possible to tape without having to worry about network congestion.
    • If you're referring to those Super DLT tapes that hold 160 GB uncompressed and 320 GB compressed, then I know which type of data tapes you're referring to. I once worked in a data center that stored some of its backups onto those Super DLT tapes, and those tapes were kind of flimsy with regards to the latch on the ends of the tape media itself that was used by the tape robots to pull the tape out of the spindle. Those plastic latches would break off a lot. The data medium itself may be more reliable than
      • Similar to SDLT, but not quite.

        SDLT Tape [transtec.de] vs VXA-2 [ibm.com].

        VXA-3 will exceed the capacity of the SDLT and will be the exact same size, using the same tapes and autoloaders. All I have to do is swap out the drive and gain double my storage.

  • Microtech (Score:3, Informative)

    by bill_mcgonigle ( 4333 ) * on Tuesday March 15, 2005 @10:29AM (#11943314) Homepage Journal
    Check out Microtech's [microtech.com] ImageAutomator line.

    It's windows-based but I've set up a few of them to pull their data from a Samba share. Think of it as an appliance. I wrote some software on the linux side to control it - they have specs available for their file formats if you want to explore writing your own. That software has probably done about >30K CD's so it's definitely workable.

  • Pioneer DRM 3000 (Score:4, Informative)

    by Ayanami Rei ( 621112 ) * <rayanami@@@gmail...com> on Tuesday March 15, 2005 @10:48AM (#11943479) Journal
    Product Brochure [pioneerelectronics.com]

    This is probably overkill, but it is a really cool piece of equipment, and it doesn't rely on shitty windows software to do it's job. Unfortunately it costs $10000 fully loaded with 4 DVD-RW drives.
    • We've got one of the 7000 units, bought during the heady dot-com days.

      The Windows-based management tool sucks, lemme just say that. But the cool part is that it can share a virtual drive on the network, and then when people make folders off the root directory, those folders are archived as DVD's. So you can let your users set up their own folders and archives without having to manage anything. The system will automatically burn them when they reach DVD capacity, or when you specify to do it in the manag
      • Who the hell thought that using a DVD array would be better than a disk array/SAN? Just the thought of a mechanical anything moving constantly to keep up with random access makes my head hurt.
        If you don't have at least 16 DVD readers it's pointless. You'd need at least that just to saturate SCSI bandwidth. And all those moving parts. GAASAAAH.

        And $10000+ when 1TB RAID would cost the same at the time of purchase... double GAAAH.

        *Cough cough* *wheeze*

        I sure hope they fired the guy who thought of that brill

  • RAID != A Backup Method (Score:4, Insightful)

    by seigniory ( 89942 ) <bigfriggin @ m e . com> on Tuesday March 15, 2005 @11:26AM (#11943860)
    RAID isn't going to help you if your building goes up in flames.

    RAID isn't going to help you if a file is deleted accidentally.

    RAID isn't going to help you when someone comes in and steals your boxen.

    In 15 years I have never, ever, EVER recommended that someone back up to optical media as their only recovery method. DAT drives can be had in the sub-$200 range, and the $/MB cost is cheaper than DVD media - and much more reliable.

    I realize that this doesn't really answer your question, but it's an important point that shouldn't be overlooked.
    • I don't know about you, but all my data will be fine when my building goes up in flames.

      Rsynced to a remote RAID device. Accidentally delete a file? I suppose setting up an undelete function would work.

      And as for stealing my boxes. Get a guard dog already.
      • Find an "undelete function" that will recover your data when a user "accidentally" runs a SQL package against the wrong data. Or one that will roll back to a version from 3 days ago when a (l)user in your company realizes that someone else got rid of a chapter that took 3 weeks to write.

        Recovering a deleted file != recovering altered data (which is far more common).

        As for your diaster recovery / business continuity preparedness (and your assumed desire for continued employment), you might want to take a

        • Find an "undelete function" that will recover your data when a user "accidentally" runs a SQL package against the wrong data.

          Shrug. I keep a nightly SQL dump on my RAID. That's not what the undelete function is for. You said "When a user accidentally deletes a file." Not "When a user totally screws up and pillages the system."

          Or one that will roll back to a version from 3 days ago when a (l)user in your company realizes that someone else got rid of a chapter that took 3 weeks to write.

          And they don't

          • re there DR classes that don't require you to have expensive backup solutions?

            DR (Disaster Recovery / Readiness) is really more a state of planning and procedure and is (or shoudl be) independent from technology. Sure, tech is (obviously) involved, but should never be the driving force.

            From a higher level, take a good look at what you'd need to do if, in a worst case scenario, you lost everything - pretend that "Poof!" it doesn't exist anymore. If you can figure what it takes to get 100% of your shizn
            • (blink)

              You work in enviroments that I've only heard of in legends.

              I work in places where "The tapes are going bad, and they're expensive to replace, and the tape drive is years out of warranty, fire the operator!"

              Man, a proper DR budget. If they could afford that, they could afford to hire people instead of hacking their budget and getting temps/contractors.

              Ah well. I've got to get back to working on my local RAID array. The array went corrupt, and I'm hoping upgrading to the 2.6 kernel will help.

  • on linux for a few years now. It's actually a fairly quick hack that I've never gotten around to fixing up. Each of my computers runs a backup program at 4am on monday morning, and saves a tar.gz of the dirs. The tar.gz is owned by an account that is not used by anything else. Then, at 5am, my main computer (the one with the burner) uses scp (with dsa id keys) to download the tar.gz files. It then mkisofs's and cdrecord's, so that I have a CDRW backup of my data.

    The programs are pretty crap (only ver

  • Depending on your needs, you might be able to get away with a handful of DVD burners. If, for example, you know that a backup will always take less than 3 DVDs, buy 3 or 4 burners. Then, make your backup, split it, and write a piece to each burner.

    Spending a few hundred dollars on burners may be better than spending a few thousand on a robot.

  • RAID1 Array as backup (Score:3, Interesting)

    by stefanlasiewski ( 63134 ) * <slashdot AT stefanco DOT com> on Tuesday March 15, 2005 @01:51PM (#11945393) Homepage Journal
    There are many comments about how RAID isn't a backup scheme. While that's mostly true, I've known some shops that actually do use a RAID1 array as a backup scheme.

    Not sure of all the details, but it works like this:

    1. Setup 2 disks with RAID1 mirroring. Designate one as the backup disk.
    2. Copy files to the mirror
    3. Once a week, unmount the backup disk, take it out and insert a new disk.
    4. Turn on the machine. When the array comes back online, the disks sync.

    I'm still trying to figure out if this is a wise idea, but it's low-cost and effective. 1 Device and 4 decent-quality 200GB disks will cost you what-- $1200? How much do 200GB tape solutions cost now?

    Anyone hear of some similar solutions?
    • That's a gheto backup! But man, it would work.

      Sure puts your drives and raid controller (or CPU if you're running software raid) through hell for a few hours.

      I hope nobody plans on using it for anything :)
    • That is technically not RAID. Its kinda of a hack using RAID software. I do the same thing with a script that I call 'raid1.pl', but its not a RAID system at all. I simply use rsync to sync up two drives at night.

      The difference between this and real RAID1 is that with RAID1 you do 'rm -rf /something/important' and its gone from both disks (or an overwrite, or anything). In my experience, if a backup is done nightly, even if a whole day's work is lost, it is preferable to having potentially YEARS of dat
  • Please don't confuse redundancy and archival needs. To simplify: Redundancy (e.g. RAID) protects you against hardware failures. Archival protects you against "oops I deleted it" failures. Both are important.

    $0.02,
    ptd

  • http://www.akfentertainment.com/vcic/encore.htm

    How well it supports is anyones guess. We have an old Cedar CD duplicator that I would love find the protocal for. The strange thing is the robotics are controled via SCSI.
    Anyone know how to capture SCSI data?

  • rimage 2x burning rant (Score:3, Interesting)

    by tweedlebait ( 560901 ) on Tuesday March 15, 2005 @03:04PM (#11946158)
    2 years ago-
    Worst tech support experience I've consistantly had was with rimage and it lasted over 1 month!! All I was trying to do was replace a broken drive they had assured me was standard way back in the presale days. They had no replacements, wanted me to buy a new unit at $6k. My old software which was great in its earlier days had gone through some 'improvements' post veritas buyout and wasn't working anymore. Rimage had a new $$product that was supposed to replace all these features and more. I demoed it all and it was horridly inefficient- IIRC- 4 cd-r's took 3 hrs to make 4 cds, nevermind the crashes. The old unit with 4x drives was much faster. Then they too told me nobody actually uses this out of the box, they just buy the dev kit and customize it. Then they recommended I do the same for my team of developers. Sounds like a handy tech center stock answer. Guess what. I dont have a team of developers! I suggested they use their team of developers to make their own stuff work as advertised.

    After much strife they finally provided me with someone at another company who could provide me with a copy of their customized firmware so i could replace my drives with identical ones. The robotics finally gave up the ghost a year later.

    My replacement for it was from a less known company handled through discmakers. Discjuggler runs it with a special autoloader product called imagejuggler which works very nicely (you can also use DJ.NET and they have a web interface too. They could improve a few things but all in all robotics are top notch and simple, no weird firmware, replacable (upgradable!) drives (as long as your burning app supports it, which dj handles many) and I can easily burn different images simultaneously. Furthermore both discmakers and DJ support was great.

    http://www.discmakers.com/duplicators/products/eli te_series.asp [discmakers.com]

    and yup its only for windows. but at least you get browser ui.

    • We're using DVDs to back up 5 2TB NAS servers of video files. Our hardware is a 4-burner Flexwriter, similar to this one: http://www.amtren.com/products/sa4.shtml [amtren.com]

      The drivers come from Padus. We don't use their GUI much (DiskJuggler and ImageJuggler)...instead we primarily build jobs and submit them from a another machine via a perl script and their command-line tool, pfcnet. The level of automation we needed for this project simply required scripting. I haven't looked into their DJ.net product...it may ha

  • Copy shows I tape onto a DVD with interface done with script. Any possibility? That'd be a good ask /.
  • Rimage Developers API [rimage.com]

    The Rimage Client API :
    Tight integration with your application
    Order and system status in real time - no polling
    Fault tolerance
    APIs in C, C++, VB, Java, .NET
    Industry standard XML support
    Multi-platform support

    Rimage Network Publishing Multi-Platform :
    Monitors a network folder for text file orders
    Generates XML from file orders and submits to Rimage system
    Multi-platform support

    Based on poking around in (our company owns the DiskLab) the SDK, it'll do almos

Slashdot Top Deals

You are always doing something marginal when the boss drops by your desk.

Close