Ask Slashdot: Asynchronous RAID-1 Free Software Backup For Laptops? 227
First time accepted submitter ormembar writes "I have a laptop with a 1 TB hard disk. I use rsync to perform my backups (hopefully quite regularly) on an external 1 TB hard disk. But, with such a large hard disk, it takes quite some time to perform backups because rsync scans the whole disk for updates (15 minutes in average). Does it exist somewhere a kind of asynchronous RAID-1 free software that would record in a journal all the changes that I perform on the disk and replay this journal later, when I plug my external hard disk on the laptop? I guess that it would be faster than usual backup solutions (rsync, unison, you name it) that scan the whole partitions every time. Do you feel the same annoyance when backing up laptops?"
mdadm can do this (Score:5, Informative)
Obligatory (Score:5, Informative)
RAID is not backup.
Re:Obligatory (Score:5, Informative)
True. I'd recommend he check out rdiff-backup, which keeps snapshots of previous syncs. Fantastic tool.
Exclude directories (Score:5, Informative)
Are you backing up EVERYTHING on the laptop -- OS and data included? Even if you are only backing up your home directory there is stuff you don't need to backup like the .thumbnails directory which can be quite large. Try using rysnc's exclude option to restrict the backup to only what you care about.
DNA
AKA mrascii
Re:Time Machine (Score:4, Informative)
TimeMachine takes about 15 minutes to do the prep work before it starts copying for me, on a 2012 Retina MBP with 16Gb of RAM and only 256GB of disk space ... 64 GB taken by an unbacked up BootCamp part and another 120 or so eaten in Windows VMs that don't get backed up either ... i.e. Its not a slow spinning platter backing up a terabyte of data.
I see no indication of any Journal, it certainly isn't making it faster. Pretty freaking slow actually.
COW or desync'ed RAID (Score:5, Informative)
In this case, it sounds like you want a fast on-demand sync rather than a RAID.
However, you could possibly use dm-raid for this if you're a linux user.
Have the internal disk(s) as a degraded md-raid1 partition. When you connect the backup disk, have it become part of the RAID and the disks should sync up. That said, it likely won't be any faster than rsync, quite possibly slower as it'll have to go over the entire volume.
Alternate solutions: /home is a combination of /mnt/home-ro (ro) and /mnt/home-rw (rw, COW filesystem). When external media is connected, /mnt/home-rw is synced to external media, then back over /mnt/home-ro
* Have a local folder that does daily syncs/backups. Move those to the external storage when it's connected.
CAVEATS: Takes space until the external disk is available
* Use a differential filesystem, or maybe something like a COW (copy-on-write) filesystem. Have the COW system sync over to the backup disk (when connected) and then merge it into the main filesystem tree after sync
For example,
Re:mdadm can do this (Score:5, Informative)
Effectively you create a RAID 1 mirror. When you remove the external drive the RAID degrades. The raid bitmap keeps track of changes. When you plug the external drive in you just have to tell it to bring it up to date. Which syncs the only changes.
CrashPlan (Score:4, Informative)
CrashPlan [crashplan.com] is free, but not open, and I think will do everything you need. You can backupto an external disk, over the network to one of your own machines, or back up to a freind who also runs it. Great key based encryption support. If you want, you can pay them for offsite backups (which is a great deal as well, in my opinion). It's cross-platform, and easy to use. Never underestimate the benefits of off-site backups.
Re:find & diff (Score:5, Informative)
It's different in that you don't have to sit and wait for it and doing the backup will consist of only the actual copying
I suggest you look again at rsync. /proc and there are some directories such as /tmp and /run which you may not care about).
- It compares changed files and copies only what has been changed. Changed files are identified by differing mtimes (by default).
- rsync can also handle removed files with the --delete option.
- It can do the entire filesystem tree in a single command
- There are filter options so you can include/exclude what paths to copy (eg you don't want to copy
Re:mdadm can do this (Score:4, Informative)
Re:Do it on a lower level. (Score:4, Informative)
Well, of course I goofed, it's not that easy (well it is, read on). A snapshot keeps track of what has changed, yes, but it records not the new state, but the old state. What you want to transfer over is the new state. So you can use the snapshot for the location of changed state (for its metadata only), and the parent volume for the actual state.
That's precisely what lvmsync [github.com] does. That's the tool you want to do what I said above, only that it'll actually work :)
Re:Obligatory (Score:3, Informative)
Upgrade your rsync! (Score:5, Informative)
You're holding it wrong. ;)
rsync 2.x was horribly slow as it would scan the entire source looking for changed files, build a list of files, and then (once the initial scan was complete) would start to transfer data to the destination.
rsync 3.x starts building the list of changed files, and starts transferring data right away.
Unless you are changing a tonne of files between each rsync, it shouldn't take more than a few minutes using rsync 3.x to backup a 1 TB drive. Unless it's an uber-slow PoS drive, of course. :)
We use rsync to backup all our remote school servers. Very rarely does a single server backup take more than 30 minutes, and that's for 4 TB of storage using 500 GB drives (generally only a few GB of changed data). And that's across horrible ADSL links with only 0.768 Mbps upload speeds!
Going disk-to-disk should be even faster.
ZFS - incremental/snapshot? (Score:5, Informative)
two pools, internalPool, externalPool
use ZFS send and receive to migrate your data from internal to external, you and do whole fs or incremental if you keep a couple of snaps local on your internal disk, this can get excessive if you have a lot of delta or you want a long time.
http://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html [oracle.com]
of course you will need a system that can use ZFS, there are more options for that than time machine, its block level and its fast, and it doesn't depend on just one device, you can have multiple devices (I like to keep some of my data at work, why? my backup solution is in the same house that would burn, if it burned...)
Btrfs send/receive (Score:5, Informative)
Btrfs send/receive should possible be doing the trick. After first cloning the disk and before every subsequent transfer create a reference-snapshot on the laptop and delete the previous one after the transfer.
$ btrfs subvolume snapshot /mnt/data/orig /mnt/data/backup43 /mnt/data/backup42 /mnt/data/backup43 | btrfs receive /mnt/backupdata /mnt/data/backup42
$ btrfs send -p
$ btrfs subvolume delete
I havn't tried this for myself, so the necessary disclaimer: this may eat your disk or kill a kitten ;-)
Re:Whooosh (Score:4, Informative)
My solution on windows is to take some of the directories with 10,000 files and put them into an archive (think clipart directories).
I hope your are not an IT professional. Windows comes with a perfectly good backup solution built-in. It will use Volume Shadow Copy Service (VSS) to track changes as they occur and subsequently only do backup of the changes blocks. No need to scan *anything* as the journaling file system has already recorded a full list of changes in the journal.
The backup is basically stored in a VHD virtual harddisk (and some catalog metadata around it), so you can even attach the VHD and browse it. It will by default let you browse the latest backup, but the previous versions feature will let you browse back in time to any previous backup still stored in the VHD (oldest backups vill be pruned from the backup when the capacity is needed). The VHD is a inverse incremental backup because it stores the latest backup as the readily available version and only the incremental (block level) differences between previous backup sets.
Moreover, VSS also ensures persistent consistency for a lot of applications that are VSS aware (VSS writers), i.e. database systems like Oracle, SQL Server, Active Directory, registry etc. VSS coordinates with the applications so that exactly when the snapshot is taken, the applications ensure that they have flushed all state to disk. This means that applications will not need to be stopped to get a consistent backup, i.e. database systems will not see a restore of a backup that was taken from a running system as a "crash" (as they would without such a service) from which they must recover through some other means (typically a roll-forward log).