How Do You Sync & Manage Your Home Directories? 421
Posted
by
kdawson
from the kitchen-sync dept.
from the kitchen-sync dept.
digitalderbs writes "A problem plaguing most people with multiple computers is the arduous task of synchronizing files between them: documents, pictures, code, or data. Everyone seems to have their own strategies, whether they involve USB drives, emailed attachments, rsync, or a distributed management system, all of which have varying degrees of success in implementing fast synchronization, interoperability, redundancy and versioning, and encryption. Myself, I've used unison for file synchronization and rsnapshot for backups between two Linux servers and a Mac OS X laptop. I've recently considered adding some sophistication by implementing a version control system like subversion, git, or bazaar, but have found some shortcomings in automating commits and pushing updates to all systems. What system do you use to manage your home directories, and how have they worked for you for managing small files (e.g. dot configs) and large (gigabyte binaries of data) together?"
Dropbox (Score:5, Interesting)
I recently started playing around with Dropbox for some smaller folders than my entire home directory and haven't yet run into any major problems. And the versioning it provides is nice as well, and as a plus they don't consider the deleted files that they still retain versions of as part of the quota.
Myself... (Score:3, Interesting)
Not strictly "home directory", but... (Score:3, Interesting)
Strictly speaking this isn't about my "home directories", but I use a combination of things to sync the information I need between my desktop and laptop:
1. Windows Live Sync (formerly known as Foldershare). Yeah, I know, it's from Microsoft, and only available for Windows and OS X, but it works pretty seamlessly and without much intervention on my part. I sync specific directories of projects I am working on using this; it's a set it and forget it tool.
2. Foxmarks for bookmarks.
3. Zotero, version 1.5 of which has the ability to sync resources between multiple computers. This is a godsend.
The old-fashioned way? (Score:5, Interesting)
backintime, and rdiff-backup (Score:5, Interesting)
For small backups, every ten minutes, I use backintime (based on rsync). For larger, nightly or more rare backups, I use rdiff-backup. Both work over the LAN, or to locally-mounted hard drives.
USB drive (Score:5, Interesting)
What about a FUSE FS powered by a MySQL DB? (Score:3, Interesting)
Re:Svn (Score:5, Interesting)
I use git, with flashbake and cron to automate commits, and a simply cron job to automatically update a backup copy on an external hard drive.
Windows - SyncBack (Score:4, Interesting)
I spent a long time tackling this, as I am situated at different locations on different days.
I have 2 desktops and a laptop which must remain sync'd and encrypted. I use TrueCrypt for the encryption.
On my Windows boxes - SyncBack handles it. It can be triggered on write or on insertion, or just periodically. Has version control support. Will sync over FTP (poorly) and can create zip files or burn Cds etc. It's a swiss army knife of sync tools.
The key for getting the most out of a sync program is granularity. Inevitably, you'll have exceptions, and you don't want a PASS/FAIL result for your entire backup set. It works much better to sort files into categories and sync the individual groups than to try to make one profile that does your entire disk array. My 2 cents.
FTP (Score:2, Interesting)
FTP back and forth, select the root and overwrite whatever's newer. Unless the time on the files gets screwed up, it works fine. Worst case scenario, which is the dates/times getting messed up, the FTP client downloads everything. No big deal. I do it daily for all kinds of files.
As older, wiser programmers than myself have always told me: KISS: Keep It Simple, Stupid.
Unison (Score:5, Interesting)
Re:Different tools for different purposes (Score:3, Interesting)
I agree with GP, that different tools are good for different purposes. I personally use Git across several computers for code, documents, and other small files. Pictures go to my Flickr Pro account.
Re:Windows users? (Score:4, Interesting)
For home use (a mix of Linux and Windows boxes) where things are more involved I started using Unison [upenn.edu] for a cross platform solution but in the end settled on a simple RSync for the Linux data and SyncBack SE [2brightsparks.com] for the more complicated Windows stuff. SyncBack SE might not be free (it's $30), but it is lightning fast, extremely flexible and can handle very sophisticated synchronisation and backup tasks including versioning, support for more than one target, remote targets via FTP and email), bandwith controls... Worth a look!
Unison plus rdiff-backup (Score:1, Interesting)
Re:Dropbox (Score:3, Interesting)
Re:Unison is the only way (Score:3, Interesting)
I've been a very happy Unison for many many years. But after recently switching to Ubuntu for my laptop and desktop I finally discovered the joy of Unison over SSH and also discovered the auto=true and terse=true command switches which makes the whole thing automatic, just like Dropbox (I think there's one more switch and I won't even have to press 'g' to run the sync).
I've tried other solutions - DropBox is too much in the 'cloud' for me; rsync is only one way (what's with that?), Robocopy is Windows only, svn/hg/bzr/git store versions and all I want is two way sync.
Unison is the peak of perfection in two way sync; there is nothing else...
The right tools for the job (Score:4, Interesting)
I don't share EVERYTHING, but I share some things:
ZFS FTW (Score:3, Interesting)
My Linux home directory is pretty tiny - only the dot files for my Linux environment (.gnome etc). I keep all my work documents and files on my OpenSolaris fileserver where ZFS provides resilience using RAID1 and point in time restores using the snapshot capabilities. I NFS mount the ZFS filesystem to my Linux box and CIFS share the same filesystem to my Windows PC and Mac.
My MP3 collection and photo albums are handled by iTunes and iPhoto respectively, syncing with my iPhone. The Mac backs these up to a Time Machine disk which in reality is a ZVol on my OpenSolaris server published as a LUN using iSCSI.
The ZFS filesystems and volumes are backed up to external USB drive using the "zfs send" command.
Blatant plug: I've documented most of the above experience on my blog.
For bookmarks, I use Xmarks to synchronise with the cloud, and take notes using Evernote.
Unison; and maybe git in the future. (Score:4, Interesting)
Currently? Just unison -quiet, running from cron. (I have it wrapped in a script that does locking, since Unison doesn't seem to lock against itself reliably, for reasons I don't understand.) I've had two problems worth watching out for:
1) Try to avoid running it against NFS. It walks the entire synced area every time you sync. Local disk will be two orders of magnitude faster.
2) Be careful syncing between case-sensitive and case-insensitive filesystems. Unison will start failing out if you ever create two files differing only in case.
Beyond that, I'm looking to start using git to version both my code and my textual data. I'm not intending to use git itself to sync the repositories; I'm going to use it for versioning only, and keep syncing using Unison. The reason is because I'm the only user, and for my own convenience I'd like the working copy to be synced. All I really need out of git is versioning anyway; I already have a workable solution for syncing.
Re:Svn (Score:4, Interesting)
Or how about, why on earth would I use something like CVS for files (movies, mp3 files, photos of my kids) that can be quite large and will never change?
I too am looking for things to help manage the huge piles of various files I have accumulated and am leaning towards something like beagle http://beagle-project.org/Main_Page [beagle-project.org] and rsync/unison for backups.
Ultimately though I think dividing my files up into meaningful directories is a good start, especially if I start by putting everything that doesn't change into a subdirectory of a main directory named "Static".
rsync + OpenSolaris (ZFS) w/time slider (Score:3, Interesting)
Nighly (or more frequently, if you like) rsync to an OpenSolaris server running ZFS w/ Time Slider.
Quality versioned backups with little effort, plus data integrity (checksums built into the filesystem), compression, and (if desired) RAID-Z(2) goodness! In addition, the provided time slider interface allows easy browsing of versions.
Just my 2c...
shfs mounts by cron, rsnapshot (Score:4, Interesting)
I use shfs mounts by ( to make sure it stays mounted even if connection is interrupted) and ssh tunnels for everything else, with preshared keys to a central server / proxy, and rsnapshots for backup on the central server with hot swap drives.
This works on desktops, remote office, and for notebooks. I essentially don't trust my employees or myself to remember to encrypt everything or use "secure" protocals all the time, and so I remove the need to remember from the whole process. I can then focus on securing one system. Great if everything else is secure, but just in case. Very good for notebooks jumping from open wireless to open wireless systems, and also keeping track of employees activity in one location. I can log fairly easily everything they do or don't do (yea, the 2 hour coffee break sticks out like sore thumb in the logs).
Among other things this also has the nice side effect that should say a notebook or desktop be stolen, it will phone home as soon as it is connected to the internet and send detailed information about what it is doing.
Re:Dropbox (Score:3, Interesting)
Rsync and rdiff-backup (Score:1, Interesting)
I use cygwin rsync (via vixie's cron) to sync .git repos and large files. Though /etc (again to an external repo via a
files from my various Windows clients to my
Linux backup host. On my Linux backup host I
use rdiff-backup (again as a cron job) to
backup / sync to an external USB drive. I
tried gibak to externally version my $HOMEs
(.git as a symlink to an external USB drive
filesystem) but had problems (git was not
built to manage 20GB of content)
Anecdotally, git had problems processing
nested
git was a poor choice in versioning my $HOMEs,
it proved to be a great choice for versioning
my
symlink). For this, I used metastore and
etckeeper (a git wrapper utility).
Re:"Distributed homedirs" or "CVS'd configs"? (Score:3, Interesting)
Am I the only one experiencing this, or do y'all think it's some kind of trend? It could honestly just be that I messed up some settings or don't know what I'm doing, but F9 worked so well that I'm tempted to just go back to it. Ubuntu is of course an option too, but one i haven't explored much yet. but with all of the options suggested here, i probably have a lot of options to look into. rsync works brilliantly for backups. still, i would prefer NFS working right again, because the peace of mind knowing that any one client on our network can go down without taking anyone's data with it and i can add a new client with so little work has been really nice.
I'm open to suggestions, but since this isn't the 'ask slashdot' section, I'll just summarize what I can contribute to the thread: NFS, as eln says, works very well when your network is well-wired and stable, but is useless for home dirs on notebooks that will be used away from the LAN. and Fedora 10 and 11 have given me problems with NFS home dirs.