Forgot your password?
typodupeerror
Data Storage

File Organization — How Do You Do It In 2011? 356

Posted by timothy
from the let-god-sort-'em-out dept.
siddesu writes "After 30 years of being around computers, I have, like everyone else, amassed a huge amount of files in huge amount of formats about a huge amount of topics. And it isn't only me — the family has now a ton of data that they want managed and easily accessible. Keeping all that information in order has always been a pain, but it has gone harder as the storage has increased and people and files and sizes have multiplied. What do you folks use to keep your odd terabyte of document, picture, video and code files organized — that is, relatively uniformly tagged, versioned, searchable and ultimately findable, without 50 duplicates over your 50 devices and without typing arcane commands in a terminal window? I found this discussion from 2003 and this tangentially relevant post from 2006. How have things changed for you in 2011? And how satisfied is your extended family with the solution you have unleashed upon them?"
This discussion has been archived. No new comments can be posted.

File Organization — How Do You Do It In 2011?

Comments Filter:
  • Directories (Score:5, Informative)

    by Anrego (830717) * on Sunday February 13, 2011 @12:46PM (#35192644)

    .. seriously.. they still work for me.

    I’ve got a 12TB file server (~6TB filled). It’s arranged as follows:

    documents/
    incoming_downloads/ (before you ask.. yes.. _legit_ downloads)
    media/
    media/video/
    media/video/movies/
    media/video/tv_shows/
    media/video/tv_shows/some_tv_show/
    media/video/standup
    media/video/etc..
    media/music/
    media/images/
    media/images/various_subfolders/
    code/
    virtual_machines/
    tmp/
    backup_links/
    backups/

    That’s always been enough for me. Never got into all this tagging/meta data stuff. If there’s anything I’d ever want to search on... I put it in the file name. Indexed every night via slocate.

    backup_links is part of my hacked together backup system.

    The thing is raid6, setup so two drives can fail without loss of data. I see this as adequate “backup” for stuff that is replaceable (the large portion of my media is rips of DVDs I own... so although it would be a huge pain in the ass to re-rip them all... it’s not impossible). Stuff that is irreplaceable, I backup to separate hard drives (via hot swap trays).

    I leave one backup drive plugged into the machine, and keep the other elsewhere. I periodically swap these drives. I have a script that just rsyncs the files and directories pointed to in backup_links (the irreplaceable ones) to the currently plugged in drive (and yes I verified that I’m not getting a backup of my links ;p). This way I always have one drive that has a pretty recent backup (runs nightly), and one drive that has at most a month or so old backup if the plugged in one fails for some reason.

    backups is backed up files from other machines.

    Keeping everything in one place helps with the organization I think. Most of the other machines on this network are basically just OS installs. All the real files are on the file server. My desktop runs of a small SSD, which is not even half filled.

    • No Porn? (Score:4, Funny)

      by Anonymous Coward on Sunday February 13, 2011 @12:48PM (#35192652)

      I think you left a directory out. ;)

      • What do you think is in media/video/movies/ ?

        Joking aside, that's pretty much what I do. I've never had some turning-point moment where I've thought "I need to do this differently", and it does still Just Work.

        Though my collection is way under 1TB at the minute, so I suppose i'm still where everybody else was about five years ago.

      • Re:No Porn? (Score:5, Funny)

        by Anrego (830717) * on Sunday February 13, 2011 @12:53PM (#35192678)

        media/video/etc..

        I figured it didn't even need to be said ;p

    • Re:Directories (Score:5, Interesting)

      by RuiFerreira (791654) on Sunday February 13, 2011 @12:52PM (#35192666) Homepage
      I basically use the same structure as you but I have an extra directory called "attic" where in practice I end up putting everything.
      • Re: (Score:2, Funny)

        by Anonymous Coward

        I used to download stuff directly on my desktop (my habits have since then changed). One day, my desktop was completely full (meaning that there was no more iconspace, I know I could've just used an Explorer window but it was too much of a hassle), so I made a folder called "shit", and I dragged all of my stuff in there (most of the stuff was shit, so the folder was aptly named). It took a month or two to refill my desktop, then I made another folder called "more shit", and I again dragged everything on the

        • by Jorl17 (1716772)
          Back in the day when I was 11 and used Windows, that's what I used to do. I just didn't call it "shit", I called "Old Desktop Stuff". I ended up wasting 5 DVDs to save all those folders when I migrated to another machine.
      • Mine's even simpler than your attic: I just throw absolutely everything the / directory.
    • by traindirector (1001483) on Sunday February 13, 2011 @01:07PM (#35192744)

      I also still use a similar directory structure, but I've made once change in the past few years that makes it much easier to manage: I keep the special, personal, irreplaceable in a separate hierarchy.

      This negates the need for something like a backup_links directory, and makes it much easier to just share the "normal" media directory with everyone/thing on my home network and then handle permissions on the personal stuff with more granularity. It's also much easier when I know I'm looking for a photo I've taken or a document I've made that it'll be in the personal hierarchy under those categories rather than the main ones.

      It's a small change, but keeping a separation between stuff I've made and the easily replaceable stuff I've acquired has gone a long way to making my personal data and treasures more secure--both from loss and accidental sharing.

    • I have media drives that hold the bulk and they are easily organized into games/pictures/books/movies/tv/music. Smaller document/coding directories are on my C drive for source/text/spreadsheets I make myself.

      I don't tag anything. For my pictures. I simply name the directories Year_date_mainContent. (ex 2010_12_25_Xmas). Media names are self evident, but I also run XBMC for video, so I guess that has internal tagging. But still easy to find video outside of XBMC which I only use about 50% of the time.

      I alm

    • Another option is to split data over machines. Use a media server, and keep media there. Maybe pictures and personal video's can be on your normal laptop as well, but your MP3 collection, movies and tv-shows don't need to be on your laptop.

      Then I use one old desktop with Ubuntu on it as backup-machine. I use Crashplan for this. It has a free option to backup to your own machine, or to backup to a friend's machine. I backup several machines (from my parents as well) to this one machine. Then this one machine

    • by bazorg (911295)

      ... and what if your family add their own files and do not observe your directory rules? What if someone joins the household and bring their own 500GB of files with a different directory structure?

      • by timeOday (582209)
        I agree, his structure appeared to be for only 1 person. The top level of organization, really, is /home/username, with permissions set so people can't accidentally mis-file things under others' directory. I've still found that useful on my home computers even though there's no concept of enforced 'security' between users (they could all sudo to each other without a password, though they don't know it). This is because I have one desktop with a submenu for each person that launches their apps in their h
      • by Grishnakh (216268)
        It's fairly simple: they have their own directory at /home/username, and they can do whatever they want there.

        On my system, stuff which might be shared between machines (media) is in a separate top-level directory (and also on a separate RAID volume). But it's fairly simple: /media/TV, /media/music, /media/photos, etc. Anyone new would just have to conform to that, or perhaps have separate subdirs: /media/music_user1, /media/music_user2 for instance.
    • by TeknoHog (164938)

      Same here. I also use symlinks to organize music based on genre, even though all artist directories are under the music dir. This way it is also possible to file one band under multiple genres.

      Actually, pathnames are also metadata. Or at least they can be used to provide a lot of metadata, when properly used. Fancy metadata/tagging systems also need some discipline to work, and sloppy people can lose track of their data despite the fancy tools. I choose the old-fashioned way as it works for me, and is re

    • For which we say: Directories: it works, bitches. [xkcd.com]

    • Re: (Score:3, Funny)

      I find these useful, but for family stuff I can't recommend a simple hard drive crash enough. They will suddenly know where copies of everything important is, and it'll come down to only a few gigabytes....

  • by Anonymous Coward

    Fap and non-fap, and must-not-fap.

  • If hasn't been accessed in 3 years then it gets deleted [porn files 6 months].

    • by woboyle (1044168)
      Well, I have data that I want to keep, but haven't looked at in 5-10 years (or more). Just like a book on the shelf. I may not have read it in a dog's age, but I still want to keep it (when was the last time you read Little Women/Little Men?). So, we need a means to archive, but keep accessible, data that has value over time, but may not be currently relevant.
    • by skids (119237)

      Similar: if I haven't used it in a while, it gets flushed to a backup CD-R (because I still have stacks of them) and thrown in a drawer. If it's important enough to find it later, it will be important enough to slog through the backups.

      Except for code. If it's old I check to see if it is still needed. If it is, I modernize it, make sure it still works, and commit it somewhere online. Not working on anything proprietary makes that easy.

    • by hedwards (940851)

      In the past that was more of a necessity, these days at the rate that storage capacity is increases other priorities tend to arise. For instance the number of files I have is more contingent upon my ability to organize and keep organized the files as well as perform regular backups. Given that a standard DVD is somewhat less than 9gb and that you can get a 1tb disk for under a hundred dollars, that's enough for a fairly typical DVD collection right there, considering most DVDs don't use anywhere near the fu

  • by mikael_j (106439) on Sunday February 13, 2011 @01:01PM (#35192718)

    I've tried forcing myself to use various schemes including relying completely on metadata and search. The last couple of years this is how I've ended up setting things up:

    "Public" network storage

    This is for data that should be accessible to the entire network at home. NFS mounted on all my machines, stored on ZFS volume on my file server.

    • Software/Applications - Application installers and ISOs.
    • Software/Games - Game installers and ISOs.
    • Video/compressed - Download directory.
    • Video/Movies - Hard links from Video/compressed, naming set to work with Plex (for looking up movie info from imdb).
    • Video/TV Shows - Hard links from Video/compressed, similar naming as for movies.
    • Music/Rips - Music I've ripped myself, organized by artist and album name.
    • Music/Downloads/Singles - Single songs downloaded, organized by genre.
    • Music/Downloads/Albums - Whole downloaded albums, organized by genre.

    Private network storage

    I use my home directory on the file server (also on the ZFS volume) for storing personal files and mirroring home directories from client machines in ~/Backup/homes/.

    Local storage

    On individual client machines I generally try to stick with whatever the operating system tries to make me use with an rsync script that syncs everything to the file server (automatically for desktops, run manually on portable machines).

    This is what works for me. I would probably have stuck to the "just use metadata" approach if most user interfaces didn't seem to try and make it a major chore to edit and view metadata...

  • by Twinbee (767046) on Sunday February 13, 2011 @01:02PM (#35192724) Homepage
    I have recently found an incredibly fast search tool called Everything [voidtools.com]. We're talking about Google-like searching where the results pop up as you type. It must be something on the order of a fifth of a second for my 1.5 million files. This kind of technology should be widespread - it makes searches actually *pleasant* to do. Anyway thanks to Everything, I worry less now about where I store my files, and I also try to pack in keywords into the filename.

    Anyway, this kind of program is just a glimpse of what a future OS would look like. Imagine a system where everything is stored in tags and where folders become obsolete or used far less often. What you have then is a database or metadata file-system. The relatively new Haiku OS uses such a system, and I wrote about the massive advantages from this old page:
    http://www.skytopia.com/project/articles/filesystem.html [skytopia.com]

    Honestly, we'll all be better off the sooner we switch.
    • by JSG (82708)

      Everything (just looked at the homepage) looks just like "locate" (slocate, mlocate etc) which is a long standing *nix system tool. Oh and with a GUI frontend. There's plenty of those for locate as well.

      As to that sort of metadata based FS, it seems to be really hard to do properly and despite it seeming like a good idea, not many are screaming out for it. If they were we'd all have one by now.

      My money is on it being hard to do whilst not sacrificing performance. FSs are bloody hard - watch the developm

      • by foobsr (693224)
        not many are screaming out for it

        Probably since those who use a CMS do not scream?

        CC.
    • Reiser was going to go into that direction (at least if I understood the description on the namesys web sites correctly). But then, development stopped because Reiser turned out to be a real killer ...

    • by timeOday (582209) on Sunday February 13, 2011 @02:11PM (#35193132)

      Imagine a system where everything is stored in tags and where folders become obsolete or used far less often.

      It bothers me when people think tags are fundamentally different from folders (directories) in the first place. I'm going to re-introduce directories as "hierarchial tags" and blow everybody's mind.

      Maybe it's because people think of directory membership as exclusive? But it isn't. You can link a file into as many directories as you like with the 'ln' command. If that hasn't caught on, and if Windows Folders don't even really support that, it's because most people just don't bother... and the same is/will be true of tags by any other name.

      • by 0123456 (636235)

        If that hasn't caught on, and if Windows Folders don't even really support that, it's because most people just don't bother... and the same is/will be true of tags by any other name.

        Indeed. When I copy 2,000 images off my digital camera, I am not going to go through and add tags to every single one, whereas I might well cut and paste them into three directories for the places that I visited on that trip.

        Tagging all files requires a huge amount of effort that most people simply aren't going to do.

      • Thank you.

        I've been trying to figure out the best way to organize my data lately, and I've been looking into tagging systems. I have tried using symbolic links before, but didn't like the system I came up with because I was unable to move the original file. I think I've always avoided using hard links because a long time ago, someone told me they were dangerous for some reason (I know enough to get by in Linux, but I'm far from an expert on the details of how things work).

        Your post prompted me to quic
        • Hard links can be dangerous if you forget about the fact that you have multiple links. You get what appear to be multiple copies of your files, but in reality there's only one copy of the data. If you modify the contents under any of the links, they all get irrevocably changed. I've been burned by this before.

          However, that problem may not be much of an issue if you're just making a read-only archival file collection.

          • by timeOday (582209)
            You're right, but I think any tagging system is complex enough to create similar issues. Say you divorce "Joan" and decide late one night to delete all pictures tagged with "Joan." Whoops, lots of those pictures had your kids in them, too. Whereas if you put the photo in a directory for each person in the photo (using links, or plain old "copy" on filesystem with data de-duplication), the picture wouldn't be erased unless you deleted the directories for every person in it, which is kind of cool.

            My pers

      • by lennier (44736) on Sunday February 13, 2011 @06:52PM (#35194986) Homepage

        You can link a file into as many directories as you like with the 'ln' command.

        Well, sort of. You can create a hardlink or symlink in the Posix model easily enough, certainly. But the link is only one way - you can't easily find, given a file, where all its links are. So they can tend to get caught up in the bit-rot. And there's enough of a stigma around symlinking - let alone hard-linking - that very few tools can be relied on to support it in all cases.

        A true tagged or non-exclusive-directory filesystem would, I assume, have proper two-way linking between a file and and its links, so you could query a file and get a list of its tags/locations. And all the tools, without exception, would fully support it. This would include things like copying a 'folder' to removable media - you would need to standardise what it means. You can't just copy the links and you can't just turn the links into unlinked files.

          What you could do, perhaps, is store all the originals (including folders) in a single universal folder as a globally-unique identifier (it can't be just system-unique, because what if you copy a file to someone else's machine?), then make the other folders on a system contain only hardlinks, and have the file-and-folder copy algorithm copy both a subset of the originals folder and all the appropriate tag folders...

        It gets messy, is what happens, because things like disk drives fundamentally have a notion of containment (my file is either on this disk or it's not, it doesn't help if it's 'virtually somewhere out there in the cloud' once I've pulled the network plug) while tags don't. I'm sure we could solve these problems, but they need to be solved correctly and with mathematical rigor at the lowest layer of the filesystem. I don't see any serious attempts to do that in any of the tagged filesystem approaches I've seen yet.

  • by zwei2stein (782480) on Sunday February 13, 2011 @01:05PM (#35192738) Homepage

    Simple: Delete stuff.

    Do you need all those instalation files for 10 year old shareware? Do you really need Gigabytes of movies you will never watch again? Music Collection so big that your playlist is months on lenght? Irrelevant TV shows? More ebooks than you can possibly read?

    What you really need to keep are personal files - photos, home video, documents. Those can easily be managed - tag by occasion, file under year/month. done. (they do not take that much space either and people get tired of documenting everything sooner or later.).

    • by mikael_j (106439)

      I have no problem deleting stuff that's easy to find again, the problem is all that hard-to-find data. The movie you can't find a DVD or Bluray of and just finding a torrent with seeds took days of searching (not to mention downloading it at a blazing 50 kB/s from the one person seeding it). Removing data like that means that if you ever want to watch it again it will take you days or weeks of preparation.

      The problem is that data like that adds up. At first it's just a couple of movies, after a while half t

      • Well, stuff that is hard to find again is so for reason: it is not that good. or popular. those are not synonyms, but it works out like good indicator or worthiness.

        Collections grow because or your mindset - if it is "expensive" to download file, you are unlikely to part with it easily. You just put your own pricetag on file (or link with file emotionally as downloading it again is going to be painfull). Such files just gather dust, seen once and then just considered too precious to delete without taking ac

        • by mikael_j (106439)

          Well, stuff that is hard to find again is so for reason: it is not that good. or popular. those are not synonyms, but it works out like good indicator or worthiness.

          They are indeed not synonyms. Especially when it comes to things that are no longer part of popular culture it is very possible that those who hold the rights to the original material don't consider it profitable to release it on DVD or Bluray and most pirates have no interest in releasing it either. It can still be quite good though.

          Collections grow because or your mindset - if it is "expensive" to download file, you are unlikely to part with it easily. You just put your own pricetag on file (or link with file emotionally as downloading it again is going to be painfull). Such files just gather dust, seen once and then just considered too precious to delete without taking actual quality of content into account.

          While I may not watch the same movies from the 1950s over and over again on a weekly basis I do tend to watch them again every now and then and it would suck to have to spend a

    • Re:Learn to delete (Score:5, Insightful)

      by Hatta (162192) on Sunday February 13, 2011 @01:18PM (#35192824) Journal

      Do you need all those instalation files for 10 year old shareware?

      Sure do. In fact I just installed StuffIt Deluxe on an SE/30 last weekend

      Do you really need Gigabytes of movies you will never watch again? Music Collection so big that your playlist is months on lenght? Irrelevant TV shows?

      The bigger the collection, the more fun shuffle is.

      More ebooks than you can possibly read?

      You never know which one you'll need to refer to.

    • by foobsr (693224)
      Simple: Delete stuff.

      Yes, great, I have done it, but it is difficult to learn.

      Even more so if you are suspected to fall for a slogan like 'because we can', I guess.

      CC.
  • For all my editable docs I create/edit for work, I use OpenGoo/FengOffice. I can keep track of versions, and apply any tagging, etc.

    For my media files (on a separate server)I use Rhythmbox for audio and XBMC for video.

    And I back up everything somewhere else, just in case. I don't have terabytes of stuff though. Close to a terabyte.

  • Delegation (Score:2, Insightful)

    by rhendershot (46429)

    I gave my son his own computer and, like many IT strategies, told him I'd back up what he asked me to. I made him responsible for his own collection, as am I. They may duplicate but hardware is so cheap. When we watch recorded TV shows sometimes we are both interested in keeping a copy, and that's ok. A gig here or there really doesn't matter when I can add 2TB for a $100.

    That's very different from the scenario we faced when his brothers were kids. A 100MB hard drive was then pretty significant. I had

  • by chill (34294) on Sunday February 13, 2011 @01:10PM (#35192762) Journal

    My main file server, where anything not in immediate use is stored, is organized mostly for human convenience. That is, a tree-hierarchy of folders.

    media
    media/video
    media/video/movies
    media/video/tv
    media/video/shorts
    media/video/educational
    media/audio
    media/audio/music
    media/audio/drama
    media/audio/comedy
    media/audio/educational
    media/pictures
    media/pictures/family (with various subfolders like "zoo", "picnic", "christmas 2010", etc.)
    documents
    documents/work/[person's name]
    documents/school/[person's name]
    documents/misc
    web/[site name]
    programming/[person's name]/project
    family history/
    misc/

    At the end of the year, or when I do a mass data import, I spend more time getting the meta-data and tags correct than anything else. All of my audio and video are properly tagged. Ditto for any documents.

    Almost all video is accessed with "smart" programs, like Amarok or XBMC which automatically pull in things like lyrics, trailers, cover art, etc. That stuff is almost never accessed thru the directory tree. The interfaces on the programs are way too good -- assuming the stuff is properly tagged.

    The web and programming folders are basically .tar.gz files that are backed up and copied over (drag-n-drop via smb mounted share). They're archives of whatever project someone is working on their local system. I've set up cron/scheduled tasks to update those daily on everyone's PCs, even the kids.

    Most media folders are read-only, to prevent accidental deletion. My account is the master and I can upload stuff there, but I don't want accidents from people wanting to just watch a movie. 600+ DVDs/BluRays, including movies, educational & television shows all on a 2 Tb file server in h.264 format. All *music* is FLAC format, with Amarok auto-transcoding if people want to transfer to an iPod. All other audio, like drama/comedy/educational is 128 Kbps MP3 for ease of streaming. And old comedy albums aren't exactly THX-quality to begin with.

    • by dargaud (518470)

      media/pictures/family (with various subfolders like "zoo", "picnic", "christmas 2010", etc.)

      I'll expand on that. The very best way to sort personal pictures is to put them in dated directories with dated filenames, then add relevant info to the folder/filename. For instance 20100201_SkiEverest/20100201_064554_Ascent.jpg. It's sometimes hard to remember how you named a file, but it's easy to remember more or less the date (or the month if not the year). I use a script to date the images according to the exif info [gdargaud.net].

      • by chill (34294)

        Pictures I'm not overly concerned about, because I tag them properly with EXIF information. Besides date and event, I'll put in people's names and notes.

        Tags are fantastic, if you use them.

  • by icemaze (1865436) on Sunday February 13, 2011 @01:11PM (#35192772)

    Who has the time to hand-pick all the relevant tags for every file they download? Yeah, me neither.
    Finding time to put things in their own directory, and not dumping them all in "downloads", is a great accomplishment.

    However finding a meaningful, hierarchical structure is non-trivial. I'm still working on it.

    • Tags are no less useful than any other form of organization. They're more useful to me when used with software that keeps track of location for me.

      Well thought out is well thought out, regardless of whatever system you use.

    • Who has the time to hand-pick all the relevant tags for every file they download? Yeah, me neither. Finding time to put things in their own directory, and not dumping them all in "downloads", is a great accomplishment.

      However finding a meaningful, hierarchical structure is non-trivial. I'm still working on it.

      I'd settle for being able to tag a file/folder with 'temp' and have the folder/file automatically delete $x days after I last touch it.

      The reason I can't just 'rm -rf *' in my downloads directory is I don't want to delete the stuff I just downloaded a few minutes/days ago but haven't sorted properly yet.

      I support I should just write a damn script.

      • by hedwards (940851)

        The solution to that is to create a folder which is for things you've downloaded recently then move, either manually or via script, any of the ones you haven't used in the last X days to say an attic folder. If you use a file from the attic you move it back to the original folder and periodically you create a new attic and delete the old one. Which should work fine as long as you're not really picky or likely to not use something for a long period of time and then suddenly find a use for it.

        Alternatively yo

  • by xtal (49134) on Sunday February 13, 2011 @01:15PM (#35192788)

    Search, don't sort.

    • Search, don't sort.

      Good idea in theory, but in practice -- at least for me -- most files aren't going to have enough data in the filename and tags (if any) to search for pictures from Uncle Bob's second wedding. Unless, of course, you've sorted all your pics and tagged/named all the pics from that event.

      Some kind of picture/video/audio searching that would be effective seems a long way away, at least if you're looking for the aforementioned wedding as opposed to all pictures with blue flowers.

      • What's not descriptive enough about vid002034.avi? I am sure searching for the vacation or family event will turn this video up in your search results without a problem.
      • by maird (699535)

        Search, don't sort.

        ...most files aren't going to have enough data in the filename and tags (if any) to search for pictures from Uncle Bob's second wedding...

        How many dates did it take place on and how many other things did you take pictures of on the same date(s)? The date of the wedding should be at least enough to get a search result with a high density of what you're looking for.

    • by hedwards (940851)

      That works better if you're not in control of the files. From personal experience searching does precisely bupkiss for backing files up or excluding ones that you don't want backed up. It also doesn't do you much in terms of figuring out which files are duplicates and deciding which files should be deleted.

      Right now I've got several folders for the importance of the files under it to me, and I've named them based upon the duration of backups that I feel I could afford to lose without being distraught. Some

  • Where you keep all your valuable data, so if we ever hack into your computer, we know where to steal (or at least make copies of) your pron collection

  • I'm working in a Mac OS X environment, but this should work for Linux too: I have groups for the various classes of stuff, e.g. photos, household files (like taxes and Christmas letters), etc. Each group has a group home associated with it, and I mount those from my server as needed. (The server's a RAID 5 box). Irreplaceable stuff like photos are copied a couple of times, once to a disk on a separate machine and periodically to a portable USB drive that I keep at a friend's house. (I have 2 of them an

  • 1. Post it on the web, or run your own apache instance.
    2. Use google to find back your data.
    3. ?
    4. Let others also profit from your data.

    • by woboyle (1044168)
      Some savant once said "DON'T TRUST ANYONE!" with your money or your wife (or today, your data). I think that includes Google...
  • google desktop (RIP) (Score:4, Interesting)

    by meeotch (524339) on Sunday February 13, 2011 @01:18PM (#35192822) Homepage

    I had great success with Google Desktop Search (on windoze) for a while. It would index my mail, files, and web history (if instructed to) - and the best part was hitting one key to get an instant, minimalist search box with auto-preview. From there, you could jump straight to what you were looking for, or open a further page to narrow the search.

    Sadly, it doesn't work with Thunderbird 3.0, and Google doesn't appear to care, or even to be supporting it anymore. So now I'm on a hodgepodge of GDS, Windows built-in search, and the sucky T-bird search bar.

    I honestly can't believe that nobody has duplicated this Spotlight-esque functionality yet. I realize there are other desktop search options, but none of the ones I've come across have that one-key mini search that goes away as easily as it is called up. For an operation that I'm performing dozens of times daily, that's pretty crucial. It even replaced the file browser for me - much easier to call up the GDS box & type a couple letters than to grab the mouse and drill down into some directory structure - even if I know exactly where I'm going.

  • sticky labels on each floppy disk.
  • by rbrander (73222) on Sunday February 13, 2011 @01:27PM (#35192888) Homepage

    I'm pretty much a "have a lot of structured directories" guy myself; I don't see your complaint about rising file sizes, or even total number of files. They've pretty much increased linearly in number while the speed of the linux "locate" command has gone up exponentially with Moore's Law. It's the other way around from management trouble - with TB hard drives, I have so much space I leave around TV shows and other media files I'll likely never watch again, "just in case".

    At work, the search problems are harder, because I've got quite the multi-tasking job where I may spend just minutes on some problem, then be asked for an update months later, totally skeptical that I ever addressed the issue. And my favourite file-management with that is the most insane-sounding of all: one big directory. I sort it by date and rely on the fact that I take time to write out helpful file names like "downtown_condition_assessment_newmall_4_ernie.xlsx" (not actually that long, I use abbrevs in RL). Only files that have a whole lot of subject-matter friends get their own subdirectory; lonely "one-off" files go in the Big Pile.

    The "sort the directory by date" uses the theory behind "lifestreams" [wikipedia.org] promoted by Eric Freeman and David Gelernter at Yale. It really is the best thing I've found (same 30 years) to stimulate the memory - seeing the names of other things you did at the same time; you can actually sense yourself getting close to the file as you remember, "Oh yeah, I worked on that in the spring".

    An additional word of Fear & Loathing for "document management systems" like LiveLink by Formark. Required to use this by work (shared directories are strictly for 'short-term' storage), it's awful. Terribly slow, the search function approaches useless, and it's hard (and slow, did I mention slow) to even re-sort a directory (sorry, that's a 'filter down' in Livelink's vocab) by name or date or whatever. After promising that photos would be displayed with thumbnails by the great new Version 4 for two years, it came, broke some stuff that was working, and did not provide thumbnails - all media files are unsearchable in any way. I suspect for long-term archiving, putting documents in a database would have advantages, but for active business usage, it's been crippling.

  • These are what I've come up with.

    For Windows, I create C:\Software and C:\Hardware. Drivers, DirectX updates, and such all go in Hardware. Any software I install goes in Software. Games are the reason to use Windows, and are huge consumers of hard drive space, so they rate their own subdirectory, C:\Software\Game. (I've also decided to drop plurals from directory names I create. Was getting annoying having "pic", "pics", "pictures", "images", etc.) It doesn't have to be "Software", all it has to be

  • by Xian97 (714198) on Sunday February 13, 2011 @01:35PM (#35192932)
    Everything [voidtools.com] is what I use on the PC to quickly find any file I am looking for.

    On the Mac I use Spotlight.

    While it would be nice to be completely organized, these tools let me find my files anywhere they are located on my PC. I try to keep things organized into folders, but I am always falling behind so these are what I can use in the interim.
  • I don't use any tools. I just have all the content on two sets of external disks (copies of each other; I use external disks because I don't have large enough computer and I don't like the idea everything to be under current at all times). It's a pain to manage. I think Linux (or your favorite OS) desperately needs a 2-tier backup system with deduplication (but still making sure you have enough copies for recovery) and a good user interface.

    Ideally, I would say, in file manager, unarchive me this file, and

  • What happened to Beagle for Linux? It used to work pretty well for me, and now it seems to have been abandoned.
  • I put everything on the desktop. When there is no more place on the desktop, I create a subfolder named "temp" and then put everything in it, including the last "temp" subfolder.

  • when need a smart metadata filesystem. The system needs to be a simple and automatic system which file extensions and file headers are used to create the base level tags. Other tags could be added for items like music and video but the 'bread and butter' of the tag system needs to come from obvious information in the file and filename.

  • (waiting for someone to say 'CLOUD')

    Not folders, "libraries", and sure as hell no tags (I tried that w/ Picasa for my pictures a few years ago; made a mess; deleted Picasa -- returned to sensible dir structure "pictures/TOPIC/year/month" and I'm fine). And I separate code by language ("code/4th/TOPIC" or "code/c++/TOPIC" ..I even keep a /fortran dir though I haven't used it since 80s, and a /48sx— some of my favorite code even though it's essentially unusable). Homebrew backup across local network d

    • by Omestes (471991)

      Learned hard way w/ iTunes. I despise programs that rearrange your files for you, make ridiculous subdirs w/out permissions, etc. I have to use iTunes, but I look forward to the day when I can get rid of anything apple and/or adobe. Hell, not even MS forces directories on you (not incl. the OS itself, I guess).

      That behavior in iTunes is no longer the default, at least. And, last I checked, it could keep a large, unorganized, directory organized within itself, without mucking with the directory.

      Personally I love iTunes. I have over 40Gb of music in it, and the idea of manually organizing it sounds like absolute hell. I tried that once, hated every second of it, so I'm a large fan of a program that will do all the boring work for me.

      Actually MS does, or at least their sloppy design does. Its one of my peeves, I

  • I finally dealt with this problem once and for all in the following way. I found the best personal wiki out there (Zim: http://zim-wiki.org/ [zim-wiki.org]), and wrote a simple python script (http://www.inrim.it/~magni/zimDMS.htm [inrim.it]) that scans nightly my folder structure, keeping up-to-date my wiki. My wiki, therefore, is a perfect mirror of my folder structure, with the added bonuses that I can navigate to each folder, comment it, describe its content, insert images, insert links to other folders, and finally by a single c
    • by kcbnac (854015)

      Interesting...how long does it take to scan the folder structure? (If say, you wanted to run it *right now* because you'd made a bunch of changes to commit notes on; I'm curious on the performance of said system)

      I wonder because I like the simplicity of a basic file structure, but the ability to add metadata of some sort without risking the file structure...this seems to be a (relatively) simple way of doing this...

      Now only if it could be kept current real-time, without storing the files in a funny system.

  • Everything I have is on an UnRaid box, so it's organized into shares (virtual drives)

    Personal Files
    Photos (actually, it might be called Images)
    Music Archive (all my originals)
    iTunes (my working volume/set, compressed to mp4)
    Applications
    Home Movies (Raw, In Process, & Finished folders)
    Files (everything else goes here, organized by what it is - health, rockets, cooking, guitar...etc)
    Work Files
    M

  • - irreplaceable "live" files, backed up daily (about 200 megs): files I created myself, ie mails, docs, code... It's the "My Documents" folder.
    - irreplaceable archives, backed up monthly to dvd (about 4 gigs): mainly photos and home videos: "My Pictures" and "\Archives", plus my "live" files.
    - everything, backed up monthly to an external HD: mainly my painstakingly ripped CDs and DVDs, plus all of the above. "Flacs", "MP3s", "Films", "Series", "XXX"

  • That's the FOSS version of the venerable Andrew File System (Debian packages available). I use it together with MIT Kerberos V and OpenLDAP. It may not be the easiest system to set up and maintain, but what you get for your efforts -- a distributed file system -- is pretty cool even on a single server.

    The problem with NFS and Samba solutions is that the manner in which servers and hard disks are organized has too much influence on directory structure: as disks and servers are changed over time, the struc
  • by fahrbot-bot (874524) on Sunday February 13, 2011 @02:51PM (#35193362)
    Applying the Infinite Monkey Theorem [wikipedia.org] I put everything into one folder, assigning each file a pseudo-random name. Although there's only one of me, in time, I'm confident that a pattern will emerge...
  • Spotlight and fuhgeddaboutit.

    Basically I don't worry about it any longer. Spotlight let's me search the entire file system, and subsets of it like emails, and OS X is pretty good about automatically generating metadata. Good enough for most circumstances, anyway.

    And since you can search the contents of files, this makes looking for that PDF Joe Blow sent you last week dead easy.

    Honestly, I have ~/Documents and a few subfolders, and that's about it. Between Spitlight and Quicksilver I don't have to worry abo

  • Directories! Occasionally, desktop search provided by Nepomuk if I can't find it in less than two minutes or so of manual searching. Once in a blue moon, good ol' find piped to sort piped to less.
  • I ran into problems with duplicates over the years from copying files off my laptop before installing a new OS and for other reasons. I used dupmerge, which identifies the duplicate files and uses hardlinks to keep only a single copy. Freed up quite a bit of space for me.
  • Of everyone who posted their fancy choice of directory structure *nobody* told us where they keep their ~/.pr0n
  • I don't want to start an argument about which system is best for this task, but rather I have some ideas about how you could do this if you are on a Mac. I have over 20TB of storage in my home studio, with hundreds of thousands of files. Its one thing to have that much data, but its another to have it well organized. To that end, here are the apps I currently use on the Mac to organize my data

    iTunes - 2TB of media on NAS drives
    iPhoto - family and personal pics
    Aperture - handles DSLR semi pro collect
  • Max two directories deep.
    Reverse date system
    Everything gets a folder (no unfiled files)

    eg:
    Music/Pink Floyd - Dark Side of the Moon
    Photos/2008-11 Europe Vacation

  • Nemo Documents (Score:4, Interesting)

    by daserver (524964) on Sunday February 13, 2011 @05:33PM (#35194472) Homepage
    If you are on Windows you might want to give Nemo Documents [nemo-docs.com] a try. It gives a time based view and allows one to use tags. Disclaimer: author posting ;-)
  • by Bomarc (306716) on Sunday February 13, 2011 @08:40PM (#35195568) Homepage
    Having just started cleaning my house, this story comes close to my heart. Looking around, I have 6 boxes of old “documents”. What to do with them?

    First to cover the common areas:
    Video:
    I have two TIVO boxes, one is high definition, both recording constantly.
    I have one system with 8TB of storage to sort/organize the incoming TIVO recording.
    I’m setting up two 60TB servers for my “movies and TV shows”. (Each will handle 26 hard drives). I use the term “setting up” as I’ve run into some issues with these systems.

    Binary:
    I have a 2TB system set up for binary files. (This would be development, OS, drivers, patches and the like). You never know when you will need a DOS bootable disc.

    Music:
    I have one system (with 2TB storage) to handle my MP3’s. (Still need to sort/organize/remove duplicates). Currently this one also houses my image collection, important documents and the like. It is acting as kind of a catchall for everything else.

    Data:
    I’ve recently set up a system to handle “data” (document based); with 130 GB of space. I’m using “Home Document Manager” [dokix.com] . Though not mature, they are more amenable to fixing the problems.

    And now to the point: Organization.
    Overview
    The first – glaring issue is lack of a good storage house. Most management systems sort a single file in a single location, sometimes with tags. A good example of the problem that I found: what if I have a Medical Bill, which is being kept for Legal reasons, which I will need at Tax time? What if I have a MP3, Music Video and Movie that I would like to tie together (or heaven forbid multiple playlists)? Or Movie props that I’ve purchased off eBay.
    I would not like to keep the medical bill after 3 years, but for legal reason would like to keep it for seven. I don’t want to delete the “item”, but I no longer need to be reminded about the “bill”. I don’t want to have multiple copies of the same item, which makes searching a nightmare. And “tags” are a start, but are not granular enough.

    Video organization:
    Extreme Movie Manager. Ok, it has some bugs, but it does a VERY good job. With its multiple views, and multiple ways of keeping track of movies, it is the best one that I’ve seen.

    Music: Currently I’m (just) using Media Monkey and MS Media Player. Media Money has a severe limitation in that it does not handle video (read music videos-Watch "Vertical Lines" by Leather Hands to get the point). I attempted to use an “automated sorting” system, however it has significant issues, the biggest being it took MPS’s from a known group (1970’s for example), and moved them to “Unknown”, “Unknown”. Can’t use that. I also used Clone Master, and found that I have almost 2500 duplicate (MP3) files. Unfortunately, it “guesses” the wrong one most time for the likely file needing to be deleted.

    Binary is actually the most straightforward simple file structure

    Other issues:
    Video Servers: I’m also running hard drive selection into issues with the video servers. The problem is: Enterprise class SATA drives are expensive, “small” (only 2TB), fast (as such they use a lot more energy). “Green” drives are cheap and plentiful and use a lot less power (and generate a lot less heat) however they are not compatible with the RAID controllers needed.

    Video Playback: I have a decent system to handle the Blue-ray, high def requirements. However the software also has problems: In/with high def you can’t read the “default” fonts displayed

The meta-Turing test counts a thing as intelligent if it seeks to devise and apply Turing tests to objects of its own creation. -- Lew Mammel, Jr.

Working...