How Do You Organize Your Data?

kpellegr asks: "After returning from a well deserved holiday, I was faced with an exploding inbox. While organizing and deleting my mail, I realised I was having trouble classifying each mail into one specific folder. I had the feeling I should be able to link to one email from several folders (e.g. product information should be linked to from the 'vendor' folder, as well as from a specific project folder where this product is used). The more I thought about this, the more I realised that trees (such as the Windows filesystems) are not really ideally suited for organizing data. On UNIX-like filesystems, symbolic links allow the creation of simple graphs for organising data, but I have the feeling data could be organized more efficiently. How does the Slashdot crowd organize their data? How do you manage files, email, contacts, meetings and all the relationships that might exist between them?"

Flat ASCII files ...

[IchBinEinPenguin]

... and grep

;-)

three easy folders

[MadLibs]

Inbox.
Deleted.
Sent.

the "find" function is a godsend.

Flat files and full-text search

[gvc]

I have all the email I've ever received stored chronologically in flat files. I use full-text search and navigation tools to locate what I need.
I use much the same technique for organizing the papers in my office.

In general you can spend effort imposing some organizational schema on your data, hoping that your organization will enable you to find information later. Or you can leave the data as it lies, and spend the effort at retrieval time, once you know what you're looking for.

Current tools, particularly those in Windows, aren't particulary amenable for this purpose, but they're getting better. For example, you can download a seearch engine and index your hard drive much like the web.

Even primitive tools like grep work pretty well for a few hundred megabytes of mail.

Until OS X...

[Tokerat]

...organizing data was quite simple for Mac users. (All you Mac people out there have to admit: You're right with me on this. Don't lie!)

The process was simple:

1. Save everything to the Desktop.
2. When you couldn't see the background pattern anymore, create a new folder called "Desktop crap" or something, and move all the files into it.
3. Move the folder on to the hard drive.
4. Repeat.

:-D

Re:Virtual Folders

[Spunk]

And when you delete a message from one folder, it's deleted from all of them!

If you're aware of the virtual folder concept, this can be very powerful. But is a first-time user going to expect it? Of course not, he thinks the folders work like everywhere else, and copy means make a copy, not just a link. So many emails were lost at the last job where I used it, for this reason.

Obligatory BeOS Reference

[sethadam1]

BeOS used file attributes and file system queries to organize data. Longhorn's WinFS is built on this concept. The real question isn't how to organize your files, it's why does your data need to be in files? Why are folders so closely entwined with our computing experience? This type of grouping is best suited for your clothes in your dresser. In real life, tossing everything into a pool and pulling out what you need by characteristics ("attributes") is much more useful.

Making Tree-view work

[Artifakt]

IMHO, tree structures work best for people who don't mind branching structures that sound illogical, just so every branch has at least a few files in the end folders. For example, if you have e-mail relating to 10 programs you use, and you are comfortable lumping 8 or so filed e-mail docs in "freeware" because that's all you have there, and then not making folders for "shareware" and "commercial software", but organizing those by company because you have an average of five files per company there, then tree view can work for you. If you 'have' to make folders by company for each company once you start with that method, and a lot of those folders end up containing only 1 or 2 documents each, you aren't really gaining productivity by being more 'logical'. It looks neat and organized, but you end up with a number of folders = or > than the files, and so it requires as much mental effort as looking at a single list of files, unsorted. Possibly you could set a goal, like my directory structure should look only about 1/4 as complex as the raw filenames all viewed together, or each folder should have at least 3 items in it, but no more than 12.

Re:OmniGraffle, OmniOutliner, and more

[Dr. Sp0ng]

As for software, I use OmniGraffle and OmniOutliner from OmniGroup. OmniOutliner is especially simple, yet unique. I wonder why no one else has an idea organizer that is so incredible? I couldn't do my job without it.

I'll second this. I don't actually shell out for software very often, but I paid for OmniOutliner. It's a terrific program. I'd pay for OmniGraffle, too, but I'm on a tight budget these days and can't justify spending the $69.95 they want for it. That's not to say it's not worth it, just that I can't afford it at the moment.

I wish there were more companies like OmniGroup out there. They write some good stuff. Now if they'd just add tabs to OmniWeb!

Re:Inefficiently

[stephens_domain]

Magic prioritization:
Step 1: Reply to the email ASAP with a question. Emails are (at best) only half thought out to begin with, so this is typically necessary anyway. It is best if it is something the person will have to look up or follow up on, rather than something they will know right away.
Step 2: Delete the email.

If it takes the person two weeks to get back to you, you know that it is not important AND you just bought yourself two weeks.
If your phone rings 30 seconds after sending the email, it is urgent.

Everything else falls in the middle somewhere, but you get the idea. In my case, probably close to 5% of these never get a response (or get a quick reply that they will look into it, but no final answer), or having been forced to think about their request, they send a response that they need to work on the details of the request before I begin working on it.

Re:Get an integraded enviroment

[nolife]

You can take that a step further and use IMAP with fetchmail and procmail. Set it up once and fine tune as needed and have one set of folders and filtering available to any IMAP client.
The backup is easy if working with standard mailbox style folders because the format is text, readable by any viewer. You can tgz your mail directory to a file via cron. I back mine up on a rotating basis to a different drive. For things I know I will never need but want to keep anyway or for archiving important things, I create a new IMAP folders with my client, move the messages over to that folder, tgz it and move it out of the mail directory. If I ever need it, I can extract it back to the mail directory and view it again or I can add more mail to the archive file later with a few commands I am not familiar with the native format of any mail clients anymore because I have been using IMAP for years. I switched for two reasons, I got tired of always trying to convert proprietary mail formats everytime I wanted to change mail clients and I wanted access to all of my mail regardless of what type of machine or where I was coming from. I will never go back no non-IMAP. The fetchmail and procmail functionality are an added bonus. You get the most from IMAP when it is running 24/7 on a stable machine somewhere on your local/home network. If you don't have such a thing already in place, it might not be worth the initial effort.

MSWindows Inconsistencies

[solprovider]

I was not really asking for help. I am in the habit of ALWAYS right-button dragging when using MSWindows. That way I can choose what I want it to do regardless of the situation-sensitive default. (I thought it was obvious from the post that I understand how it works.)

The problem with the using CTRL and/or SHIFT + Drag is that it takes two hands. Files are not heavy. I like that I can move them with one hand.

The qualification for when it creates a Shortcut is the same as everything else in MSWindows: what is the file extension? Executable files (.EXE and .COM) create Shortcuts when dragged. Apparently MS does not consider BAT files to be executable. All other files (including .BAT and .DLL) are moved if on the same drive.

Checked in Win98: Dragging SPOOL.EXE creates a Shortcut. Has that changed for WinXP, or is the spooler no longer a .EXE?

My point was that the parent to my post suggested that Lotus Notes was not following the standard set by MS, and I was suggesting that MS had no standard.

---
The moderators are having FUN! My post above is currently:
+ 2 Insightful
+ 2 Informative
-3 Troll
For a total of:
+ 1 Troll

I guess some Slashdotters REALLY do not like Lotus Notes, or having it suggested that Notes follows the Unix ideas for files.

Re:Easy

[nxs212]

Outlook 2000/2002 is excellent, even though it does crash once in a while. My biggest gripe with it is that it doesn't explain to the end user what exactly a Personal Folder is and why keeping it on a laptop's hard drive is very bad. Most of our laptop users don't do backups and we had a guy almost in tears because his ibm stinkpad HD failed due to "sticktion" and he came close to loosing 2 years of work. Just because you can organize and keep all your files inside Outlook, it doesn't mean you should. Use a normal file server, that gets backed-up nightly, to store your files and tell your users to do the same.
Also, watch those suckers blow once they reach that magical 2GB limit! You'll need to run a repair program just to get back your data.
Bill Gates strikes again! (Bill once pondered -who's gonna need more than 640kb of ram?:) 20 years later - who's gonna have a PST (Personal Folder) file bigger than 2GB? Oops!)

Information retrieval and human factors

[majid]

Part of the reason this problem is so hard is that it has been approached mostly from a technological perspective, rather than finding out how humans think and organizing the system around that.

There is a significant body of knowledge around this subject that was developed by librarians. See this article [firstmonday.org] for an introduction.

Another example: Jef Raskin's Canon Cat information appliance eschewed files completely. You located a document by typing words that are in it, in efect making the whole document its own filename.

The approach I find most powerful is set-oriented. I use an app called IMatch [photools.com] to manage my digital photos. Its sophisticated set-oriented category system [majid.info] makes it very easy to locate an image. That is what Microsoft is attempting with Longhorn's unified data store, or in more forward-looking projects like MyLifeBits [microsoft.com].

Re:I put everything in one folder

[Anonymous Coward]

kpellegr you not making sense. The "Windows filesystem" is an hierarchic filesystem modeled on what Unix used/uses [but did not invent].

You say:
"On UNIX-like filesystems, symbolic links allow the creation of simple graphs for organising data".

WTF are "simple graphs" in the context of a filesystem?

This is not an OS based issue. This is an application/individual solution based issue. Neither Windows nor Unix addresses the organizational issues you raise.

Please be more specific in your arguements.
[Same goes for any moderators who allowed this tripe to be posted without an editorial comment in the first place.]

Symbolic links in Unix are almost exactly like "shortcuts" in Windows.

I am by no means a Windows supporter but your statements don't make sense.

Re:[obvious]

[jgomez1]

This is slashdot; what do you expect?

The Spanish Inquisition??

Re:Archaeological Filing system

[Anonymous Coward]

It takes just as long no matter how you sort it.

Only on the host machine. If you're using a shell, piping through head or tail reduces the amount of data printed across your network/dial-up connection, which can obviously save time.

How do I sort my data? Loosely.

[AutumnLeaf]

The more I try to strongly 'type' my data, the longer it takes to deal with it. Big general buckets work the best for me.

I don't always succeed at that, but I do try. Sometimes I don't produce the same neural network or mneumonic-map that I did two years ago for the same datum, and then it gets lost. So the more general, the better.

An aging user interface metaphor

[mst]

That is 3 different results from the same user action! So how do folders work everywhere else?

Well, IMO the real problem is not whether one maufacturer or another has his own user interface rules, it is the fact that folders and documents were introduced as the universal metaphor for arranging data on a computer in the first place.

And now we are stuck with the restrictions imposed by that representation, which will often lure first-time users into believing that just because it looks like real-life a folder it will behave like a real-life folder. No matter how you then try to squeeze the concept of links, views, etc, into some kind of association with this rather limited concept, you are likely run into problems. What, really would be the real-world counterpart of a symbolic link, a virtual forlder (!), a view, etc?

The file/folder metaphor comes from an age where files were few and far between for the average user. Maybe we need a completely new user interface concept to deal with today's overload of data.

I've found two different mechanisms.

[Slartibartfast]

First and foremost, leave everything in your inbox.
"But the point of this excercise is to _organize_ my ifnormation!"
Well, yes. Which brings me two the second mechanism(s): use Evolution's v-folders. I really wish that more clients supported v-folders, because they the ideal metaphor for e-mail soft links. So, now you're stuff's "organized" in one folder, and many sub-folders. Why is the "one folder" bit important? Because -- and here's the nifty part -- you can now grep/Perl/regex the hell out of it with a fair bit of facility.

$.02

Re:I put everything in one folder

[gfxguy]

The problem is when you try a hierarchy. Let's say you have a blonds folder, and a brunettes folder, but then you get images of a blond and a brunette together.

Sure, you say "Put it in the lesbian folder", but what if it's also B&D? Do you put it in the B&D folder, or the lesbian folder?

These are important questions. Ideally you could have one file that symbolicly goes into all four folders (but doesn't take up all that space). Then, regardless of what you're in the mood for, you can find it.

hm

[ed.han]

you seem to be suggesting that we "metatag" documents/files. this is actually a great idea, if you think about it.

unfortunately, the problems associated w/ creating such a system effectively and then diligently assigning the appropriate values to each and every document/file becomes prohibitive.

f'rex: let's take the example of pr0n. you could metatag based on area(s) of interest: e.g., b&d, lesbian, groups, etc. assigning the proper values would allow you to search for the ahem desired file.

however, if you want to implement such a system on an existing datastore, you've got a pretty daunting task ahead of you.

theoretically, something in the OS or search tool you're using could offer the option to assign the appropriate tags.

but then you run into some problems: what if something isn't [x] enough to be [x], but contains those elements? is minor [x] enough to get classified that way? do you need an integer value or something to describe just how [x] something is?

and this would hardly be universal: one person's art is another person's pr0n.

ed