Which Filesystem Do You Use On Portable Media For Linux Systems?

An anonymous reader writes "Most people use MS filesystems on Disk-On-Keys, and portable hard drives, as these are readable from most machines. But this way you lose the files' permission information, which many times is very inconvenient (you must agree that having Ubuntu asking you whether to execute or display every text file or image you open from a DOK is annoying). Using 'regular' Linux filesystems like ext keeps the permissions, but may require using the superuser when switching machines (as the UIDs are different). So do any of you have a creative solution for this problem?"

HFS+

[binary paladin]

For the most part, I still use FAT32 since everything can read it. Simple as that.

However, Linux has no issue reading HFS+ and my main machine is a Mac so it does the trick too.

Mount noexec

[MBCook]

If you don't want things to look executable, mount it with the noexec option (which you could put in fstab). That way nothing on the device, even with FAT, will appear executable.

Since you are moving the files between computers, is the permissions loss really a problem? Aren't you just going to copy things off anyway?

If you need to limit access to certain users, you could use encrypted loopback file systems. But really, why not just use separate USB keys for different sets of permissions.

FAT is a lowest common denominator for a reason. If you want to interact with Windows, your only other real choice is NTFS, which isn't a bad option.

Sure you could use Ext3, or Reiser, or BTFS, or something else, but then you can't use your flash drive on any machine, thus defeating it's purpose.

I use the FAT filesystem most sticks come with

[C3ntaur]

Then, if I need to preserve Linux file settings I'll zip, tar, or cpio and store them on the stick that way.

Hardly a Linux-only problem.

[Yaztromo]

This is hardly a problem unique to Linux, although as you point out Linux does have its own special requirements that may make using FAT32 a bit problematic.

My home network is a combination of Mac OS X clients and Linux servers (Debian is so easily made so Mac friendly...). I have a USB key that I don't tend to use too often (online storage has removed much of that need), but I did decide at one point that easy interoperability between OS's was important, while at the same time needing OS-specific support from time-to-time, for specific applications and data.

My solution? I formatted my key for FAT32, and then created some disk images on the key formatted them to whatever OS-specific format was suitable (HFS+, ext3, etc.). By leaving sufficient room on the main FAT32 volume, I can readily store platform-neutral data, and inside the images I can store whatever OS-specific data (such as applications) that don't need to be accessible on every system I encounter.

This does require an extra mounting steps. In OS X, it entails plugging in the key, and then double-clicking on the DMG file to mount it. In Linux, I have to mount the ext3 image using the loop pseudo-device. Of course, this is only necessary if attempting to access data in one of the OS-specific formatted images: accessing shared data merely requires mounting the key itself (generally automatically handled by the OS).

It's hardly perfect, but it does mean you can have one key that can have both shared and OS-specific data on it for as many OS's as you'd like to have at your disposal.

Yaz.

ntfs-3g for mac

[Anonymous Coward]

Sure it does.

http://macntfs-3g.blogspot.com/

Explore2fs

[Pow.R Toc.H]

If you don't want much hassle, just use explore2fs (http://www.chrysocome.net/explore2fs [chrysocome.net]). It's an userland application which does not install any drivers or the like on Windows, and as such will execute as any user. However, you'll still need to transport it to the machines you'll want to use, and thus you'll need a tiny space of FAT32 for doing the trick.

If you have rights to install drivers on the windows machines you use, you can try the EXT2 driver available on www.fs-driver.org [fs-driver.org]. It will mount your EXT2/EXT3 volume as a drive letter so you can transfer files between partitions.

Re:Well, now ...

[causality]

So do any of you have a creative solution for this problem?

Isn't the whole point of this "problem" that there shouldn't be a solution to the problem?

That was my thoughts exactly, I don't think the guy understands how security works. If you remove the qualifications to access a file to perseve only -- say -- the need for user name to match, then what the hell kind of nonsense security is that?

This may be a case where the physical security (possession of the portable media) is much more important than the filesystem permissions. Generally speaking, the portable media itself is a storage-only device and does not have the mechanisms in place to enforce file permissions, relying entirely on the machine to which it is connected for such tasks. Therefore, if you are not using encryption, then you should always assume that anyone with physical control of the media is going to be able to obtain the files on it. If nothing else, they can plug it into a machine that they own/control and copy the files as the superuser. If you are using encryption, then the filesystem permissions should be moot when talking about portable media.

Re:tarballs

[josath]

Maybe if he used z instead of j...bzip2 is a notoriously slow compression algo.

Forget FAT/VFAT, make your own.

[strredwolf]

First of all, FAT is patent encumbered and Microsoft's willing to go to court to protect it; so that's out. That includes the old UMSDOS file system Linux had at one time.

Someone needs to make a good file system that matches FAT, but is more extensible. A good choice is ext2 now... if we dropped a few things that wouldn't work nicely. Like device nodes, pipes, and Unix sockets. Like ownership, since it's assumed that the person mounting the system would own the files on it, along with groups. Simply access restrictions; they wouldn't apply.

This will simplify the structure a bit, which is a nice bonus and could let it be put on floppies. In other words, it's a light, anonymous, extended file system. LAEFS.

I got an .h file. Anyone want to help develop it as a FUSE driver?

Re:ext3

[Kwiik]

Of course each oft the systems have to have their own passwd file, wtf.. unless you're authenticating to ldap or another service.

The solution is to have the folder you are mounting to in fstab, and tell it to always mount media as a specific UID/Group

Also, the "display or execute" message isn't a problem with the OS, it's a file manager issue. I'm guessing if it annoys you that much, then somebody would have by now submitted it as a bug/idea, and the project admins could decide whether to make it not display that message & just do what's right, or to put an option in somewhere to disable the message. It shouldn't have to ask since it can look up the file extension and then pass it through the binary wrappers

Re:NTFS

[Nimey]

Mods: not funny. I've formatted large USB sticks as NTFS before. Works fine for r/w on Linux and Windows, not so much for Macs, because OSX doesn't have native write support for NTFS.

Installable File System

[Jahava]

I had the exact same problem a while back. My solution was a little less straightforward than some, but is still simple enough. Basically, I leverage the freeware software Ext2 IFS [fs-driver.org], which installs software onto Windows that allows it to recognize the contents of Ext2/3 partitions.

Basically, I have my disk formatted with two partitions:

• A 1GB FAT32 Partition
• The rest as an Ext3 Partition

On the FAT32 partition, I place the latest version of Ext2 IFS. When I access the system on my main Linux box, I just mount / use the Ext3 partition.

When I visit friends or family and I plug it into their Windows box for the first time, Windows recognizes the FAT32 partition, so I can install the Ext2 IFS software that I put onto that partition. From then on (and every subsequent access), Windows automatically mounts it!

Windows doesn't reflect the Ext3 permissions, but if you have physical, portable access to an unencrypted hard drive, those mean nothing anyway. And, of course, make sure to ask friends and family before installing filesystem drivers :)

Re:ext3

[PopeRatzo]

Which Filesystem Do You Use On Portable Media For Linux Systems?

The best answer is: "It doesn't matter."

I use whatever file-system happens to be on the portable media I happen to be using. I move data from Windows to Macs to Linux machines all day long and I never, ever have to think about which filesystem is on the portable media.

This is a good thing.

Re:UID's

[imroy]

It would be nice if the default was to pick a random arbitrary and large UID so the chance of UID clashes would be remote.

You know what would be great? If someone made a daemon for mapping UID's between machines [die.net]. That'd be fantastic, but I'm sure no one else has thought of such a thing.

Re:Mount noexec

[massysett]

If you don't want things to look executable, mount it with the noexec option (which you could put in fstab). That way nothing on the device, even with FAT, will appear executable.

I just tested this out, and it does not work. mount(8) says that mounting something noexec means that execution of files on the filesystem is not allowed. However, that does not necessarily mean that the execute bit will be unset. Instead, on my system the execute permission bit is set, but attempting to execute something gives me "permission denied."

I've had the problem the questioner asks about before; the cleanest solution is to use "fmask=133" in the mount options, as described in mount(8). (There are also umask and dmask options, but I find fmask=133 is sufficient. umask=133 will screw you up because then you can't enter directories.)

This solution works well for me because I just mount everything from the command line and using the settings in fstab. (This actually is not as bad as it sounds, although I do most work in the command line shell anyway.) Setting the fmask for GNOME in Ubuntu likely would involve editing some HAL XML file somewhere in order to say what fmask options you want, though I'm surprised Ubuntu hasn't done this already.

Before I figured out how to use the fmask option, I had a little script to chmod the files on my photo memory cards as I copied them to my computer.

Re:easy..

[Drinking Bleach]

It was called UMSDOS, but it hadn't been maintained for years and was eventually removed from the kernel around 2004.

Re:What about UDF?

[imroy]

I experimented with UDF a couple of years ago. As always, Windows is the problem. No matter what I did, Windows did not see the thumbdrive as a drive letter. And Google didn't show up any useful pages either.

Re:ext3

[timeOday]

I was thinking, perhaps incorrectly, the poster has a bootable linux on USB, and he wants to boot up different computers with it, and access his home directory on those computers, but is having trouble because his files have a different UID on each computer. The idea was the bootable USB linux could adapt to different disks by having a different passwd file for each, which his username mapping to the appropriate UID on each.

OK, that was probably too much to assume, but I have been facing this particular problem a lot lately with accessing the host filesystem on VMs that I move around.

the obvious solution

[Anonymous Coward]

I am unsure what is meant by "disk on key".

If you mean a simple storage area for copying files to and off-of your USB key... then:
    -> YES fat32 is best.

But if you mean that you have installed linux onto your key and boot from there... then I have a lot of experience with this.

Solution: create at least 2 partitions on your bootable USB key. One filesystem should be 'ext2' and contain your root linux partition. The other should be formatted to vfat. The fat partition is then accessible to both Win and Linux. So just copy files onto that partition from whatever O/S you are currently running.

Note: always use 'ext2' on USB or SD cards as there are less writes to the disk. Since portable media reports to have a limited number of writes. Also, do not create a swap partition if you run your primary linux O/S from the portable media.

Re:How do you deal with Linux destroying file date

[Rick17JJ]

When I use the rsync command, none of the dates or permissions is altered. I use rsync from the command line, about once a week to back up my files onto an external USB hard drive. There is also at least one point-and-click GUI type front end for rsync, which I have not yet tried.

I used GParted to reformat my external hard drive as an EXT3 partition. GParted is an easy to use, free point-and-click GUI type front end for free Parted partitioning program. I have also used GParted to reformat several small USB keys as EXT3. GParted will can also create other types of partitions, such as FAT16, FAT32, NTFS, EXT2, EXT3, JFS, XFS, Reiserfs, and Reiser4 partitions.

If I were using the tar command to bundle my directories and files up into a tar ball or a compressed tar ball, the permissions and dates and everything would have been preserved inside the tar ball. In that case I could have stored the tar ball on a FAT32 partition, without loosing permissions or the correct dates. But instead of doing that, I have been using the rsync command to create a backup copy of all the directories and files and everything onto the external USB hard drive.

But... there IS a solution!

[mcrbids]

It's like the OP asks for a solution to a problem when using Linux, and then there's a grand cacophony of people denying that there is a problem! (when there is, why else was it presented?)

There are two solutions that I use:

1) Use tar to zip up the files with attributes, Then copy the .tar or .tgz file to the USB drive. Both Winzip and Archiver read tar files, so they can still be read on Win/Mac. Disadvantage? You can't easily save changes...

2) create a loopback file on the USB drive, format Ext3, then mount when it's needed. You get easy writes on Linux, but you can't read on Win/Mac w/o special drivers.

English translation

[Anonymous Coward]

However, Linux has no issue reading HFS+ and my main machine is a Mac so it does the trick too.

"No issue reading" is Linuxese for "For write support, you must format specifically as Mac OS Extended, Case Sensitive, No Journal, and one of Mac|Linux formats in a way that the other can't read at all, have fun figuring that out."

No joke.

Re:NTFS

[novakreo]

NTFS-3G [blogspot.com] works just fine on Mac OS X with MacFUSE [google.com].

Can't format MS FS with non-admin in Mac OS X 10.5

[antdude]

Has anyone notice TrueCrypt v6.2a won't let you format MS FS (for a container) in a non-admin account? It only works with full administrator account. That doesn't makes sense!

Re:ntfs-3g for mac

[slimjim8094]

You need to mount it async; I don't know if there's an alternate driver for it (or just a mount option) but it speeds up transfers by about 30x

Re:ext3

[H3g3m0n]

One worrying thing is the new exFat filesystem in use in newer version of Windows. You can be sure after the patent lawsuites suit's on normal Fat that there is patented stuff in exFat. Unfortunately a a new FS is going to be required soon as flash drives start to hit the larger sizes. With every desktop running Windows and most of the population not even knowing what a filesystem is, everyone will end up using it. It's also the standard for SDXC.

This will of course leave Linux users screwed and every one ever who wants to make a windows compatible device like an mp3 player will need too pay a MS tax.

It's unfortunate that we couldn't switch to some open system. I believe UDF is fairly open and is the BluRay standard so should work on many systems. Anyone know if UDF support writing under Windows Vista/7? Can it even be read off flash media?

About the only thing I can say about the filesystem is that it won't be long befoure USB drives go the way of floppy drives as everything moves online.

Looks like Windows 7/Vista do support writing: http://en.wikipedia.org/wiki/Universal_Disk_Format#Native_OS_support [wikipedia.org] so it might be the best FS to choose from in future for compatibility when you don't need to use older XP era systems as latest Linux and OSX both seem to have write support. The only thing is I can't find any information about patents or license fees so I don't know if its an open standard or not. Still won't help with every digital camera manufacturer, SDXC and mp3 player switching to the MS system.

Re:ext3

[H3g3m0n]

Most newer OSes seem to support UDF writing, so hopfully it will work in future: http://en.wikipedia.org/wiki/Universal_Disk_Format#Native_OS_support [wikipedia.org]

Reiserfs has faster reads

[Anonymous Coward]

Reiser is a write slow read fast file system.
Ballanced trees organize the data. This is really what you want on flash media.
The tail packing will compact the data into the filesystem so it's efficient.

You gain more space available than you'd otherwise get from the other offerings.

There is a driver which fits within the Windows file-system driver model.
Not sure that it works.

I've used Reiser for at least eight years. No major issue were experienced during this time.

Dataloss and corruption is something I've only read about. I've never actually experienced it
with this filesystem.

Ext3 randomly lost it's superblock on my external drive a few days ago and I was unable to recover any of the backup blocks. There was an error message about being unable to set a superblock. I researched this extensively on google over vast numbers of forums ( 5 ). Only I found there was no answer other than a suggestion to backup the data if it's valuable.

I'd never recommend Ext3 for anything. Hans Reiser was correct in freezing developement on Reiserfs3.
I think the only modifications that were made to Reiser3 were relating to SELINUX sometime back. I am a little skeptical of this but I've had no real stablilty issues. I don't use SELINUX.

If you can get by with it use Reiser but if you need exchablility then go with Fat32.

I'm not sure how to deal with user-id's.

Re:ext3

[Rick17JJ]

When installing Linux on more than one computer at home, I always make sure that I have the same UID number, on each computer. That way I avoid having permission problems, when accessing my files from an EXT3 formatted external hard drive, using one of my other Linux computers.

On all three of my computers, my user name is Rick and my UID number is 1000. With most recent versions of Debian or Ubuntu, the first user created is assigned the UID number of 1000. So in most cases, I would get the same user ID by default. But if that was not the case, there are are ways to assign myself a specific UID number or to change my UID number.

As someone pointed out in another comment elsewhere, the command below could be used to change a user's SUID number. I modified his example to use my user name.

usermod -u 1000 Rick

When assigning a new user from the command line, I have used something like this to create my user name from scratch, which the UID of 1000:

useradd -u 1000 Rick

Using the chown command, with the correct extra parameters, could be used to recursively change the ownership of all files, within a user's home directory, if that should be necessary after the change.

I do have an external hard drive and several USB keys formatted as EXT3. Because I have the same UID number on each Linux computer, I do not have any permission problems when accessing the files.

I can look inside my passwd file to see what my UID is by typing this:

cat /etc/passwd

Re:NTFS

[Hal_Porter]

True that. This free tool can format hard disks bigger than 32GB as FAT32, upto 2TB in fact

http://www.ridgecrop.demon.co.uk/guiformat.htm [demon.co.uk]

There's also a command line version

http://www.ridgecrop.demon.co.uk/fat32format.htm [demon.co.uk]

There a 4GB per file limit, which is a fundamental limit of FAT32. The 32GB per volume limit is just Microsoft's way of encouraging people to use NTFS.

Don't be retarded, there are no alternatives

[Colin Smith]

Look up the network effect.

Any filesystem which doesn't have universal availability is useless for this purpose. FAT works everywhere.

Re:FAT32 out the window

[sardaukar_siet]

The file size limit in FAT32 volumes is 4GB (http://en.wikipedia.org/wiki/File_Allocation_Table#FAT32) and not 6GB.

Re:ext3

[MikeBabcock]

Oh man, you mean the accounting database system we maintain on Linux for our customers doesn't actually exist? I hate parallel realities.

Re:tarballs

[schmiddy]

Solution: pbzip2 [compression.ca]. With 4+ cores it should be comparable to gzip.

Re:Poratibility

[Simetrical]

The Linux NTFS drivers are working well now. That's what I use on my shared partition.

Except if the portable drive is pulled out of the machine without being properly unmounted. Then the filesystem is unclean, and the Linux NTFS driver doesn't know how to replay the journal. So the thing becomes unusable until you stick it into a Windows machine and then remove it properly. Of course, you shouldn't be pulling it out without unmounting . . . that can cause serious data loss on some cheap USB drives, apparently, regardless of journaling.