Updated: 15 July 2014.
Extended file attributes are name/value textual pairs which can be associated with files, to describe them beyond what the standard filesystem attributes provide.
Most Linux and BSD modern file systems have had this capability for years. So does Mac OS X. Apart from minor interface differences, the feature works identically on all three systems.
The extended attributes can be very valuable for storing file metadata (e.g. 'author="John Smith"', 'subject="country landscape"'), in the many cases where you do not want or can’t store this data in the file internal properties.
Especially, a tagging system using extended attributes as storage for the tags would have interesting properties.
However, at the moment, very few applications use file extended attributes, if any. 'Beagle' used to store data in there. According to this freedesktop page, it seems that 'Rox' and 'Apache' currently use them. Also 'Dropbox' (see the comments section).
The most likely explanation for this slow adoption is the still slightly reluctant support for extended attributes found in Linux distributions and general filesystem utilities, which have a lot of lag with actual filesystem capabilities.
Using extended attributes is only viable as long as they have a good chance of being preserved by common filesystem operation and tools. Metadata is precious, you don’t want to lose it by using the wrong tool.
When I first wrote this document, the most glaring issue was the lack of a universally available (cross-platform) file archiver format which would preserve extended attributes. I could not find any direct way to create an archive on a given type of system wich would be extractable on the two others with their standard tools (e.g. create on Linux, extract on FreeBSD and Mac OS X).
However, things changed since, and both GNU and BSD recent versions of 'tar' have support for storing extended attributes. They use incompatible formats, but they are both easily available on the 'other' system, so that this is not a major show-stopper any more. There is still the minor inconvenient that both utilities require non-default arguments to store or extract extended attributes, which is another indication that these are still somehow considered as second-class data.
The lack of NFS support is a big issue.
Here follows my small inventory of what works, and what does not. It uses the xattr abbreviation for extended attributes.
If you notice an error, or have information that could improve this page, please email me at jf at dockes.org ! Or use the comments section at the bottom.
The good
The freedesktop site has an embryonic standard about defining and sharing xattrs.
System configuration
On Linux, FreeBSD, and Mac OS X, xattrs are enabled in the default kernel.
On Linux, depending on the distribution, you may need a special mount option ('user_xattr') to enable them for a given file system, e.g.:
/dev/somedevice / ext4 errors=remount-ro,user_xattr 0 1
This seems to be needed on Ubuntu, but not Debian or Fedora (where it won’t hurt anyway).
Allan Daemon added a useful note to the Disqus comments, about using extended attributes with NTFS filesystems on Linux. I’ll cite it here in case you don’t see the comments section:
I began to use xattrs on Linux ( Ubuntu 12.04.5 Precise) to tag some files. Great part of them are in a partition to store data intended to be compatible with majors OSs, so It’s formated as NTFS. And xattrs are working greatly in all my file systems ( ext4, NTFS). In NTFS it uses ADS - Alternative Data Streams to store user.* xattrs. In NTFS you should be careful to mount it: use the option user_xattr or streams_interface=xattr (both are the same) This seems to be the default. Using streams_interface=windows allows you access it as usual windows data streams, ie, opening the file filename$attrname to read/write user.attrname of filename.
Specific command line utilities
Linux has the 'getfattr' and 'setfattr' utilities to retrieve and set xattrs. On Debian, you need to install the 'attr' package to get them.
FreeBSD has 'getextattr', 'lsextattr', 'rmextattr', 'setextattr'
Mac OS X has 'xattr'.
The pxattr package (disclosure: I wrote it), which has both an API and a command line utility, manages user attributes in a uniform and cross-platform way. It includes an attributes archive/restore facility.
Common commands
Common file system commands manage xattrs almost properly.
-
'mv' preserves xattrs, with no need for specific options, even when crossing mount points. Both filesystems must support xattrs in the latter case, of course. Quite unfortunately, GNU 'mv' does not warn you when the target filesystem does not support xattrs (e.g.: an NFS mount) and silently loses your data. A bug report was created for this issue, but as far as I can see nothing has been done about it at this point (2014-07). If you feel that this behaviour is wrong, please make your opinion known by adding to the bug report.
-
The GNU version of 'cp' (default on Linux) will only preserve xattrs if given the option '--preserve=xattr', also implied by the '-a' option. 'cp' will not duplicate attributes by default, which is a reasonable choice, even if it will be wrong in some situations (where you will have to use the options). See the Ugly section for FreeBSD cp…
-
On Mac OS X, 'cp' preserves xattrs by default.
Archiving and backup
'rsync' 3.0.9 with option '-X' or '--xattrs' will preserve attributes when transferring between Linux, FreeBSD, and Mac OS X.
FreeBSD 10.x 'tar' records extended attributes by default. You need to use option '-p' to restore them while extracting.
Recent versions of GNU tar include support for xattrs archiving. The '--xattrs' option must be used both when creating the archive and when extracting (tested 2014-07 on Ubuntu 14.04).
In any case and quite unfortunately, libarchive and the GNU version of 'tar' have chosen different implementations for xattrs storage, and they are not compatible. The port of GNU tar available on FreeBSD does not include extended attributes support, but I guess that implementing it might not be too difficult.
MAC OS X 'tar' uses still another approach to xattrs storage. Recent versions appear to be based on 'libarchive', with extensions to support the specific MAC OS X resource files (reference).
rdiff-backup supports xattrs on Linux and Mac OS X.
'fsarchiver' is supposed to preserve xattrs, but it only runs on Linux.
Alternatives or complements to using the archivers:
-
Use an extended attributes command line utilities to create an attributes archive and restore the attributes after extracting the tree. The command line tools available on each system would make this reasonably simple, but some minor scripting would still be needed. This works flawlessly between Linux systems using the 'gefattr'/'setfattr' tools from the 'attr' package. You can also use the pxattr command line utility to do the same thing in a platform-independant way.
-
Use a system specific tool and extract the archive on the same type of system it was created on, and 'rsync' the result to the other system.
GUIS
'eiciel' is a nice small GUI utility to manage (display and edit) a file’s xattrs. It is available on Linux and FreeBSD. In addition, installing it will add an 'Extended Attributes' pane to the 'Nautilus' properties. Unfortunately, both the standalone GUI and the panel lack possibilities to manage several files at a time.
Other
The Recoll full text search utility has full support for extended attributes.
The not so good
One fact which could find a place in the not so good or the really bad section, depending on your intended usage of extended attributes, is that the maximum size constraints are very different between different systems and prevent full data portability (for example when mirroring file systems). See this comment by Giovanni Panozzo for more details.
Archiving and backup
'rsync' 3.0.9 with option '-X' or '--xattrs' will preserve attributes when transferring between Linux, FreeBSD, and Mac OS X, but it will not detect a change in the attributes when performing an update later on. Thanks to Mikkle for pointing this out. This issue appears to be corrected in rsync 3.1. Checked on Linux rsync 3.1.1 (by me) and FreeBSD (see the comments section).
The 'dar' archiving tool supports xattrs, but the command line is a bit strange for the casual user (its real purpose is to be used by sysadmins for backups). However, the following works for me and does preserve the xattrs:
dar -R /path/to/directory --create archivename
Will create an archive file named archivename.1.dar, which you can then restore into any directory, using:
dar -x /path/to/archive
Dar does not seem to support xattrs on FreeBSD though, and only the latest version (not yet in macports) supports xattrs on Mac OS X, so the solution is not fully cross-platform.
File managers
'Thunar' and 'Nautilus' seem to mostly preserve attributes, but won’t display them.
|
Note
|
I am told (Disqus comments) that Nautilus only works with textual attributes. It will truncate binary data at the first null byte. This should promote it to the 'ugly' section actually. |
You can get 'nautilus' to display an extended attributes panel inside the 'properties' dialog by installing the 'eiciel' package.
'eiciel' can also be set as an extended action in Thunar to provide an extension for managing extended attributes.
The ugly
Archiving and backup
'cpio' does not support xattrs at all, neither does 'zip' or 'pax' as far as I know.
'star' which is mostly command-line-compatible with 'tar' supposedly supports extended attributes, and could be an alternative to the GNU and BSD tar. However:
-
'star' is not packaged on Debian systems because of issues with the developer.
-
I could not get star 1.5.2 to archive and restore extended attributes on a Debian machine at all. The commands used were:
tar -artype=exustar -xattr-linux -c -f archive.star toto.jpg mkdir t cd t star -artype=exustar -xattr-linux -x -f ../archive.star # The resulting toto.jpg has no extended attributes.
Common commands
The FreeBSD (11.0) version of 'cp' has no way whatsoever to preserve extended attributes. Don’t bother with the 'coreutils' port, the GNU 'cp' that comes with it is built without extended attributes support.
The 'find' command does not have a way to select files on extended attributes. There is a patch for extending it though, and seemingly, a plan to add the function to the standard version: http://ellipsix.net/blog/2009/01/extended-attributes-with-find.html
With some contorsions, it is possible to patch up something over 'find' though. E.g. (thanks to Greg Knittl for suggesting this):
find . -exec bash -c 'getfattr --name "user.test" {} &>/dev/null' \; -size 4c -print
The 'bash' part is there just to redirect the noisy messages from 'getfattr', which should have been fitted with a silent option.
NFS
No NFS support for extended attributes at this time, but there is apparently a chance that this will be some day part of NFS v4 ? http://namei.org/presentations/linuxcon09_nfsv3xattrs.pdf http://tools.ietf.org/html/draft-naik-nfsv4-xattrs-00
As far as I can see (2014-07, Ubuntu Trusty? 14.04), extended attributes are still silently lost when moving or copying files to an NFS v4 volume.
emacs-vm
If you set extended attributes on an mbox folder, emacs-vm will thrash them. It probably uses a create/rename operation to make the mbox update atomic and the xattrs don’t tag along. This is very wrong.