There was a long series of comments at the article about mdfind that got very confused talking
about OS X metadata. I thought I'd try to straighten some of that out in a separate post - though honestly I'm still easily confused myself!
First, what metadata are we talking about? For an old Unix hand, the metadata is information stored in the inode: file size, permissions, pointers to datablocks, link counts.. that's traditional metadata.
However, there's more metadata today - not just in Unix systems, but especially in Mac OS X. There are extended permissions, acl's, xattributes, Spotlight related metadata.. it's very hard to ferret all this out of Google because similar terms are used for dissimilar features.
Macs had "resource forks" early on. OS X still has resource forks. but apparently Apple would like to move away from those. That's probably why things get so darn confusing: search for information on metadata and OS X and you'll find lots of pointers to things that talk about resource forks, but usually that's deprecated and doesn't usually apply to OS X.
Let's take Spotlight metadata first. These are specific keys that Spotlight indexes. For example, you can do things like this:
Once the Mac OS does kick-off the extraction of metadata from a file, it does so through a Spotlight Importer. Spotlight Importers are plug-ins for the Mac OS that a developer provides specifically for helping files created by their applications to be searchable within Spotlight. Spotlight crawls through its list of changed files, handing each one to the appropriate importer. The importers then read the files, compile a list of metadata, and then hand the metadata back to Spotlight. At this point, the changed file is available for searching within Spotlight.
Avoid the use of external files to store metadata content. All critical metadata should be in the same file as the data. The system store of metadata should be considered volatile.
I want to quibble a little: if it's stored in the data file, it's really not metadata, is it? But never mind. Some apps do it that way. For example, ID3 tags. But other apps do not. For example. In my ~/Library/Caches/Metadata I found some interesting stuff. *Some* apps store Spotlight metadata there. I found:
$ ls ~/Library/Caches/Metadata
Billings Microsoft Safari
Camino Precipitate com.evernote.Evernote
When you enable Spotlight indexing within Entourage, a "cache" file is created for each item within your Entourage database. If you have 100,000 e-mail messages in your Entourage database, 100,000 cache files will be created. If you want to see the cache files, you can find them within your Library/Caches/Metadata/Microsoft folder.
Each cache file contains all the metadata that will be needed for indexing by Spotlight. All changes within Entourage are reflected to the cache files. Create a new item and a new cache file will be created. Updated an item and its cache file will update. Delete an item and its cache file will be deleted. With all these changes, Spotlight receives file change notifications and eventually will ask the modified cache files to go through the import process using the Entourage Spotlight Importer.
But there's no iTunes folder there..
There are also defaults. If I create a text file with "date > file", an "mdls"
will show Spotlight keys:
So what do we know? Well, we know it's up to the application responsible for a file to provide importer code. It's up to the same app to decide where to store metadata. Obviously, that implies that for some data that would be the across all files of this type, there's no need to store it anywhere - the importer could generate the response when Spotlight asks.
That's as far as I've gone.. maybe someone else can add more.