2005/05/07 Spotlight, mdfind (Mac OS X Tiger searching)

Most users will probably see Spotlight as an enhanced, very fast file searcher: something that builds an index of files and their contents, and can very quickly search that index and return results. Actually, Spotlight is much, much more than that, but if that's all you need, it sure beats "find" and "grep". That's true even if all we are talking about is finding text.

I'm not a bit interested in the GUI interface to Spotlight. It's fine for what it is, but the command line "mdfind" is much more interesting. But before we get to that, let's look at where Spotlight stores its index:

$ sudo bash
# pwd
/.Spotlight-V100
# ls -l
total 183136
-rw-------   1 root  admin         0 May  6 14:59 .journalHistoryLog
-rw-------   1 root  admin  32591872 May  6 15:02 .store.db
-rw-------   1 root  admin  28573696 May  6 14:55 ContentIndex.db
-rw-------   1 root  admin       391 May  6 14:59 _exclusions.plist
-rw-------   1 root  admin       378 May  1 20:34 _rules.plist
-rw-------   1 root  admin  32591872 May  6 15:02 store.db
# cat _exclusions.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>EXCLUSIONS</key>
        <array>
                <string>/Users/apl/snapshots</string>
                <string>/Users/apl/Movies</string>
                <string>/Users/apl/Music</string>
                <string>/Users/apl/Pictures</string>
        </array>
</dict>
</plist>
# 

Notice the EXCLUSIONS list? You can add to that with the System Preferences tool, but this is where it is actually stored. However, as we'll see in a moment, this is NOT all that Spotlight (and mdfind) ignores. You need to be aware of that when using mdfind.

For the simplest use, just do "mdfind whatever". Can you use that in a script? Why not?

for i in `mdfind Stuff`
do
  scp $i me@somewhere.com:$i
done
 

But mdfind is much more powerful than that:

 mdfind  'kMDItemTextContent == "*Seneca*" && kMDItemFSName != "*emlx"'
 mdfind  'kMDItemTextContent == "*Seneca*" && kMDItemContentType != "com.apple.mail.emlx"'
 

That's searching metadata. What metadata can you search? Well, anything that's available and "mdls" will show you that:

$ mdls t.txt
t.txt -------------
kMDItemAttributeChangeDate     = 2005-05-06 15:44:32 -0400
kMDItemContentCreationDate     = 2003-12-15 18:11:55 -0500
kMDItemContentModificationDate = 2005-05-06 15:44:31 -0400
kMDItemContentType             = "public.plain-text"
kMDItemContentTypeTree         = (
    "public.plain-text", 
    "public.text", 
    "public.data", 
    "public.item", 
    "public.content"
)
kMDItemDisplayName             = "t.txt"
kMDItemFSContentChangeDate     = 2005-05-06 15:44:31 -0400
kMDItemFSCreationDate          = 2003-12-15 18:11:55 -0500
kMDItemFSCreatorCode           = 0
kMDItemFSFinderFlags           = 0
kMDItemFSInvisible             = 0
kMDItemFSLabel                 = 0
kMDItemFSName                  = "t.txt"
kMDItemFSNodeCount             = 0
kMDItemFSOwnerGroupID          = 20
kMDItemFSOwnerUserID           = 501
kMDItemFSSize                  = 2552
kMDItemFSTypeCode              = 0
kMDItemID                      = 1802523
kMDItemKind                    = "Plain text document"
kMDItemLastUsedDate            = 2003-12-15 18:11:55 -0500
kMDItemUsedDates               = (2003-12-15 18:11:55 -0500)
 

What mdls (and thus mdfind) sees can change:

$ mdls z
z -------------
kMDItemAttributeChangeDate     = 2005-05-06 16:30:56 -0400
kMDItemContentCreationDate     = 2004-06-08 13:06:19 -0400
kMDItemContentModificationDate = 2005-05-06 16:30:55 -0400
kMDItemContentType             = "public.data"
kMDItemContentTypeTree         = ("public.data", "public.item")
kMDItemDisplayName             = "Z"
kMDItemFSContentChangeDate     = 2005-05-06 16:30:55 -0400
kMDItemFSCreationDate          = 2004-06-08 13:06:19 -0400
kMDItemFSCreatorCode           = 0
kMDItemFSFinderFlags           = 0
kMDItemFSInvisible             = 0
kMDItemFSLabel                 = 0
kMDItemFSName                  = "Z"
kMDItemFSNodeCount             = 0
kMDItemFSOwnerGroupID          = 20
kMDItemFSOwnerUserID           = 501
kMDItemFSSize                  = 2552
kMDItemFSTypeCode              = 0
kMDItemID                      = 2434137
kMDItemKind                    = "Document"
kMDItemLastUsedDate            = 2004-06-08 13:08:16 -0400
kMDItemUsedDates               = (2004-06-08 13:08:16 -0400)
 

Permissions can also affect metadata, which in turn changes how Spotlight and mdfind see a file:

$ diff t t.txt
$ ls -l t t.txt
-rwxr-xr-x   1 apl  staff  2552 May  6 15:41 t
-rw-r--r--   1 apl  staff  2552 May  6 15:44 t.txt
$ mdls t
t -------------
kMDItemAttributeChangeDate     = 2005-05-06 15:41:14 -0400
kMDItemContentCreationDate     = 2005-05-06 15:41:11 -0400
kMDItemContentModificationDate = 2005-05-06 15:41:11 -0400
kMDItemContentType             = "public.data"
kMDItemContentTypeTree         = ("public.data", "public.item")
kMDItemDisplayName             = "t"
kMDItemFSContentChangeDate     = 2005-05-06 15:41:11 -0400
kMDItemFSCreationDate          = 2005-05-06 15:41:11 -0400
kMDItemFSCreatorCode           = 0
kMDItemFSFinderFlags           = 0
kMDItemFSInvisible             = 0
kMDItemFSLabel                 = 0
kMDItemFSName                  = "t"
kMDItemFSNodeCount             = 0
kMDItemFSOwnerGroupID          = 20
kMDItemFSOwnerUserID           = 501
kMDItemFSSize                  = 2552
kMDItemFSTypeCode              = 0
kMDItemID                      = 5482482
kMDItemKind                    = "Unix Executable File"
kMDItemLastUsedDate            = 2005-05-06 15:41:11 -0400
kMDItemUsedDates               = (2005-05-06 15:41:11 -0400)
 

I put the string "fogpr1" in a file, and then copied it to a few different names. Using "grep" finds all of them, but "mdfind" does not:

$ grep -l fogpr1 *
Z
abcde
abcde.doc
abcde.doh
abcde.txt
foo.txt
t
t.txt
$ mdfind fogpr1
/Users/apl/t.txt
/Users/apl/abcde.txt
/Users/apl/foo.txt
 

Spotlight and mdfind also ignore "." files, even if you have told Finder not to ignore them:

$ defaults write com.apple.finder AppleShowAllFiles TRUE
$ killall Finder
 

For more technical information on Spotlight and mdfind, see http://arstechnica.com/reviews/os/macosx-10.4.ars/9 and http://developer.apple.com/macosx/spotlight.html.

Click here to add your comments




Fri Jul 1 13:06:57 2005: Subject: spotlight user search   anonymous


How does one tell spotlight to search in all users folders. It is currently formatted to ignore anything with in a users folder.



Fri Jul 1 15:01:45 2005: Subject:   TonyLawrence


man mdimport



Don't miss responses! Subscribe to Comments by RSS or by Email

Click here to add your comments

If you want a picture to show with your comment, go get a Gravatar

Jump to Comments

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.

We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.