Linux Section

Tar wild card interpretation

I had this email earlier this week:

I am trying to restore a file "\GL050".  I can see it on the tape
listing, but I can't get edge to find it.  I have tried listing it
the following ways:

./usr1/file/\\GL050
./usr1/file/\GL050

And I've tried it in quotes.  I also put the file path in a filelist
and tried to use edge that way.  It just will not find it.
 

Note that she is using Microlite Edge (http://aplawrence.com/Reviews/supertars.html), but that's really unimportant: you can observe the problem with tar.

$ cd /tmp
$ mkdir foo
$ touch "foo/\\filewithbackslash"
$ ls foo
\filewithbackslash
$ tar cvf test.tar foo
foo/
foo/\\filewithbackslash
 

We know have a tar file (test.tar) containing the problem file. Let's try restoring it as she did:

$ tar xvf test.tar foo/\\filewithbackslash 
tar: foo/\filewithbackslash: Not found in archive
tar: Error exit delayed from previous errors
$ tar xvf test.tar "foo/\\filewithbackslash"
tar: foo/\filewithbackslash: Not found in archive
tar: Error exit delayed from previous errors
 

Our wildcards look like they should work:

$ echo  "foo/\\filewithbackslash"
foo/\filewithbackslash
$ echo  foo/\\filewithbackslash 
foo/\filewithbackslash
 

At this point you may be saying "Are you crazy? Both those tar's work - with or without the quotes!"

And someone else would retort "Are YOU nuts?? No they don't!"

Who is nuts or not nuts has to be decided by some other method. Whether or not your tar happily works or complains as shown above simply depends upon how it handles wildcards and whether or not it gets to see them.

The complaining tar above identifies itself as "(GNU tar) 1.14" and was executed on Mac OS X 10.4.5. A non-complaining tar on a RedHat Linux systems says that it is "(GNU tar) 1.13.25".

Isn't that odd: the newer version seems to work "incorrectly". There's an interesting section of the "info tar" for the 1.13.25 version:

   There are some discussions floating in the air and asking for
modifications in the way GNU `tar' accomplishes wildcard matches.  We
perceive any change of semantics in this area as a delicate thing to
impose on GNU `tar' users.  On the other hand, the GNU project should be
progressive enough to correct any ill design: compatibility at all price
is not always a good attitude.  In conclusion, it is _possible_ that
slight amendments be later brought to the previous description.  Your
opinions on the matter are welcome.
 

Info on the Mac version lacks that paragraph - though it still strongly implies that our syntax should have worked:

"Globbing" is the operation by which "wildcard" characters, `*' or `?'
for example, are replaced and expanded into all existing files matching
the given pattern.  However, `tar' often uses wildcard patterns for
matching (or globbing) archive members instead of actual files in the
filesystem.  Wildcard patterns are also used for verifying volume
labels of `tar' archives.  This section has the purpose of explaining
wildcard syntax for `tar'.

   A PATTERN should be written according to shell syntax, using wildcard
characters to effect globbing.  Most characters in the pattern stand
for themselves in the matched string, and case is significant: `a' will
match only `a', and not `A'.  The character `?' in the pattern matches
any single character in the matched string.  The character `*' in the
pattern matches zero, one, or more single characters in the matched
string.  The character `\' says to take the following character of the
pattern _literally_; it is useful when one needs to match the `?', `*',
`[' or `\' characters, themselves.
 

That seems pretty plain, doesn't it? But it sure doesn't work as advertised.

Hold on, someone in the back is waving their arm frantically. They have a question. What's that? A little louder, please. Oh, yes.. the shell *does* expand wildcards.

If it can.

When it cannot, or when we prevent it, it's tar's responsibility entirely. Both man pages take note of that:

   The distinction between file names and archive member names is
especially important when shell globbing is used, and sometimes a
source of confusion for newcomers.  *Note Wildcards::, for more
information about globbing.  The problem is that shells may only glob
using existing files in the file system.  Only `tar' itself may glob on
archive members, so when needed, you must ensure that wildcard
characters reach `tar' without being interpreted by the shell first.
Using a backslash before `*' or `?', or putting the whole argument
between quotes, is usually sufficient for this.
 

Not sufficient on the Mac, though.

Another interesting anomaly: although the Mac man page doesn't mention it, the changelog of the .14 release mentions some new flags, including:

--wildcards
--no-wildcards
    When using wildcards (the default), *, ?, and [...] are the
    usual shell wildcards, and \ escapes wildcards. Otherwise, none
    of these characters are special, and patterns must match names
    literally.
 

Turns out that the older version groks those flags too, and adding them doesn't help the Mac test at all.

So how do you solve this if your tar doesn't do complete wildcards? One way is to do an interactive restore where you have to affirm each file before it is restored. In this particular case (using Microlite Edge) the pattern

"foo/?filewithbackslash" 
 

cuts down on the number of matches and if there is no other "?filewithbackslash" will quickly restore the desired file.

GNU tar is now at 1.15, by the way: I haven't tested to see how it reacts to cases like this. I also haven't compiled 1.14 from scratch on Linux; this behaviour may be unique to the Mac instance for some reason I'm just not aware of.

The usual lessons apply: the same command works differently on different platforms, things change, read the man and info pages but don't trust them, and always be prepared to experiment.

7 comments

More Articles by Anthony Lawrence - Find me on Google+

Click here to add your comments




Wed Apr 5 14:14:00 2006:   BigDumbDInosaur


Of course, if the wonk who created the file had *not* used a \ in the filename (which, BTW, would ambiguous at best in DOS or Windows) this wouldn't be a problem, eh? Why people would do such things baffles the daylights out of me. Aren't there enough characters in the alphabet to come up with reasonable filenames? Or am I just a grouchy old dinosaur?



Wed Apr 5 14:36:07 2006:   rbailin


Using single quotes to escape the backslash works as expected:

 

/tmp> mkdir foo 

/tmp> touch foo/\\file 

/tmp> l foo 

total 0 

-rw-r-----   1 root     sys            0 Apr  5 10:27 \file 

/tmp> tar cvf test.tar foo 

a foo/\file 0 tape blocks 

/tmp> tar tvf test.tar 

tar: blocksize = 4 

rw-r-----  0/3        0 Apr 05 10:27 2006 foo/\file 

/tmp> tar xvf test.tar 'foo/\file' 

tar: blocksize = 4 

x foo/\file, 0 bytes, 0 tape blocks 

/tmp> 

--Bob



Wed Apr 5 14:46:27 2006:   rbailin


I just noticed that this does work on SCO 5.0.7:

 

/tmp> tar xvf test.tar foo/\\file 

tar: blocksize = 4 

x foo/\file, 0 bytes, 0 tape blocks

Notice in my previous post that: tar cvf test.tar foo
only backed up the file with the backslash and not the
directory 'foo' also.

This may have to do with default behavior of tar on
different unix versions. What O/S was your client running?

--Bob



Wed Apr 5 14:49:01 2006:   TonyLawrence


I didn't mention the os because that's the point: wildcards work differently on different platforms/tar versions.




Thu Apr 6 16:28:30 2006:   rbailin


Sorry about that. I sort of missed the whole point
of the post, didn't I? Although in my defense this
is a predictable outcome of short attention span
on my part, and "burying the lead" of the article
on your part.

I assumed it was an inherent problem with tar, and
couldn't imagine that wildcard resolution is still
a problem with open source software after X number
of years.

--Bob



Thu Apr 6 17:49:21 2006:   TonyLawrence


It is hard to imagine, isn't it?

I've been trying to find out more about that comment in the 1.13 release concerning possible changes.. so far it's been tough to Google because too much other stuff comes up.

Don't miss responses! Subscribe to Comments by RSS or by Email

Click here to add your comments

If you want a picture to show with your comment, go get a Gravatar

Jump to Comments

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.

We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.