Jim Mohr's SCO Companion

Index

Copyright 1996-1998 by James Mohr. All rights reserved. Used by permission of the author.

Be sure to visit Jim's great Linux Tutorial web site at http://www.linux-tutorial.info/

Shells and Basic Utilities

Commonly Used Utilities

Previous: Shells and Basic Utilities

There are hundreds of utilities and commands plus thousands of support files in a normal SCO ODT or OpenServer installation. Very few people I have met know what they all do. As a matter of fact, I don't know anyone who knows what they all do. Some are obvious and we use them every day, such as date. Others are not so obvious and I have never met anyone who has used them, like xtd.

Despite the overwhelming number of them and their often cryptic names and even more cryptic options, there are many commands that are very useful and very powerful.

I have often encountered users, as well as system administrators, who will combine many of this commands into something relatively complicated. The only real problem is there is often a single command that would do all of this for them.

In this section we are going to cover some of the more common commands. I am basing my choice on a couple of things. First, I am going to cover those commands that I personally use on a regular basis. These are ones that I use to do things I need to do, or those that I help end users use to get done what they need to. Next, is the SCO UNIX system itself. There are dozens of scripts, scattered all through the system that contain many of these commands. By talking about them here, you will be in a better position to understand existing scripts should you need to expand or troubleshoot them.

Since utilities are usually part of some larger process (such as installing a new hard disk or adding a new user), I am not going to talk about them here. I will get to the more common utilities as we move along. For a wonderful overview, see Ellie Cutler's SCO UNIX in a Nutshell from O'Reilly & Associates.

Looking For Files

There are many ways to do to things that you want. Some use a hammer approach and force the answer out of the system. In many cases, there are other commands that do the exact same thing without all the gyrations. So, what I am going to try to do here is step through some of the logic (and illogic) that I went through when first learning SCO UNIX. That way we can all laugh together at how silly I was and maybe you won't make the same mistakes I did.

Every dialect of UNIX that I have seen has the ls command. This gives a directory listing of either the current directory if no argument is given or a listing of what every files or directories are specified as arguments. The default behavior under SCO UNIX for the ls command is to list the names of the files in a single column. Try it and see.

It is a frequently (maybe not common) misconception for new users to think that they have to be in a particular directory to get a listing of it. They will spend a great deal of time moving up and down the directory tree looking for a particular file. Fortunately, they don't have to do it that way. This issue with this misunderstanding is that every command is capable of working with paths, as it is the operating system that does the work. Remember from our discussion of SCO basics. Paths can be relative to our current directory such as ./directory or absolute, such as /usr/jimmo/directory.

For example, assume you have a sub-directory of your current working directory called letters. In it are several sub-directories for the type of letter, such as business, school, family, friends and taxes. To get a listing of each of this directories, you could do:

ls ./letters/business

ls ./letters/school

ls ./letters/family

ls ./letters/friends

ls ./letters/taxes

Since the ls command let's you have multiple commands on the same line, you could have issued the command like this:

ls ./letters/business ./letters/school ./letters/family ./letters/friends ./letters/taxes

Both will give you a listing of each of the five directories. Even for five directories, typing all of that is a pain. You might think you can save some typing if you simply entered:

ls ./letters

Well, this gives you a listing of all the files and directories in ./letters, Not the sub-directories. Instead, if you entered:

ls ./letters/*:

The shell will expand the wildcard (*) and give you a listing of both the ./letters directory as well as the directories immediately below ./letters, like the second example above. If each of the sub-directories is small, then this might fit onto one screen. If, on the other hand, you have 50 letters in each of these sub-directories, they are not all going to fit on the screen at once. Remember our discussion on shell basics? You can use the pipe (|) to send the command through something like more or pg so you could read it a page at a time.

What if the taxes directories contained a sub-directory for each year for the past five years, each of these contained a sub-directory for each month, each of these contained a sub-directory for federal, state and local taxes and each of these contained 10 letters?

If we knew that the letter we were looking for was somewhere in the taxes sub-directory, the command ls ./letters/taxes/* would show us the sub-directories of taxes (federal, local, state), and it would show their contents. We could then look through this output for the file we were looking for.

What if the file were looking for was five levels deeper? We could keep adding wildcards (*) until we reached the right directory, as in:

ls ./letters/taxes/*/*/*/*/*

This might work, but what happens if the files were six levels deeper. Well, we add an extra wildcard. What if it were ten levels deeper and we didn't know? Well, we could fill the line with wildcards. Even if we had too many, we would still find the file we are looking for.

Fortunately for us, we don't have to type in ten asterisks to get what we want. We can use the -R option to ls to do a recursive listing. The -R  option also avoids the "argument list too long” error that we might get with wildcards. So, the solution here is to use the ls command like this:

ls -R ./letters/taxes | more

The problem with that is we now have 1800 files to look through. Piping that through more and looking for the right file will be very time consuming. If knew that it was there, but we missed it on the first pass, we would have to run through the whole thing again.

The alternative is to have more search for the right file for you. Because the output is more than one screen, more will display the first screen and at the bottom display --More--. Here we could type a slash (/) followed by the name of the file and enter. More will now search through the output until it finds the name of the file. Now we know that the file exists.

The problem here is the output of the ls command. We can find out whether a file exists by this method, but we cannot really tell where it is. If you try this, you will see that more jumps to the spot in the output where the file is (if it is there). However, all we see is the file name, not what directory it is in. Actually, this problem also exists even if we didn't search for it.

An alternative would be to pipe the output through pg. Both pg and more have the ability to search for specific text. If not used as a pipe, but using the name of a file as an argument, both can jump to a specific line number. For example, the command:

pg +42 file_name

will jump to line 42 of the file file_name before displaying any output. If, however, we use more or pg as the end of the pipe, the plus-sign (+) will move us forward so many screens.

There are a couple advantages that pg offers. First it has the ability to move backward. In contrast to the --More-- from more, pg uses a colon (:) as its prompt. Here, we can use a minus (-) to move backwards screens.

However, more does have one advantage over pg that I use quite often. This usually occurs when I have several files in a directory that contains some information I need to edit. I could use more to search the files until I found the right one, then use vi to edit the file. However, more  makes things easier. Once I get the --More-- prompt, I simply press ‘v' and that brings me into vi. (I found this out by accidentally pressing 'v', when I wanted to press the space bar to move down one screen)

Note that this does not work when using more as the end of a pipe. You cannot edit standard input and that's where more is getting its input from.

If you use more as the command and not the end of a pipe, instead of just seeing --More--, you will probably see something like:

--More--(16%)

This means that you have read 16% of the file.

However, we don't need to use more for that. Since we don't want to look at the entire output, just search for a particular file, we can use one of three commands that SCO UNIX provides to do pattern searching: grep, egrep, and fgrep. The names sound a little odd to the SCO UNIX beginner, but grep stands for global regular expression print. The other two are newer versions that do similar things. For example, egrep search for patterns that are full regular expressions and fgrep searches for fixed strings and is a bit faster.

Let's assume that we are a tax consultants and have 50 sub-directories, one for each client. Each is then broken down by year, type of tax (state, local, federal, sales, etc). A couple years ago, a client we have bought a boat. We have a new client who also wants to buy a boat and we need some information in that old file.

Since we know the name of that file we can use grep to find it for me, like this:

ls -R ./letters/taxes | grep boat

Even if the file was called boats, boat.txt, boats.txt or letter.boat, the grep will find it. The reason is because grep is only looking for the pattern boat. Since that pattern exists in all four of those file names, all four would be potential matches.

The problem is that the file may not be called boat.txt, but rather Boat.txt. Remember, unlike DOS, UNIX is case-sensitive. Therefore, grep sees boat.txt and Boat.txt as different things. The solution here would be to tell grep to look for both.

Remember our discussion on regular expression in the section on shell basics? Not only can we use regular expressions for file names, we can often used them in the arguments to commands. The term regular expression is even part of grep's name Using regular expressions, the command might look like this:

ls -R ./letters/taxes | grep [Bb]oat

This would now find both boat.txt and Boat.txt.

Some of you many see a problem with this as well. Not only does SCO UNIX see a difference between boat.txt and Boat.txt, but also between Boat.txt and BOAT.TXT. In order to catch all possibilities, we would have to have a command something like this.

ls -R ./letters/taxes | grep [Bb][Oo][Aa][Tt]

Although this is perfectly correct syntax and it will find it no matter what case the word ‘boat' is in, it is too much work. The programmers who developed grep realized that people would want to look for thing regardless of what case they were in. Therefore, they build in the -i option. This simply say ignore the case. Therefore the command:

ls -R ./letters/taxes | grep -i boat

will not only find boats, boat.txt, boats.txt and letter.boat, but it will also find Boat.txt and BOAT.TXT as well.

If you've been paying attention, you might have noticed something. Although the grep command will tell you about the existence of a file, it won't tell you where it is. This is just like piping it through more, only we're filtering out something. Therefore , it still won' tell you the path.

Now, this isn't grep's fault. It did what it was supposed to. We told it to search for a particular pattern and it did and displayed that pattern for us. The problem still is the fact that the ls command is not displaying the full paths of the files, just their names.

Instead of ls, let's use a different command. Let's use find instead. Just as its name implies, find is used to find things. What it finds is files. If we change the command to look like this:

find ./letters/taxes -print | grep -i boat

This finds what were are looking for and gives us the paths as well.

Before we go on, let's look at the syntax of the find command. There are a lot of options and it does look foreboding, at first. we find it is easiest to think of it this way:

find <starting_where> <search_criteria> <do_something>

In this case, the "where" is ./letters/taxes. Therefore, find started its search in the./letters/taxes directory. Here, we had no search criteria, we simply told it to do something. That something was -print out what it finds. Since the files it finds all have a path relative to ./letters/taxes, this is included in the output. Therefore, when we pipe it through grep, we get the path to the file we are looking for.

We also need to be careful since the find command we are using will also find directories named boat. This is because we did not specify any search criteria. If instead, we wanted it just to look for regular file (which is often a good idea), we could change the command to look like this:

find ./letters/taxes -type f -print | grep -i boat

Here we see a the option -type f as the search criteria. This will find all the files of type f for regular files. This could also be a d for directories, c for character special files, b for block special files, and so on. Check out the find(C) man-page for other types that you can use.

Too complicated? Let's make things easier on us by avoiding grep. There are many different things that we can use as search criteria for find. Take a quick look at the man-page and you will see that you can search for specific owner, groups, permissions even names. Instead of having grep do the search for us, let's save a step (and time) by having find do the search for us. The command would then look like this:

find ./letters./taxes -name boat -print

This will then find any file named boat and list their respective paths. The problem here is that it will only find the files named boat. It won't find the files boat.txt, boats.txt or even Boat.

The nice thing is that find understands about regular expression so we could issue the command like this:

find ./letters./taxes -name ‘[Bb]oat' -print

(Note that we had to include the single-quote (‘) to avoid the square brackets ([]) from being first interpreted by the shell)

This command told find to look for all files named both boat and Boat. However, this won't find boat. We are almost there.

We have two alternatives. One is to expand the find to include all possibilities as in:

find ./letters./taxes -name ‘[Bb][Oo][Aa][Tt]' -print

As in the example above, This will find all the files with any combination of those four letters and print them out. However, it won't find boat.txt. Therefore we need to change it yet again. This time we have:

find ./letters./taxes -name ‘[Bb][Oo][Aa][Tt]*' -print

Here we have passed the wildcard (*) to find to tell it took find anything that starts with ‘boat' (upper- or lower-case) followed by anything. If we add an extra asterisk, as in:

find ./letters./taxes -name ‘*[Bb][Oo][Aa][Tt]*' -print

We not only get boat.txt, but also newboat.txt, which the first example would have missed.

This works. Is there an easier way? Well, sort of. There is a way that easier in the sense that there are less characters to type in. This would be:

find ./letters/taxes -print | grep -i boat

Isn't that the same command as we issue before? Yes, it is. In this particular case, this combination of find and grep is the easier solution, since all we are looking for is the path to a specific file. However, these examples show you different options to find and different ways that you can use it.

Looking Through Files

Let's assume for a moment that none of the commands we issues came up with a file. we mean there was not a single match of any kind. This might mean that we removed the file. On the other hand we might have named it yacht.txt or something similar. What can we do to find it?

We could jump through the same hoops for yacht as we did for boat. However, what if the customer had a canoe or a junk. Are we stuck with every possible word for boat? Yes, unless we know something about the file, even if that something is in the file.

The nice thing is that grep doesn't have to be the end of a pipe. One of the arguments can be the name of a file. If you want, you can put several files, since grep will take the first argument as the pattern it should look for. If we were to enter:

grep [Bb]oat ./letters/taxes/*

we would search the contents of all the files in the directory ./letters/taxes looking for the word Boat or boat.

If the file we are looking for happens to be in the directory ./letters/taxes, then all is well. If things are like the examples above where we have dozens of directories to look through, this is impractical. So, we turn back to find.

One of the useful options to find is -exec. When a file is found, you use -exec to execute a command. We can therefore use find to find the files, then use -exec to run grep on them. Since you probably don't have dozens of files on your system related to taxes, let's use an example from files that you most probably have.

Let's find all the files in the /etc directory containing "/bin/sh”. This would be run as:

find ./etc -exec grep /bin/sh {} \;

The {} substitutes the file found by the search, so the actual grep command would be something like:

grep /bin/sh ./etc/filename

The "\;”, is a flag saying that this is the end of the command.

What the find command does is to search for all the files that match the specified criteria, (in this case there was no criteria so it found them all) then run grep on them searching for the pattern [Bb]oat.

Do you know what this tells me? It says that there is a file somewhere under the directory ./letters/taxes that contains either ‘boat' or ‘Boat'. It doesn't tell me what the file name is called. The reason is the way the -exec is handled. Each filename is handed off one at a time, replacing the {}. It would be as if we had entered individual lines for:

grep ‘[Bb]oat' ./letters/taxes/file1

grep ‘[Bb]oat' ./letters/taxes/file2

grep ‘[Bb]oat' ./letters/taxes/file3

If we had entered:

grep ‘[Bb]oat' ./letters/taxes/*

the grep would have output the name of the file in front of each matching line it found. However, since each line is treated separately when using find, we don't see the file names. we could use the -l option to grep, but that would only give me the file name. That might be okay if there was one or two files. However, if the line in the file mentioned a "boat trip” or a "boat trailer”, these might not be what we was looking for. If we used the -l option to grep we wouldn't see the actual line. Catch-22.

In order to get what we need, we have to introduce a new command: xargs. By using it as one end of a pipe, you can repeat the same command on different files without actually having to input it multiple times.

In this case, we would get what we wanted by typing:

find ./letters/taxes -print | xargs grep [Bb]oat

The first part is the same as we talked about above. The find command simply prints all the names it finds (all of them in this case, since there was no search criteria) and passes them to xargs. Next, xargs processes them one at a time and creates commands using grep. However, unlike the -exec option to find, xargs will output the name of the file before each matching line.

Obviously, this example does not find those instances where the file we was looking for contained words like yacht or canoe instead of boat. Unfortunately, the only way to catch all possibilities is to actually specify each one. So, that's what we might do. Rather than listing the different possible synonyms for boat, let's just take the three: boat, yacht and canoe.

To do this we need to run the ‘find | xargs' command three times. However, rather than typing in the command each time, we are going to take advantage of a useful aspect of the shell. In some instance, the shell knows when you want to continue with a command and gives you a secondary prompt. If you are running sh or ksh, then this is probably ‘>‘.

For example if we typed:

find ./letters/taxes -print |

The shell knows that the pipe (|) cannot be at the end of the line. It then gives me a > or ? prompt where we can continue typing:

> xargs grep -i boat

The shell interprets these two lines as if we had typed them all on the same line. we can use this with a shell construct that lets me do loops. This is the for/in construct for sh and ksh, and the foreach construct in csh. It would look like this:

for j in boat ship yacht

> do

> find ./letters/taxes -print | xargs grep -i $j

> done

In this case, we are using the variable j, although we could have called it anything we wanted. When we are putting together such quick little commands, we save ourselves a little typing our using single letters variables.

In the sh/ksh example, we need to enclose the body of the loop inside the do-done pair. In the csh example we need to include the end. In both cases this little command we have written will loop through three times. Each time, the variable $j is replaced with one of the three words that we used. If we thought up another dozen or so synonyms for boat, then we could have included them all. Remember also that the shell knows that the pipe (|) is not the end of the command, so this would work as well.

for j in boat ship yacht

> do

> find ./letters/taxes -print |

> xargs grep -i $j

> done

Doing this from the command line has a drawback. If we want to use the same command again, we need to retype everything. However, using another trick, we can save the command. Remember that both the ksh and csh have history mechanisms to allow you to repeat and edit commands that you recently edited. However, what happens tomorrow when you want to run the command again? Granted, ksh has the .sh_history file, but what about sh and csh?

Why not save this in a file that we have all the time? We could use vi or some other text editor. However, we could take advantage of a characteristic of the cat command, which is normally used to output the contents of a file to the screen. You can also redirect the cat to another file.

If we wanted to combine the contents of a file we could do something like this:

cat file1 file2 file3 >newfile

The would combine file1, file2 and file3 into newfile.

What happens if we leave the names source files out? So instead our command would look like this:

cat >newfile

Now, cat will take its input from the default input file: stdin. we can now type in lines, one at a time. When we are done, we tell cat to close the file by sending it an end of file character, CTRL-D. So, to create the new command we would issue the cat command as above and type in our command

for j in boat ship yacht

do

find ./letters/taxes -print |

xargs grep -i $j

done

<CTRL-D>

Note that here the secondary prompt (>) does not appear since it is cat that is reading our input and not the shell. we now have a file containing the five lines that we typed in that we can use a shell script.

However, right now all that we have is a file named newfile that contains five lines. we need to tell the system that it is a shell script that can be executed. Remember in our discussion on operating system basics I said that a file's permissions need to be set in order to be able to execute a file. In order to change the permissions, we need a new command: chmod. (Read: "change mode” since we are changing the mode of the file)

The chmod command is used to not only change access to the file, but also tell the system that it should try to execute the command. I said "try” because the system would read that file, line-by-line, and try to execute each line. If we typed in some garbage in a shell script, the system would try to execute each line and would probably report: not found for every line.

To make a file execute, we need to give it execute permissions. To give everyone execution permissions, you use the chmod command, like this:

chmod +x newfile

Now the file newfile has execute permissions, so, in a sense, it is executable. However, remember that I said the system would read each line. In order for a shell script to function correctly, it also needs to be readable by the person executing it. In order to read a file, you need to have read permission on that file. More than likely, you already have read permissions on the file since you created it. However, since we gave everyone execution permissions, let's give them all read permissions as well, like this:

chmod +r newfile

You now have a new command called newfile. This can be executed just like any the system provides for you. If that file resides in a directory somewhere in your path, all you need to do is type it in. Otherwise, (as we talked about before) you need to enter in the path as well. Keep in mind that the system does not need to be able to read binary programs. All it needs to be able to do is execute them.

There you have your first shell script and your first self-written UNIX command.

What happens if after looking though all of the files, you don't find the one you are looking for. Maybe you were trying to be sophisticated and used "small aquatic vehicle" instead of boat. Now, six months later, you cannot remember what you called it. Looking through every file might take a long time. If only you could shorten the search a little. Since you remember that the letter you wrote was to the boat dealer, if you could remember the name of the dealer, you could find the letter.

The problem is that six months after you wrote it, you can no more remember the dealer's name than you can remember whether you called it a "small aquatic vehicle" or not. If you are like me, seeing the dealer's name will jog your memory. Therefore, if you could just look at the top portion of each letter, you might find what you are looking for. You can take advantage of the fact that the address is always at the top of the letter and use a command that is designed to do look there. This is the head command and we could use it something like this:

find ./letters/taxes -exec head {} \;

This will look at the first ten (the default for head) lines of each of the files that it finds. If the addressee were not in the first ten lines, but rather in the first 20 lines, we could change it to be:

find ./letters/taxes -exec head -20 {} \;

The problem with this is that 20 lines is almost an entire screen. If you ran this is would be comparable to running more on every file and hitting q to exit after it showed the first screen. Fortunately we can use add another command to restrict the output even further. This is the tail command as is just the opposite of head as it shows you the bottom of the file. So, if we knew that the address resided on lines 15-20, we could run a command like this:

find ./letters/taxes -exec head -20 {} \; | tail -5

This command passes the first 20 lines of each file the pipe and then tail displays the last 5 lines of that. So you would get lines 15-20 of every file, right? Not quite.

The problem is that the shell sees these as two tokens. That is two separate commands find ./letters/taxes -exec head -20 {} \; and tail -5. All of the output of the find is sent to the pipe and it is the last five lines of this that tail shows. Therefore, if the find | head had found 100 files, we would not see the contents of the first 99!

The solution is to add two other shell constructs: while and read. This carries out a particular command (or set of commands) while some criteria is true. The read can read input either from the command line, or as part of a more complicated constructions. So, using cat again to create a little command like we did above, we could have something like this:

find ./letters/taxes -print | while read FILE

do

echo $FILE

head -20 $FILE | tail -5

done

In this example, the while and read work together. The while will continue so long as it can read something into the variable FILE, that is, so long as there output coming from the find. Here again, we also need to enclose the body of the loop within the do-done pair.

The first line of the loop simple echoes the name of the file so we can keep track of what file is being looked at. Once we found the correct name, we could use it as the search criteria for a find | grep command. This does require looking through each file twice, however if all you need to see is the address, then this is a lot quicker than doing a more on every file.


Next: Basic Shell Scripting

Next: Users and User Accounts

Index

Copyright 1996-1998 by James Mohr. All rights reserved. Used by permission of the author.

Be sure to visit Jim's great Linux Tutorial web site at http://www.linux-tutorial.info/