Unix and Linux startup scripts, Part 2

© Andrew Smallshaw

This is a continuation of Unix and Linux startup scripts, Part 1

Local start up scripts with the BSD rc.d system

Introduction

The rc.d system is used on NetBSD, FreeBSD and DragonFly (and possibly a few other systems) to launch daemon processes when the system goes multiuser and terminate them properly at system shut down. In the interests of brevity this article will not examine the system comprehensively: rather, this guide is focussed solely on adding new scripts to control additional daemons or perform other tasks at start up and shut down. Even with this narrow focus I will not attempt a comprehensive treatment, but this guide should suffice for most common tasks and still remain brief enough to read when you actually need to write such a script.

Sadly, however, start up and shut down are one of the least standardised areas of Unix systems. This article describes recent versions of NetBSD. Since the system has been adopted by FreeBSD and DragonFly much here will be applicable to those too. On other systems it is likely that little if any of this applies.

Background

System V-inspired start up systems rely on particular naming of start up scripts to ensure that the scripts are executed in the proper order. In contrast, with the rc.d system each script details internally what services that script requires to run, and in turn what services it provides. This means that the administrator need not determine when exactly a script should execute but at the cost of additional complexity.

The whole process starts with the invocation of /etc/rc which is executed to take the system multiuser, and is ultimately responsible for the rest of the start up procedure. The logical complement that runs when the system shuts down is /etc/rc.shutdown. It may be tempting to simply add whatever we want to those scripts and be done with it. Don't. Those are operating system files and liable to change with each upgrade.

If you really do want somewhere quick and easy to add an extra couple of commands, put them in /etc/rc.local which exists for precisely this purpose, and they will be executed towards the end of the start up process. The equivalent shut down script, /etc/rc.shutdown.local, does not exist at all as shipped, at least with NetBSD, but the shut down scripts do test for its existence and will run it if it is present.

In the short term this may be the simplest method of starting your services, but in the long run it lacks flexibility. Providing the same service on another system means isolating the relevant commands. There is also no convenient ability to start, stop or restart individual services either permanently or temporarily while the system is running.

Individual service scripts

The more elegant solution is to have each service controlled by its own script which is called by /etc/rc at the correct point. This allows things like an "/etc/rc.d/lpd stop" to stop the print service independently of anything else, or an "/etc/rc.d/lpd restart" to pick up configuration changes. A discrete file is also much easier to copy to another system if needed. However, in order to integrate with the rest of the system these scripts have a few special characteristics.

The first requirement for these scripts is the command line arguments they accept. We will only consider the three most important here. The two that really matter are "start" and "stop", which correspond to the system starting up and shutting down. The next one is "restart", which is equivalent to "stop" followed by "start": it is never used by the system but is handy for the administrator when configuration files change. Scripts may provide arbitrary additional options for manual use based on what is useful or relevant.

The other special property is the presence of directives within each script. These take the form of shell script comments. The format is rigidly defined: each directive is on a line of its own. The line must begin with a hash, followed by a single space, the directive name (in uppercase), a colon, another space and finally any parameters. The directives may be placed anywhere in the file, but they must be in one contiguous block. That is to say, once the first has been found, processing stops at the next line (including blank lines) that is not a directive. Any further directives in the script will be ignored.

Four of these directives are defined:

• REQUIRE: Followed by the name of one or more services that must be operational before we execute this script.
• PROVIDE: Followed by the name of one or more services provided by this script and which may later be required by another script.
• BEFORE: This script must be executed before an existing one. Technically, this is redundant since REQUIRE and PROVIDE are all that is needed to ensure proper ordering. However, this tag may avoid the need to modify an existing script, potentially handy if it is an operating system or third party script.
• KEYWORD: The miscellaneous catch-all directive. Several keywords are defined, but we are only going to mention "shutdown", which indicates the presence of a termination procedure which needs to be called at the appropriate time.

This is already probably enough to give you an inkling of how scripts can be written such that the correct execution order can be determined systematically. However, manually coordinating between large numbers of services could still prove challenging. To combat this four placeholder "services" are provided. In and of themselves these do nothing: their role is to allow a higher-level structure in the dependencies between scripts. For clarity their names are capitalised whereas the names of individual services are by convention in lowercase. In the order they occur during start up these placeholders are:

• NETWORKING, after which basic networking services are up and running.
• SERVERS, for early start servers that general daemons may themselves depend on.
• DAEMON reflects a system where filesystems have been mounted, basic system services are running, and the system is in a fit state to start general processes. Most processes you are going to want to start should require DAEMON and will execute after this.
• LOGIN is a point after which processes that allow a user to login may be started. This does covers not only obvious login daemons such as sshd, but also things like ftpd which allow user access. The purpose of this phase is to fence off processes providing user access from those providing security so that it is impossible for a user to login to an insecure system.

It is important to note that these placeholders are points in the start up process, not phases. For example, DAEMON is a requirement for general background processes to prevent them starting too early; it does not represent the period during which they are started.

So far we have only considered the start up process. When we come to shut down the script order is reversed. This is a matter of common sense: if service A depends on B in order to operate, we have to stop it first. Only then can we go about stopping B.

/etc/rc.subr

We now have enough details to write a suitable shell script. If it contains the necessary tags for ordering and properly handles the command line arguments it will work. However, there are a number of tasks that are general to a broad range of different applications - things like identifying the correct process to terminate at shut down, or looking in /etc/rc.subr to see if the service is enabled. Doing this robustly can be surprisingly complex and we would naturally want to avoid that in each and every script.

To try and combat this a number of predefined routines are provided. These replace common code with a single set of functions defined in /etc/rc.subr. That is why if you look at many of the existing start up scripts they consist of a number of variable definitions and a couple of function calls. Scripts using these functions certainly go together quickly but at the cost of immediate clarity. This is compounded by the documentation: while it is all there there is no user guide that distils it down to what you really need to know.

As such, we will begin with the essentials. Firstly we need to make these routines available which we do by sourcing the file. /etc/rc.subr is a set of POSIX shell functions so it follows that you must code your script in a POSIX-compatible shell: that rules out csh and derivatives.

The mains guts of a script are provided by run_rc_command, a shell function that takes the command line parameter to the script as a whole. This determines what action is requested and executes the required commands. This should usually be preceded by a call to load_rc_config, which takes the name of the command and ensures that the configuration settings for it are pulled in from /etc/rc.conf and /etc/rc.conf.d/command.

To control run_rc_command we set few variables beforehand. The only truly mandatory one is $name, which should be set to the name of the script. The next in order of usefulness is $rcvar, which gives the control variable to enable or disable the script in /etc/rc.conf. Without this definition the script will always be executed irrespective of the configuration files. In general it is advisable to set this to the same thing as $name.

For many simple tasks we only need to set one more variable: $command. This is the full pathname of the executable we want to invoke. With this in mind let's consider out first example.

Example 1: dictd

dictd is a server daemon for the DICT protocol most frequently used to serve dictionary definitions. In essence it is a simple read-only database. Being read-only simplifies things a lot since the requirements are less onerous - there are no issues about ensuring writes are written to disk properly, or that the server is shut down correctly. When the time comes that we want to terminate dictd we can simply kill it.

We will assume that dictd has been installed using default settings using the NetBSD packages system and resides in /usr/pkg/sbin. We will also assume that the program has already been properly configured and will start up with a simple "dictd" at a shell prompt. The resulting script is fairly brief considering its capability:

#! /bin/sh

# REQUIRE: DAEMON

. /etc/rc.subr

name=dictd
rcvar=dictd
command=/usr/pkg/sbin/dictd

load_rc_config dictd
run_rc_command "$1"
 

In this script we specify execution after the DAEMON point in the start up process. We need to avoid dictd starting too early. It is a network service so obviously it needs networking support. Less obvious is that pathname: there is no guarantee that /usr is on the root filesystem. Indeed by default the NetBSD installer arranges things so that it isn't.

Therefore we need to ensure that it only runs after /usr is mounted. The point at which we can guarantee this is after mountcritremote but this is an implementation detail potentially subject to change. Since there is no advantage in having this server start early we will make it require DAEMON like most other additional services.

This script as written does not mark the script as providing any additional services. That is not a problem unless we have another service that needs dictd in order to run. In this case it is unlikely but to resolve that we would add a line:

# PROVIDE: dictd
 

to the script and make the dependent script require it. Surplus PROVIDE: tags will do no harm - there is nothing wrong with every script tagged as providing some service. If no other script depends on it then the statement simply has no effect.

Now to consider the operation. If the script is executable and placed in /etc/rc.d it will be called with "/etc/rc.d/dictd start" at system boot. load_rc_config will be called and will look for a line dictd=YES in /etc/rc.conf and set a variable accordingly. If present then run_rc_command, also called with the "start" command, will start dictd.

Now login as root with dictd running and type:

/etc/rc.d/dictd stop
 

and the service will stop. We can then restart it by calling the script with the "start" command. This shows the power of the of the standard routines: although we did not define the shut down function one has been supplied for us. By default this routine will look for any dictd process and send it a SIGTERM before waiting for it to actually exit. The "restart" command is also implemented and is equivalent to "stop" followed by "start". There are also a number of other options provided automatically, for example "/etc/rc.d/dictd status" will tell us if dictd is running or not. We can add options to dictd by defining $dictd_flags in /etc/rc.conf : this will be added to the command used to invoke dictd.

A few words on security are probably appropriate here, but this is not something that we are going to consider in any great depth. These scripts execute with root privileges and as a natural consequence the scripts must not be publicly writable. In general services should not run as root unless absolutely necessary. In this instance dictd will relinquish root permissions by reinvoking itself as user nobody, and we need not do anything further, but the security implications should always be considered when adding new services.

Example 2: PostgreSQL

So, our first example works fine. However, it is a very simple example. What about a more complex situation? There are many situations when we need to do more than simply launch an executable at start up, and simply killing the server is not an appropriate method of stopping many services. Many real-world, business critical services are going to require a little more care and attention but the routines in /etc/rc.subr cover those instances, too. Many such services are databases so for our next example we will consider consider PostgreSQL. As with dictd, we will assume that the DBMS is installed from the packages collection, has been initialised and is ready to start.

However, with this kind of package the way it is set up is bound to vary from site to site. There is a sample script installed as part of the package but you may not want to use it: in its attempt to be completely general it is more complex than needed for any given installation. Its behaviour may also be undesirable for many users: for instance if there is no database cluster found the script will automatically create one. That may well not be what you want.

In this example we will consider my own installation where the database cluster resides in /home/pgsql (DBAs can argue about that amongst themselves), and the server process is run by user pgsql. Unlike dictd the server does not automatically change users so this needs doing manually. We will consider three actions for controlling the database - to start the server up and shut it down as normal, and also an extra "urgent" shutdown for use if, for example, the server is running off the UPS. We can summarise the actions necessary as:

• Server start: /usr/pkg/bin/pg_ctl -D /home/pgsql -o -i start
• Normal shut down: /usr/pkg/bin/pg_ctl -D /home/pgsql stop
• Urgent shut down: /usr/pkg/bin/pg_ctl -D /home/pgsql stop -m fast

The system allows us to specify the custom actions needed for each situation by means of more shell variables. These take the general form action_cmd - where action is start, stop or some other parameter we can pass to our script. If the action is more than a one-liner then it is advisable to define shell functions to keep these definitions brief. We are not limited to the standard actions for our script: we can define additional actions based on what is our appropriate for our script, so in this case we will add an "urgentstop" option for our expedited shut down. If we do this we need to notify the subroutines of this via the $extra_commands variable.

Finally we call run_rc_command as before which will notice our custom action scripts and invoke them as and when appropriate. The net result of this is something along the lines of the following script:

#! /bin/sh

# REQUIRE: DAEMON
# PROVIDE: pgsql
# KEYWORD: shutdown

. /etc/rc.subr

name=pgsql
rcvar=pgsql
extra_commands=urgentstop

start_cmd='su pgsql -c "/usr/pkg/bin/pg_ctl -D /home/pgsql -o -i start"'
stop_cmd='su pgsql -c "/usr/pkg/bin/pg_ctl -D /home/pgsql stop"'
urgentstop_cmd='su pgsql -c "/usr/pkg/bin/pg_ctl -D /home/pgsql stop -m fast"'

load_rc_config pgsql
run_rc_command "$1"

Once we install this script and enable it in /etc/rc.conf we can see that everything works as intended.

It is an example such as this that begins to show both the power and weaknesses inherent in the rc.d system. The script is fairly brief and comparatively simple considering what it does and its flexibility. However, there is no denying there is a great deal of red tape involved. This is a problem that only gets worse if the actions needed are more than simple one-liners. In that case we need to define separate shell functions for each action. Not a big deal, but it is more bloat not directly solving the task in hand.

As such I think a certain amount of judgment is warranted. The system of directives controlling dependencies and execution ordering is undeniably elegant, but the advantage of the other infrastructure is less clear cut for more complex examples. The use of appropriate directives is mandatory for the system to work properly, but use of the supplied subroutines to control the actions taken is not. A complex set of actions may well be clearer if coded as a conventional freestanding script even if such a script is slightly longer than one using the infrastructure to its fullest extent.

Ask yourself which is clearer and easier to write: a script like that above, full of opaque variable definitions, or a case statement that looked at $1 and took the appropriate action? Fortunately a mixed approach poses few problems in practice so there is no reason to come to a rigid conclusion regardless of the particular circumstances.

References

• The Design and Implementation of the NetBSD rc.d system, by Luke Mewburn. This paper is a fairly comprehensive overview of the entire system. There is a lot of background here and much on the rationale for the overall design of the system. However, much of it is not relevant when simply creating a new start up script.
• The NetBSD Guide, Chapter 7. Background to the system from an administrative perspective. Details the starting and stopping of individual services but has very little on actually creating new scripts for additional services.
• rcorder(8) man page. rcorder is what resolves dependencies between the start up scripts in /etc/rc.d and determines the order in which to execute them.
• rc.subr(8) man page. Describes the /etc/rc.subr routines in greater depth - we have only really scratched the surface here in terms of the capabilities on offer.
• Finally, of course, there are the /etc/rc* scripts themselves. These are all shell scripts so they are easy enough to open up and examine even if you do not have the sources installed on your system.

Unix and Linux startup scripts, Part 3 explores init replacements such as Launchd and Upstart.

Got something to add? Send me email.

5 comments

Tue Dec 8 04:45:47 2009: 7723   Michiel


Excellent article. BSD has no runlevels like Linux, but single- and multiuser-mode. The order of execution of the rc scripts in BSD is not determined by file name or directory location, but the presence of certain keywords in the header of the scripts, which are used to build a dependency graph.





Fri Dec 11 04:17:15 2009: 7738   badanov


/etc/rc.local

add your favorite daemon and you're off to the races



Fri Dec 11 13:55:05 2009: 7740   TonyLawrence


He mentioned rc.local in the fifth paragraph.

------------------------

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us