Out of DiskSpace?

APLawrence.com - Resources for Unix and Linux Systems, Bloggers and the self-employed

Out of Disk Space

If you are very low on space, you may have trouble logging in. Most of the time, that's simply that the screen is being messed up by messages, and nothing is really preventing you- just ignore the scrolling messages and continue. Very rarely, you may actually be prevented from logon. In that case, you'll need to power off the system and bring it up single user mode (be sure to answer yes to any questions about "cleaning"). You will always be able to login single user.

This article is somewhat specific for SCO OSR5, but will be useful for other Unix versions and for Linux if you ignore the os specific parts. Directories may be different; for example, many Unixes store logs etc. under /var.

The dreaded message "Out of Space on device 1/40" means that you are out of space on your root partition. If it's 1/42 or some other number, it's a different filesystem (3.2v5.0.x would be 1/42 because /stand is 1/40). Type df -v and look for the one that is 99% or more used.

Historical note: prior to SCO 5.0.6, %used was calculated incorrectly for large numbers of free blocks. The history of computer programming is just full of these silly assumptions about how big numbers need to be.

We hope that 64 bit computing eliminates this for a while - at least in new code. Most likely, rewriters of old code will be hasty and 32 bit and even 16 bit assumptions will be left in logic if not in actual storage. If so, silly bugs will continue to pop up over the span of years.

32 bits is a 4GB address space. That "address" could be disk blocks, so of course a 1,024 byte block size is a 4 Terabyte drive and larger block sizes would allow larger drives. A 64 bit number is 18.45 Exabytes. That's 1,048,576 times larger than 18.45 Terabytes, so you could address a incredibly large disk with as small a block size as you liked.

The "out of space" messages keep scrolling while you are working, at least until you clear up enough space to stop them. THESE MESSAGES ARE NOT INTERFERING WITH LOGGING IN OR TYPING COMMANDS. Just ignore them and proceeed as detailed below (but see the sidebar here also).

You may be out of space. There may be nothing you can do but go buy another hard drive. See Adding a disk drive if that's what you need to do.

You could have a confused file system: try fsck -ofull (single user mode for /dev/root, unmounted for all others).

On the other hand, there may be extra large files on your disk, and some of them may be junk or simply filled up log files. Here are some places to look, and some tools to use. Check your progress by

df -v.

Those nasty core files

Core files are simply dumps of the memory image of a program. They are generated when you type Ctrl-\ to get out of a hung program, or can be generated when the program does something illegal. If you understand adb, and understand what the program was trying to do, you could figure out why it hung or crashed. Chances are, you don't care, and (since you don't usually have the source code) couldn't do anything about it even if you could figure out the problem. Therefor, you might as well just get rid of them.

Type:

cd /

find . -name core -exec rm {} \;

Directories that can be emptied

/tmp and /usr/tmp

Start by

cd /tmp

For safety's sake, always confirm that you are where you think you are by typing

pwd

before you remove anything. Once you are satisfied that you really are where you think you are, type

rm *

However, you might be smarter to use ">" on each file first. See Freeing disk space with ">" for why you might want to use ">".

In /usr/tmp, use

rm -r *

If you have Word Perfect or other programs that create their own directories in /tmp, tread carefully there. It is probably safe to cd into any sub-directory and remove anything you find therein, but the directory itself may need to stay.

Don't EVER type "rm -r .*"

On older systems, there may be too many files for the command line to process. Use xargs:

find . -print | xargs rm

Files in /lost+found can probably be removed if they've been there for a while and you haven't had any other problems. Files in /lost+found owned by lp can almost always be removed. In desperation, go ahead and rm -r everything in /lost+found; if it was anything serious you'd probably have missed it before now.

Where has the space gone?

Although time consuming, the following procedure can be used to track down where your space has been used.

cd /

du -s *

(Some folks like to use "du -cks *", which is easy to remember as "ducks".)

This command will print the number of blocks used by each directory and file in the root. Most likely, the largest number printed is where you should be looking. You can then cd to that directory, and examine it. If it has sub-directories, you might use:

find . -type d -exec du -s {} \;

You can search for "large" files by cd'ing to a suspect directory ( or even starting at /, if you must), and typing

find . -size +5000 -print

will print the names of all files over 5,000 blocks (2,560,000) bytes. This may find many, many files, so you might want to refine it with larger numbers. You might also want to sort it:

find / -size +2000 -exec ls -s {} \; | sort -nr | more

To limit find to the filesystem you are on, add "-mount":

find . -mount -size +5000 -print

Ordinary files in /dev

Other than directories, there should be nothing in /dev except device files. Try:

cd /dev

find . -type f -exec l {} \;

(On R5, you might see a /marry/marry.lock file. That one's OK). If you find large files that look like they might be mistyped device names (/dev/rtc0, etc.), remove them.

There is also the remote possibility that your file system is simply confused. If "df -v" and "du -s /" don't agree that approximately the same number of blocks are in use, this might be the case (note that if you have mounted file systems, du will include their usage). The numbers won't agree 100% because of links. "fsck -s" will correct this, but must be done on an unmounted file system or in single user mode for the root file system.

Files that can be truncated

/usr/spool/lp/logs/lpsched, /usr/spool/lp/logs/requests

/usr/adm/messages, /usr/adm/syslog, /usr/lib/cron/log

/var/scohttp/logs/access_log, /var/scohttp/logs/error_log, /var/scohttp/logs/startup_errors

/usr/local/lib/apache/var/log/*

/etc/utmp, /etc/wtmp

/etc/utmpx, /etc/wtmpx

/tcb/audittmp (this directory may contain audit files; they can be removed)

/usr/spool/uucppublic (if you use uucp)

/usr/spool/uucp/.Log (sub-directories therein may contain log files)

/usr/mmdf/log/*.log

/usr/mmdf/lock/home/*, /usr/spool/mail/*

Visionfs keeps a LOT of logs under vfsdata/logs. You can remove any of the .Z files:

find /usr/vision/vfsdata/logs -name '*.Z' -exec rm {} \;

There may be old files in vfsdata/spool also.

Any of these could be very large. Truncate by typing

> /usr/spool/lp/logs/requests

That's a ">" followed by the name of the file (no wildcards, you must use the name of the file). This will empty the file but leave the name still there, with permissions and ownership intact.

See Freeing disk space with ">" for another reason to use ">".

Some of these files contain useful information. It's possible that you may want to leave all or part of these files alone. For example, you may wish to extract warnings and errors from /usr/adm/messages before truncating it. However, if you don't understand the messages, and cannot print them, truncate. Anything serious will probably get repeated.

egrep "CONFIG|WARN|NOTICE" /usr/adm/messages | sort -u | more

will extract some of this information.

See also Boot up Messages for more information

The contents of syslog, in particular, may have useful information that could tell you why your disk filled up. You might consider something like:

tail -100 /usr/adm/syslog | /usr/lib/lponlcr > /dev/lp0

(assuming, of course, that a printer is connected to /dev/lp0).

Remember that if a process has the file open, the space you remove will not appear in the free list until the process closes the file, either of its own choice or through being killed.

If you can't identify the process, a reboot will "fix" it, but if the process still has a problem, your disk space will disappear yet again.

On systems not using /stand, you may have copies of /unix, such as /unix.old, /unix.safe, etc. You probably want to keep at least one of them as an emergency kernel, but (if necessary), rm the other copies.

Examine /usr/spool/lp/temp for files ending in -0 and -1. You might see 2337-1, for example. These are print jobs either waiting to spool or that somehow got lost and never removed. If the print spooler is running and you don't want to confuse it, execute /usr/lib/lpshut before removing these, or simply truncate the -1 files (leave the -0's alone).

Check /usr/spool/mail. If you aren't using mail, rm everything there. However, keep in mind that the mail may very well be giving you the clue as to why your disk space is gone. Leave the :saved directory (if present), but rm it's contents.

Check /usr/spool/mmdf. Look for msg. files in the sub-directories. These are undelivered messages. Remove them (after stopping the "deliver" process"). Symptoms of this problem could include a large "sys" file in /usr/spool/mail, containing messages from cron complaining about being unable to change to sys's home directory. This can be caused by not having a home directory for the user sys; it originally was /usr/sys, but sometimes people remove it because it's (generally) empty. Put it back to avoid this problem.

mkdir /usr/sys;chown sys /usr/sys

Another cause is a stuck lock file; if you see anything .lock in /usr/spool/mail that isn't dated today, that's going to stop delivery of messages to that account.

If there were many thousands of messages, you may want to rm -r the directory they are in (to avoid leaving a large, mostly empty directory), but note the name, permissions, and ownership so that you can recreate it later. Also realize that even "rm -r" may take a long time to remove that many messages; if it appears to be hung, check your "df -v" in another window to verify that files really are being removed.

You can find "large" directories by

find / -type d -size +5 -print

Large directories may be suspect; if you can remove and recreate them, do so (this may require temporarily moving the contents elsewhere).