APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

How do I compress my backups?

© August 2013 Tony Lawrence

Well, you could just run the output of tar, cpio, or whatever through compress but if even one bit of your tape or diskette goes bad, you could lose the rest of the backup. Not recommended at all, unless of course you don't actually care if your backups work - but if you didn't care, you wouldn't be doing any, right?

Linux and other more modern versions of tar include a compress flag, but that may be no different than compressing the archive itself: individual files are not compressed, just the whole archive.

There's also the obvious limitation that you can't grab a particular file from that tape. You'd need to uncompress the whole thing to get at one file.

Read the man pages carefully. For example, GNU tar warns:

About corrupted compressed archives: compressed files have no redundancy, for maximum compression. The adaptive nature of the compression scheme means that the compression tables are implicitly spread all over the archive. If you lose a few blocks, the dynamic construction of the compression tables becomes unsynchronized, and there is little chance that you could recover later in the archive.

That should convince you that this is a bad idea.

A better solution might be afio or a third-party product like Microlite BackupEDGE (which offers much more than just compression)...

But do you even need to do this? Your tape drive probably does hardware compression anyway, so this may be unimportant. It definitely will take more time. You still should look at better backup than tar or cpio though.

Of course if it just won't fit..

bzip2, bzip2 and so on

Compression used to be a lot more important. In these days of cheap 100 GB hard drives and fast internet connections, I don't find myself using compression much. However, just about anything you download will be compressed, and if it isn't gzippped, it's probably uses bzip2, which can provide greater compression for some files.

As high speed internet becomes more common, I wonder when people will stop bothering to compress files? It already makes little sense for relatively small files. Disk space is so cheap now and high speed Internet is so common - I almost never care about compression.

Although I can't recall having this problem in recent years, compressed files can be damaged. With gzipped files, the man page says that "zcat" will read them up to the point of the damage.

Bzip2 does a little better. Because bzip2 writes its compressed data in specific sized blocks, it is possible to recover parts of a file even beyond where it has been damaged. Of course that may do you little good, but sometimes that could help.

This newsgroup post by Bela Lubkin explains the relationship between various file archivers and how they work.. He mentions compressgzip, bzip2, zip, pack and 7-zip.

Also see Best Linux backup software.

Got something to add? Send me email.

(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> How do I compress my backups?

Inexpensive and informative Apple related e-books:

Take Control of OS X Server

Take Control of iCloud

Photos: A Take Control Crash Course

Take Control of Numbers

El Capitan: A Take Control Crash Course

More Articles by © Tony Lawrence

Printer Friendly Version

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us

Printer Friendly Version

Lawyer — One who protects us against robbers by taking away the temptation. (H.L. Mencken)

Linux posts

Troubleshooting posts

This post tagged:


Unix/Linux Consultants

Skills Tests

Unix/Linux Book Reviews

My Unix/Linux Troubleshooting Book

This site runs on Linode