A bit level verify is where you read back a tape (or whatever you backed up to and compare the files one by one (usually by a "sum" or similar idea, but sometimes byte by byte).
You'd probably be surprised how often a bit level verify proves useful.
As many consultants do, I receive backup status email for several of my clients, so I can tell you that bit-level verification "failures" ( failure of one or more files to match the hard drive- not a complete backup failure) are not all that uncommon. Once in a while it's due to incipient tape failure, and it's very helpful to see a problem developing before it gets serious- if we're lucky the really important files aren't the ones the tape had trouble with.
Quite often the failure is due to unexpected activity on the system during the backup. Usually it's explainable, but a couple of times in my long history it has been an employee up to no good after hours and the bit-level verify helped show what they were doing.
Once or twice the bit-level verify has alerted us to incipient hard drive failures. Those usually would have been spotted by other means too, but it never hurts to have multiple channels of alerts. As someone said, if it's REALLY important that you get up at 4:00 AM tomorrow, you set two alarm clocks AND have the hotel desk call you.
Now and then we get "cosmic ray" failures. It's probably really just a wayward piece of dust finding its way to the tape, but the point is that one file fails to verify and we can't see any reason why. We always get suspicious of the tape of course, but sometimes it just goes away and the tape continues on for months. Just because of the odds, such failures are usually an unimportant or easily recreated file, but once in a great while it hits something very important, and that's when we're very happy to have the knowledge that, if we should need it, Tuesday's tape is NOT suitable for restoration because that important file is suspect. The point here of course is the KNOWLEDGE: we KNOW there is a problem on this tape. We probably don't need the tape (not planning on trashing the system today) but if we DID need it, we know that it has a problem.
And yes, we often do other things too. For convenience and redundancy I often have rsync or similar things tucking important bits of data here and there around the network. On really important systems we have more than one system that does tape or dvdram backup and lots of cross pollenization between the systems- it doesn't cost a lot to do this kind of thing and the peace of mind is much improved.
Another way to look at it: what value is there in a backup you aren't certain is accurate? Sure, sometimes it doesn't matter, but sometimes it matters a lot. Trust, but verify. Bit level verify, of course.
Got something to add? Send me email.
More Articles by Tony Lawrence © 2013-08-09 Tony Lawrence