APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

A Tale of Backup

2008/10/09

I've had my share of customers who won't make backups. I've had customers who have left the same media in the backup device for years on end.. sometimes those even end up with happy endings that the customer definitely did not deserve.

I've had customers whose backups failed for months on end but nobody every looked at the printed report or email to know that.

I've had customers mis-label backups, overwrite critical full backups with unimportant files, expose media to strong magnetic fields, misplace media.. I've probably seen just about everything, including having too much backup, but this one leaves me slack jawed:

I was called yesterday morning to go see a customer I had last seen sometime in 2001. They weren't my direct customer; I had helped a Synchronics dealer install a small Linux server for them. I remember having to go back once for some malfunctioning print servers, but that was it.. never heard from them again.

Flash forward seven years. The original Synchronics guy has been replaced by another and the customer has been chugging along. Suddenly on a Thursday night he gets error messages on the screen, system reboots and is stuck single user. Customer calls his Synchronics guy, he calls his Linux guy, who has a miserable cold and tells him to call me. He does, and I agree to pop in at 7:30 AM to see what I can see.

As I expected, it wanted a manual fsck. I started one, but immediately ran into multiple sector read errors. Uh-oh, that's not good. I pulled the covers off to see if maybe cables were loose or if the drive was hot.. nope, nothing like that. Just a failed drive.

But hey, no big whoop because we have DVD-RAM backup. In fact, the stack of Monday through Saturday labeled DVD's was right there on the desk. Even better, there was an extra, completely unconnected drive in the machine. My vague memory was that the original intent was to have mirrored drives but we had run into some silly problem, not mirrored, and just left the drive there. Fantastic, we should have this puppy purring along in just a jiffy..

But the customer was giving me a strange look. I swear it was the "I'm the one who tried to get a cookie and spilled everything on the floor" look you'd get from a five year old.

"I haven't done backups in a while", he said sheepishly.

I'm sure I did a cartoon character double-take. Microlite Edge runs from cron, all he had to do was put in a DVD before leaving for the night and take it out in the morning. Not exactly difficult.

"How long?", I asked. Longish pause, and he said "A while. Years." and shrugged his shoulders helplessly. I thought to myself that it's a good thing he owns the business, because if he were an employee he'd be in a lot of trouble.. ok, it is what it is.

"I'm going to pull the drives and take a couple of your backups and we'll see what I can do", I said. I figured I'd take both drives, maybe restore the OS to the unused one, mount the problem drive and see what I could copy.

When I got back to my office I first attached the drive to my Linux server, and mounted it. At first I thought I was home free because it didn't even complain that the filesystem was dirty and as I scouted around a few directories the data looked good. Unfortunately, when I went to the /usr/syn directory, it all fell apart: hard read errors galore. I shut down and relayed the bad news to the Synchronics guy.

"Well, maybe not so bad", he said. "The customer says he doesn't really care about current tickets and all, he just doesn't want to have to re-enter all his part numbers. Those should be on the backups even if they are old."

Okey-dokey. One smallish problem: these were the old-style cartridge DVD-RAM's. I don't have that kind of drive anymore so I'd have to break open the case and pull out the DVD. I did that and tried reading it with Microlite Edge - it read through the /dev files and failed dead with an I/O error. Not good.. but maybe I carelessly scratched the disk when I broke the case, so I tried again. Success! I read off /usr/syn and then repacked it in a standard tar file. The newest file was from 2006 - almost three years old.

So that's where we stand. Synchronics guy says he'd like to switch the customer to Windows. Fine, I'm not going to argue with him. I'll put the files on a Windows readable CD and he can do whatever he likes..

Of course he could send the drive out to a data recovery firm. That can be pricey, and my impression was that the customer isn't interested in spending money.

It would have been so simple if he'd just put the DVD's in the drive every night.



Got something to add? Send me email.





(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> -> No matter how easy you make it some people won't do backups


4 comments



Increase ad revenue 50-250% with Ezoic


More Articles by

Find me on Google+

© Anthony Lawrence







Thu Oct 9 02:03:03 2008: 4631   BruceGarlock


How about trying 'SpinRite' on it? I have had that save a few drives for me now. Well worth the price, and then some. (link) -- of course it wont fix physical issues, but if there are bad sectors, it does a great job. I have had many drives not be able to be repaired with 'fsck' on Linux machines come back to life with SpinRite. It has also fixed many Windows drives when 'chkdsk' couldn't fix the errors. It's a great "toolkit" item.

- Bruce Garlock



Thu Oct 9 11:48:48 2008: 4635   TonyLawrence

gravatar
Yes, I've used Spinrite in the past - excellent product.



Thu Oct 9 13:29:29 2008: 4636   BruceGarlock


Plus, it's coded in assembly! Man, that's just crazy! I really do envy *any* assembly language programmer. My hats off to Steve Gibson :-) Not to mention, he turned down the sale of spinrite to one of the big guys (I think Norton). Can you image how bloated spinrite would have been if someone like Norton got their grubby paws on it?

- Bruce Garlock



Mon Oct 13 02:45:06 2008: 4644   badanov


Backups are a total waste of time and resources. Even if you make then yourself. If you contract someone else to do then they are a horrific drain on cash and resources...

Until the moment you need backups, and then they are worth every dime you ever put in them.

Backups are by their very nature a thankless job. One of the worst sales you can actually make to a customer is a contract to develop and maintain a backup system.

------------------------
Kerio Samepage


Have you tried Searching this site?

Unix/Linux/Mac OS X support by phone, email or on-site: Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us





I can’t go to a restaurant and order food because I keep looking at the fonts on the menu. (Donald Knuth)

The people I distrust most are those who want to improve our lives but have only one course of action in mind. (Frank Herbert)












This post tagged: