It was a good run

As I explained in more detail at A Panda Post Mortem, this website has fallen out of favor with Google. Once hovering in the top 30-40,000 sites all across the Internet, it is now falling into obscurity.

The reason it was once popular was because it really was the best site for SCO Unix information. As SCO declined, I knew this site would suffer. I tried to transition to Mac and Linux, but there are far better places to get that and eventually Google stopped its long term love affair with my scribblings.

I now find that Google-Plus and HubPages are more effective places for me to write.

I'm not shutting this site down. I'll keep it running until traffic stops outright and every now and then there may be something that makes more sense here than in those other places. That won't be often, though.

So - it was a good run. From 1997 to now I have really enjoyed many of the conversations we have had here. Those conversations don't have to stop, of course: some of you are already exchanging ideas with me at G+ and HubPages. But this site will be rather sleepy going forward.

Thank you all again and I wish you a happy life. Come join me at those other places if you like and if not, maybe we will still see each other here every now and then.

{EAV:76aa44a27ff33312}

Group projects with Google Docs

SATA RDX backup cartridge report for BackupEDGE

I have been testing an internal SATA RDX drive, made by Quantum, in an IBM x-series 226 server with dual 3.0 Ghz CPU's, 3 GB of RAM, and dual 73 GB SCSI drives configured as a simple RAID 1. The Operating System is Red Hat Enterprise Linux 5.3, Microlite BackupEDGE version 03.00.03 with the RDX cartridge configured as a file system partition. I modified the retention time to be only 3 days, as I was backing up to the same cartridge, and wanted to verify that the program would automatically delete the oldest archive to make room. When we have a cartridge for each workday, I will extend that back out to 2 weeks.

   Backup performance :

SUMMARY - BACKUP
Serial Number          = demo1011
Date                   = Tue Nov 22 01:04:12 2011
Files Encountered      = 288530
Total Data             = 31.60GB
Data Written           = 31.87GB
Volume Left            = 114.53MB
Segments Used          = 32
Elapsed Time           = 00:34:04
Data Transfer Speed    = 93.838 GB/hr
                      = 1601.513 MB/min
                      = 27988485 bytes/sec
Exit Status            = 0

   Verification :

SUMMARY - BYTE-BY-BYTE VERIFICATION
Serial Number          = demo1011
Date                   = Tue Nov 22 01:40:13 2011
Segments Used          = 32
Data Read              = 31.88GB
Elapsed Time           = 00:35:58
Data Transfer Speed    = 54.419 GB/hr
                      = 928.756 MB/min
                      = 16231202 bytes/sec
Files Encountered      = 288530
Files Excluded         = 4
Files Modified         = 7
Files Not Checked      = 2
Special Files          = 21031
Verified Successfully  = 267488
Change Log             = /usr/lib/edge/lists/simple_job//changedfiles_system_mas
ter.txt
Status                 = No problems found
Exit Status            = 0

The time to backup on our current Iomega REV internal SCSI drive, with a similar amount of data, was 36:52, and the verify was 36:25. Admittedly, the current backup happens when the system has live user activity, but the users can really notice the slowdown when the backup/verify is active. I did login to the test system while the backup was running to the RDX drive, and it did not seem to slow down as much. This was probably due to the SATA connection being separate from the SCSI backplane for the RDX drive, whereas the Iomega REV is also a SCSI device. Setting up the RDX cartridge as a SharpDrive device was slightly slower, somewhere between 1 and 2 minutes slower overall.

Where the RDX drive really shines though, is on a Selective Restore. In essence, the RDX cartridge is a hard drive, so the access time for individual files is extremely fast. I restored one of our largest files (782 MB) :

SUMMARY - RESTORE
Serial Number          = demo1011
Date                   = Thu Nov 03 16:08:53 2011
Elapsed Time           = 00:00:40
Data Transfer Speed    = 109.793 GB/hr
                      = 1875.469 MB/min
                      = 32776199 bytes/sec
Segments Used          = 10
Data Read              = 782.69MB
Files Restored         = 1
Exit Status            = 0

Forty seconds is excellent. The same restore from our REV drive took over 28 minutes.

Performance aside, the fact that Iomega no longer makes the REV drives is what is making this decision for me. Also, the cost per Gigabyte on the cartridges is extremely low (320 GB for $112.75), and we are not limited to a cartridge capacity going forward. It will also be nice to have multiple week's worth of data on each cartridge. If a user was to mess up one of their files, and not realize it for a week or two, we would have them covered.

Elapsed Time           = 00:34:04
Data Transfer Speed    = 93.838 GB/hr
                       = 1601.513 MB/min
                       = 27988485 bytes/sec
Exit Status            = 0
   Verification :
SUMMARY - BYTE-BY-BYTE VERIFICATION
Serial Number          = demo1011
Date                   = Tue Nov 22 01:40:13 2011
Segments Used          = 32
Data Read              = 31.88GB
Elapsed Time           = 00:35:58
Data Transfer Speed    = 54.419 GB/hr
                       = 928.756 MB/min
                       = 16231202 bytes/sec

Capture and report (Bash Scripting)

You've been asked to copy some jpg files to a USB disk overnight. That's easy enough - a cron job and a simple "cp -a" will do that.

But there is so much that could go wrong, isn't there? There might not be any files to copy or there might not be room on the USB disk. Somebody might have changed permissions on files or directories alreasdy in place, preventing overwrite with updated images.

Maybe your first thought was just to send all the output to a file that might get printed or mailed for someone to look at. That's easy:

#!/bin/bash
cd /Pictures 
cp -av *jpg /TINYUSB > /tmp/report 2>&>1

Though that could be a mighty long report and if nothing went wrong, who cares?

You could just capture the error output.

#!/bin/bash
cd /Pictures
cp -av *jpg /TINYUSB 2> /tmp/report 

That's better, but it doesn't have the "-v" output. Isn't there a way to get both?

Yes, there is, although it does involve a little more work. The script that follow does that and a few other checks.

#!/bin/bash
# these lines just make sure we are in the right place
cd /Pictures || echo "`date` Copy script can't find /Pictures!!" | lp
cd /Pictures || exit 1
ls *.jpg > /tmp/report
# if there is nothing to copy, /tmp/report will say so
# if there are files, /tmp/report is nulled out and will be empty
# make sure TINYUSB is mounted
/bin/mount | grep TINYUSB || echo "TINYUSB not mounted!" | lp
/bin/mount | grep TINYUSB || exit 1
for i in *.jpg
do
  cp -av "$i" /TINYUSB > /tmp/failed 2>&1
  # /tmp/failed will have both -v and error output
  # but we'll only copy it if there really was an error
  case $?  in
   0) : ;;
   *) echo "Exit code was $?" >> /tmp/failed;
      echo " " >> /tmp/failed;
      echo "`date` Copy of $i failed" > /tmp/message; 
      cat /tmp/message /tmp/failed >> /tmp/report;;
  esac
  grep -q "No space left on device" /tmp/report && lp /tmp/report 
  grep -q "No space left on device" /tmp/report && exit 1
  # no reason to go on if there is no space
done
test -s /tmp/report && lp /tmp/report
# if no errors. nothing will print because /tmp/report will still be empty

The "$?" in the case switch will be the exit code of the last cp command. If all went well, it will be 0 and we do nothing. If it's not 0, we'll add /tmp/failed to /tmp/report.

If the device has run out of space, there is no reason to keep trying, so we'll quit the script then and there.

There's no need to do that "cd /Pictures" or the "grep -q" twice.

  grep -q "No space left on device"   /tmp/report && { lp /tmp/report ;exit 1; }

Spaces after "{" and before "}" are necessary and that ";" after "exit 1" is also needed.

However, doing it twice makes it very obvious what you want to happen and why and saves you from forgetting the spaces or the semicolon.

There are probably other things that can go wrong that I haven't thought of today. If you know one, let me know and we'll add it.

Kerio Connect Archive Script

Yesterday I mentioned testing how well Kerio Connect's new automatic reindexing responds to deliberate abuse. It turns out that it responds well, so I took advantage of that to create an automatic archiving script.

This is a simple need that I am sure Kerio will add someday. The problem is that some users simply will not ever clean up their INBOX. As it grows larger, the system slows down because it has to search the #msgs directory for free slots. Large directories just bog things down; archiving prevents that.

As written here, this should run once a month. It creates dated subfolders of INBOX and moves older email into these. Here i used the "ctime" to decide age, but you could also open each message and use internal dates.

You can see the result here:

The 2011_11_16 folder was by this script created from older messages in INBOX. When it runs next month, it will create another subfolder of INBOX.

As it can take time for the automatic index checking to take place, I have this set to run at 1 minute past midnight on the 1st of the month. By the time I get up in the morning, the reindexing should be long done.

This script is provided without guarantee, warranty or anything else. Use at your own risk and check results carefully. IF YOU DON'T UNDERSTAND THE SCRIPT, YOU PROBABLY SHOULDN'T USE IT!

This script depends apon the background reindexing features introduced in Kerio Connect 7.3 and will not work on earlier versions. It may not work on later versions because of structural changes to Kerio..

Finally, this was tested on a Linux system. Perl can run on Windows but you would need to change the script.

I use the temporary directory with the rename to avoid the reindexer finding messages before everything is copied into place.. this way, it all appears at once.

#!/usr/bin/perl
$KERIO="/opt/kerio/mailserver/store/mail/aplawrence.com/";
$SAFE="/root/kdir";  # put it wherever you like
@t=localtime(time()-(3600 * 24)); # go back one day

chdir($KERIO);
foreach (<*>) {
  next if /#public$/;
  # add anyone else you want to skip
  $who=$_;
  print "$_\n";
$dir=sprintf("%4d_%2d_%2d",$t[5]+1900,$t[4]+1,$t[3]);
exit 1 if (-e "$KERIO/$who/INBOX/$dir");
# already exists
if (not -e "$SAFE/$who") {
  mkdir("$SAFE/$who") or exit 1;
}
if (not -e "$SAFE/$who/$dir") {
  mkdir("$SAFE/$who/$dir") or exit 1;
  mkdir("$SAFE/$who/$dir/#msgs") or exit 1;
  mkdir("$SAFE/$who/$dir/#assoc") or exit 1;
}
chdir ("$KERIO/$who/INBOX/");
foreach (<*.fld>) {
  print "$who $_\n";
  system("/bin/cp $_ $SAFE/$who/$dir");
  # for windows   system("copy  $_ $SAFE/$who/$dir");
}
chdir("#msgs");
$rightnow=time();
foreach(<*.eml>) {
  @stats=stat $_;
  $howold=($rightnow - $stats[10]) / (3600 * 24); # days old
  next if $howold < 7;
  print "$who $_ $howold\n";
  rename $_,"$SAFE/$who/$dir/#msgs/$_";
}
rename "$SAFE/$who/$dir","$KERIO/$who/INBOX/$dir";
}

A more complex script could archive by the correspondent or anything else you might glean from the message content. Your imagination is the only limit.

Kerio Connect Automatic Index Maintenance

The latest Kerio Connect Mailserver release includes both on demand and automatic reindexing of users mail folders. I decided to abuse that and see what happens.

To make a proper mess for the indexing to clean up, I first created a temporary folder and then used "scp" to copy a good sized INBOX into it.

mkdir  ~/safehold
cd  /opt/kerio/mailserver/store/mail/aplawrence.com/tony/INBOX
scp  -r  .  ~/safehold

Now it was time to mess things up:

rm   ~/safehold/?mgs/*
cd /opt/kerio/mailserver/store/mail/aplawrence.com/tony/INBOX/?msgs/
for i in 0000605*.eml; do mv $i ~/safehold/?msgs/; done
mv ~/safehold ./TESTBOX

What all that did was create a new mailbox called TESTBOX that only had 16 messages in it, but had indexes that were remembering many thousand. It also rudely removed those same 16 messages from the original INBOX.

So what did Kerio see when I logged in?

It thought TESTBOX had thousands of messages in it, but it couldn't actually find any of them. It also didn't know that 16 messages had left the INBOX without permission.

However, just a few minutes later it had found the 16 messages, though it was still confused on the left about the count. As I watched it over several minutes, the count slowly decreased.

I decided to see what would happen if I asked for a manual rebuild. You simply go to Users and, under More Actions, you'll find the option to Rebuild Mailbox. Oddly, that logs to the Config log (I would have thought Operations made more sense, but it didn't take very long.

Once complete (which took only seconds) , Webmail showed everything correctly. I wasn't done messing with it yet, though.

cd /opt/kerio/mailserver/store/mail/aplawrence.com/tony/INBOX/?msgs/
find . -mtime +15 -exec rm {} \;

That removed some 5,000 messages from right under Kerio's nose!

As before, it took a while for it to notice. The first hint was that if I tried to open one of the oldest messages, it came up blank.

Then whole pages disappeared. It still thought it had 39 pages of messages, but some were blank.

After less than half an hour, all was well.

One final test - I removed 6 of the .eml files from TESTBOX/#msgs after logging out of webmail. I wanted to see if they background process would notice the missing files if I were not logged in.

My reason for that was that I wanted to see if it would be possible to rearrange a users mailbox during the night and have them see the result when they logged in the next day.

It was, although it took a while for it to happen.

# ls -l
total 584
drwx------ 2 root root   4096 2011-11-15 08:37 #assoc
-rw------- 1 root root 185994 2011-11-15 08:59 deleted.fld
-rw------- 1 root root   1723 2011-11-15 09:01 index.fld
drwx------ 2 root root 315392 2011-11-15 11:02 #msgs
-rw------- 1 root root   8192 2011-11-15 08:36 properties.fld
-rw------- 1 root root  59392 2011-11-15 09:01 search.fld
-rw------- 1 root root    158 2011-11-15 09:01 status.fld

Ninety minutes later I checked again:

drwx------ 2 root root   4096 2011-11-15 08:37 #assoc
-rw------- 1 root root 185994 2011-11-15 08:59 deleted.fld
-rw------- 1 root root   1084 2011-11-15 12:21 index.fld
drwx------ 2 root root 315392 2011-11-15 11:02 #msgs
-rw------- 1 root root   8192 2011-11-15 08:36 properties.fld
-rw------- 1 root root  44032 2011-11-15 12:21 search.fld
-rw------- 1 root root    158 2011-11-15 12:21 status.fld

It seems that the quickest way is a manual rebuild, that being active in the account will fix things up slowly, and that the background process will get around to fixing things eventually.

So, given this experiment, it seems possible that we could run a night time process that rearranged messages and have everything be fine in the morning. This could be useful for sorting out Inbox messages by month, for example,

I'll do a little sample code soon and we'll see how that works.

Mysterious Duplicate IP problem solved

There is an IP address conflict with another system on the network.

I only needed to be run over by four or five cluetrains before realizing what was causing this message on a Windows 2003 server. Follow along and see when you solve it.

Here's what was going on:

A Windows 2003 server would sometimes complain that its IP was in use on the network. This would often happen after a power failure.. yes, they had a UPS, but it died and they never got around to replacing it as power failures seem to be very rare in this particular area and the server isn't all that critical. It's a mail and file server; if the power is out the lights are out and nobody cares too much about mail or files right then. So..

It would not always complain that its IP was in use. That was actually an important clue, so if you are trying to solve this before I tell you what it was, chew on that while I blabber on.

Of course the very first thing we did was shut the server off and go looking for that IP. We found nothing. We looked on the DHCP servers to see if they were perhaps sometimes handing that IP out to someone else; they were not.

Being suspicious of switches, I swapped those out with some spares. No change.

Sometimes we could clear it by disabling the nic card, counting to twenty and reenabling. That didn't always work, though.

The clues

What did work was disconnecting it from the network, booting it up, counting to the magic twenty once more and then plugging it back in. That became the official procedure for handling this. As I said, power failures are rare, so this was a once or twice a year thing.

That's a good clue also, by the way.

The really big clue was this: if you'd patiently wait an hour or so, the problem would fix itself. That's a BIG clue.

Also, if you happened to shut down the machine midday, it would never see this problem. That's the same clue as the last, if you think about it.

After the fourth or fifth time that this happened, I did catch the clues and was able to cure it in a whopping three or four seconds. Have you guessed it yet?

Two more clues for you

Here, go read this page about arp. Do you see it now? No? OK. one more clue: read my "Editors note" at the top of this old Netgear PrintServer PS110 article. You should have it now.

Actually, it was when I happened to be editing that article for a typo that the clues all came together in my head and I knew what was freaking out that Windows server.

Spoiler alert

Stop reading now if you want to think about this more before I tell you.

I'll add a little white space so you don't just slide on into it against your will..

Arp

It was manual arp addresses hardcoded into an old SCO box on the network.

At one time, there was a network print server that used the IP address we were looking for. The printer was long gone, but the manual setting was still in the SCO startup scripts.

So.. if the SCO machine rebooted before the Windows server, it had an arp entry for that IP and when the slower Windows box did its "who has?" broadcast, the SCO would helpfully provide the IP of that non-existent printer.

If the SCO machine did not restart for whatever reason, the Windows machine saw no response to its inquiry and happily used the IP we wanted it to use.

As the arp protocol specifies, if the non-existent host stays non-existent long enough (which it obviously would in this case), the machine holding the cache should forget about it.

You can see now how all the clues made perfect sense. I logged into the SCO box, found the rc2.d script that caused this problem, deleted it and that was that. Problem solved.

Book Review - Michal Zalewski's 'The Tangled Web'