Examine Kerio Connect User Spam Activity


2012/09/29

Users will always complain about spam. No matter what you do, either some message they don't like will end up in their inbox or something they did want will get put in their spam folder. You can't avoid that, but knowing more about what the users actually do can help you fine tune your spam settings so that you (hopefully) get less complaints.

It's not easy to examine the emails that users are complaining about now. You can find entries in the Operations log that say things like:


[25/Sep/2012 08:21:17][27869] {spam} Spam Filter: Message
 /opt/kerio/mailserver/store/mail/aplawrence.com/tony/
INBOX/2012_08_31/#msgs/0000e44d.eml was marked as SPAM by the user

25/Sep/2012 07:48:06] {MOVE} Protocol: HTTP/WebMail, User:
[email protected], IP: 96.237.161.229, Folder:
[email protected]/INBOX/2012_08_31, Destination Folder:
[email protected]/Junk E-mail, From: "VuvuPlaza"
<[email protected]>, Subject: "Getaway deals / Black Ops 2 /
Tickets / Swedish Massage", Msg-Id:
<<[email protected]>,
 

But then you still have to go track down the messages. That's time consuming, to say the least!

Note: the script that is presented here depends upon users dragging messages to the Junk Email folder rather than just marking them as Spam. If the "Spam" button is used as it was in the first message in the Operations log above, the message is deleted. It's not moved to Deleted Items; it is really deleted and is therefore unavailable to this script.

There are also some obvious privacy concerns to be noted when examining user's email.

A script to extract the relevant files

Fortunately, dragging messages does not interfere with Spam headers. That gives us a fairly simple way to infer complaints:

If we find a message in INBOX that has "X-Spam-Status: Yes" in its headers, it was probably dragged there from Junk EMail. I say "probably" only because that header could have been added by some other system; that would be an unusual case.

On the other hand, if we find a message in Junk Email that has "X-Spam-Status: No" in its headers, that was almost certainly dragged there from INBOX.

Easy enough, right? A simple Perl script can gather those up for us.

#!/usr/bin/perl
use File::Basename;
$KERIOSTORE="/opt/kerio/mailserver/store";
$KERIOMAIL="/opt/kerio/mailserver/store/mail/aplawrence.com";
$zipdir="/tmp";
$TEMPDIR="$zipdir/sendthese";
$COPY="cp";
$COUNTER=1;
$now=time();
($day,$month,$year)=(localtime)[3,4,5];
$year+=1900;
$zipfile="$zipdir/$year$month$day.tgz";
$ZIP="tar -acf $zipfile $TEMPDIR";
unless(-d $TEMPDIR){
    mkdir $TEMPDIR or die "Couldn't create TEMPDIR: [$TEMPDIR] ($!)";
    open(T,">$TEMPDIR/lastran");
    $now=time();
    print T $now;
    close T;
}
chdir("$KERIOSTORE/logs") or die "Wrong $KERIOSTORE !";
open(T,"$TEMPDIR/lastran");
$now=<T>;
open(T,">$TEMPDIR/lastran");
$now=time();
print T $now;
close T;
$now-=(96 * 3600);
foreach(<debug*>) {
  next if /.idx/;
  @s=stat($_);
  next  if ($s[9] < $now);
  system("$COPY $_ $TEMPDIR/");
}
chdir("$KERIOMAIL/") or die "Wrong $KERIOMAIL !";
foreach (<*>) {
  $user=$_;
  check("INBOX","X-Spam-Status: Yes","NOTJUNK");
  check("'Junk E-mail'","X-Spam-Status: No","ISJUNK");
}
system($ZIP);
#print "$ZIP\n";
@s=stat("$zipfile");
if ($s[7] > 4000000) {
   chdir $zipdir;
   system("split -db 4k $zipfile $zipfile.part.");
   # in case we want to email these somewhere
}
   
  

sub check {
  $folder=shift;
  $pattern=shift;
  $storage=shift;
  #print "$user $folder $pattern $storage\n";
  foreach(<$user/$folder/#msgs/*>) {
     #print "$_\n";
    
     $message=$_;
     @s=stat($_);
     next  if ($s[9] < $now);
     open(I,$_);
     $bad=0;
     while (<I>) {
       $bad=1 if /$pattern/i;
       
       last if $bad;
       last if /X-Spam-Status/i;
       chomp;
       last if not $_;
     }
  close I;
  if ($bad) {
   $base=basename($message);
   system("$COPY '$message' '$TEMPDIR/$COUNTER.$user.$storage.$base'");
   print "$COPY '$message' '$TEMPDIR/$COUNTER.$user.$storage.$base'";
   $COUNTER++;
  }
  }
}

Although this script only examines the INBOX, it's possible that a user might drag to or from some other folder. You could certainly expand this to check all user folders rather than just INBOX.

We are using a variant of this to help debug some spam problems at a 170 user site - you can imagine how difficult that would be without a helper script like this.

Another version

This version reads the Operations log to find moved messages, and thereby looks in all domains and all subfolders.

#!/usr/bin/perl
use Net::FTP;
$DEBUG=1; # change to 0 when fixed
$KERIOSTORE="/cygdrive/m/Program Files/Kerio/mailserver/store";
$KERIOLOGS="/cygdrive/m/Program Files/Kerio/mailserver/store/logs";
$KERIOMAIL="'/cygdrive/m/Program Files/Kerio/mailserver/sendmail.exe'";
$KERIOCONFIG="/cygdrive/m/Program Files/Kerio/mailserver/";
$zipdir="/cygdrive/c/tmp/zipped";
$TEMPDIR="/cygdrive/c/tmp/sendthese";
#$KERIOSTORE="/opt/kerio/mailserver/store";
#$KERIOLOGS="/opt/kerio/mailserver/store/logs";
#$KERIOMAIL="'/opt/kerio//mailserver/sendmail.exe'";
$zipdir="/cygdrive/c/tmp/zipped";
$TEMPDIR="/cygdrive/c/tmp/sendthese";
$COPY="cp";
$now=time();
$mymess=0;
$zipfile="$zipdir/$now.tar";
$ZIP="tar -acf $zipfile $TEMPDIR";
unless(-d $TEMPDIR){
    mkdir $TEMPDIR or die "Couldn't create TEMPDIR: [$TEMPDIR] ($!)";
    open(T,">$TEMPDIR/lastran");
    $now=time();
    print T $now;
    close T;
}
mkdir $zipdir if not -d $zipdir;
open(T,"$TEMPDIR/lastran");
$now=<T>;
open(T,">$TEMPDIR/lastran");
$now=time();
print T $now;
close T;
$now-=(72 * 3600);
chdir("$KERIOLOGS");
foreach(<debug*>) {
  next if /.idx/;
  system("$COPY $_ $TEMPDIR/");
}
foreach(<spam*>) {
  next if /.idx/;
  system("$COPY $_ $TEMPDIR/");
}
system("cp $KERIOCONFIG/mailserver.cfg $TEMPDIR");

chdir $zipdir or die;
chdir $zipdir or exit;
system("rm -r $zipdir/* $TEMPDIR/*/*JUNK/*");
findmoves();
system($ZIP);

sub findmoves {
open(I,"$KERIOLOGS/operations.log");
while (<I>) {
 next if not /[{]MOVE[}]/;
 @s=split /,/;
 $destination="";
 $source="";
 $messageid="";
 foreach(@s) {
   s/^ *//;
   $source=$_ if /^Folder:/;
   $destination=$_ if /^Destination Folder:/;
   $messageid=$_ if /^Msg-Id:/;
 }
 if ($destination =~ /Junk E-mail/ or $source =~ /Junk E-mail/ and $destination !~ /Deleted Items/) {
   push @these,"$destination||$messageid||$source";
   
 }
}
foreach (sort @these) {
  @stuff=split /\|\|/;
  $d=$stuff[0];$s=$stuff[2];$i=$stuff[1];
  $d=~s/^Destination Folder: ~//;
  $domain=$d;
  $domain=~ s/.*\@//;
  $domain=~ s/\/.*//;
  $user=$d;
  $user=~s/\@.*//;
  $folder=$d;
  $folder=~s/^$user\@$domain\///;
  $mid=$i;
  $mid=~s/Msg-Id://;
  chdir("$KERIOSTORE/mail/$domain/$user/$folder/#msgs") or print "What $domain /$user/$folder!\n";
  chdir("$KERIOSTORE/mail/$domain/$user/$folder/#msgs") or next;

  foreach(<*.eml>) {
    $file=$_;
    @s=stat($_);
    next  if ($s[9] < $now);
    open(M,"$_");
    while(<M>) {
      next if not /Message-ID:/;
      print "$domain $user/$folder $file $_" if $_ =~ /$mid/;
    $storage= "NOTJUNK";
    $storage= "ISJUNK" if $folder =~ /Junk E-Mail/; 
    unless(-d "$TEMPDIR/$user"){
      mkdir "$TEMPDIR/$user" or die "Couldn't create TEMPDIR: [$TEMPDIR/$_] ($!)";
    }
    unless(-d "$TEMPDIR/$user/$storage"){
      mkdir "$TEMPDIR/$user/$storage" or die "Couldn't create TEMPDIR: [$TEMPDIR/$_] ($!)";
    }
   system("$COPY '$file' $TEMPDIR/$user/$storage");
   $mymessages=sprintf("%0.8x",$mymess++);
      
      last;
    } 
   }
}
}




Got something to add? Send me email.





(OLDER) <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> -> Examine Kerio Connect User Spam Activity




Increase ad revenue 50-250% with Ezoic


More Articles by

Find me on Google+

© Anthony Lawrence



Kerio Connect Mailserver

Kerio Samepage

Kerio Control Firewall

Have you tried Searching this site?

Unix/Linux/Mac OS X support by phone, email or on-site: Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us





C++: NSStringWithoutYourMothersHatButWithNiceCandyFrosting - aargh! (Tony Lawrence)

Computers are useless. They can only give you answers. (Pablo Picasso)








This post tagged: