Checking DNS Black List Usage on Kerio Connect


2013/03/01

Have you ever wondered how effective your blacklists are?  How about other Spamassassin tests - how many of your messages match tests like "MISSING_HEADERS" or anything else you might be interested in?

This Perl script gathers that information.  Like all of my scripts, it is rough and crude, but it can quickly (well, quickly depending on the size of your mail store) slice and dice Spam headers and spit out counts for your amusement.

I have my spam folders set to clean out automatically regularly, and there is only my own mailbox and my wife's to count, but this is a partial list of what I got this morning:


ADVANCE_FEE_2_NEW_MONEY:  2
ADVANCE_FEE_3_NEW:  1
ADVANCE_FEE_3_NEW_MONEY:  1
..
DNSBL_B.BARRACUDACENTRAL.ORG:  160
DNSBL_BL.SPAMCOP.NET:  9
DNSBL_CBL.ABUSEAT.ORG:  2
DNSBL_DB.WPBL.INFO:  22
DNSBL_DNSBL-1.UCEPROTECT.NET:  17
DNSBL_DNSBL-2.UCEPROTECT.NET:  18
DNSBL_DNSBL-3.UCEPROTECT.NET:  2
DNSBL_DNSBL.SORBS.NET:  73
DNSBL_TRUNCATE.GBUDB.NET:  84
DNSBL_ZEN.SPAMHAUS.ORG:  34
..
UNPARSEABLE_RELAY: 34
URIBL_DBL_SPAM: 280
URIBL_JP_SURBL: 150
URIBL_PH_SURBL: 2
URIBL_RHS_DOB: 20
URIBL_SBL: 5
URIBL_SC_SURBL: 8
URIBL_WS_SURBL: 17
URI_HEX: 7
URI_NOVOWEL: 1
URI_NO_WWW_INFO_CGI: 3
US_DOLLARS_3: 2
WEIRD_QUOTING: 1
 

While that's interesting, I was really more interested in just the DNSBL entries.  Those are the blacklists I use and while I could just "grep" them out of this list, I was also curious to know how often mail ended up on multiple blacklists, so I wrote a different version of this to tell me that.

That version produces a list like this:

B.BARRACUDACENTRAL.ORG : 160
B.BARRACUDACENTRAL.ORG/DNSBL-1.UCEPROTECT.NET/ : 4
BL.SPAMCOP.NET : 9
BL.SPAMCOP.NET/DNSBL.SORBS.NET/ : 1
CBL.ABUSEAT.ORG : 2
DB.WPBL.INFO : 22
DB.WPBL.INFO/B.BARRACUDACENTRAL.ORG/ : 2
DB.WPBL.INFO/B.BARRACUDACENTRAL.ORG/DNSBL-1.UCEPROTECT.NET/ : 1
DB.WPBL.INFO/DNSBL-1.UCEPROTECT.NET/ : 2
DB.WPBL.INFO/DNSBL-2.UCEPROTECT.NET/ : 1
DB.WPBL.INFO/DNSBL-3.UCEPROTECT.NET/ : 1
DNSBL-1.UCEPROTECT.NET : 17
DNSBL-1.UCEPROTECT.NET/DNSBL-2.UCEPROTECT.NET/ : 1
DNSBL-2.UCEPROTECT.NET : 18
DNSBL-3.UCEPROTECT.NET : 2
DNSBL.SORBS.NET : 73
DNSBL.SORBS.NET/DNSBL-1.UCEPROTECT.NET/ : 3
TRUNCATE.GBUDB.NET : 84
TRUNCATE.GBUDB.NET/B.BARRACUDACENTRAL.ORG/ : 34
TRUNCATE.GBUDB.NET/DB.WPBL.INFO/ : 1
TRUNCATE.GBUDB.NET/DNSBL-2.UCEPROTECT.NET/ : 11
TRUNCATE.GBUDB.NET/ZEN.SPAMHAUS.ORG/ : 1
ZEN.SPAMHAUS.ORG : 34
ZEN.SPAMHAUS.ORG/B.BARRACUDACENTRAL.ORG/ : 19
ZEN.SPAMHAUS.ORG/B.BARRACUDACENTRAL.ORG/CBL.ABUSEAT.ORG/ : 1
ZEN.SPAMHAUS.ORG/B.BARRACUDACENTRAL.ORG/DNSBL-2.UCEPROTECT.NET/ : 1
ZEN.SPAMHAUS.ORG/B.BARRACUDACENTRAL.ORG/DNSBL-3.UCEPROTECT.NET/ : 1

What this tells me is that (for example), the GDUDB blacklist matched 84 of my mail messages. Of those 84, 34 were also matched by BARRACUDACENTRAL and a handful by other DNSBL's.  The ABUSEAT blacklist was only found twice, and one of those was unique (no other blacklist was triggered).  That tells me that I might be wise to increase the spam score I assign for ABUSEAT and GDUDB.  I'd want to check for any false positives first, of course, and run this same code against my INBOX to see if anything would become Spam because of increases, but this can help me fine-tune my filtering.

Here is the code for the first version:

#!/usr/bin/perl
# Tony Lawrence, aplawrence.com/Kerio 2/28/2013
# This parses and totals all spam tests
$DEBUG=0;
$KERIOCFG="/opt/kerio/mailserver/mailserver.cfg";
$KERIOSTORE="/opt/kerio/mailserver/store/mail/";
$folder="'Junk E-mail'";
# Change to INBOX or whatever to check other folders

chdir($KERIOSTORE) or die "No $KERIOSTORE $!";
foreach $domain (<*>) {
 foreach $user (<$domain/*>) { 
# if you have a very large number of users or messages, this glob will break 
   foreach $eml (<$user/$folder/#msgs/*.eml>) {
     open(EML,"<:crlf",$eml);
     $inspam=0;
     $messages++;
     while (<EML>) {
      last if /X-Spam-Flag:/;
      $inspam=1 if /X-Spam-Status:/;
      next if /X-Spam-Status/;
      next if not $inspam;
      chomp;
      s/\t//g;
      print if $DEBUG;
      s/tests=//;
      s/TOTAL_SCORE.*//;
      @s=split/,/;
      foreach (@s) {
        s/:.*//;
        next if not $_;       
        $spam{$_}++; 
      }
      }
     }
  }
}
foreach (sort keys %spam) {
   print "$_:  $spam{$_}\n";
}
 

That's the version that puts out every spam test. This next version only looks at blacklists, although you can add specific other tests easily as shown in the code.

#!/usr/bin/perl
# Tony Lawrence, aplawrence.com/Kerio 2/28/2013
$DEBUG=0;
$KERIOCFG="/opt/kerio/mailserver/mailserver.cfg";
$KERIOSTORE="/opt/kerio/mailserver/store/mail/";
$folder="'Junk E-mail'";
# Change to INBOX or whatever to check other folders
open(I,"<:crlf","$KERIOCFG") or die "No $KERIOCFG $!";

$foundlists=0;
while (<I>) {
 chomp;
 $foundlists=1 if /<list name="DnsBlacklists"/;
 next if not $foundlists;
 last if ($foundlists and /<\/list>/);
 next if not /<variable name="Domain">/;
 s/<variable.*>(.*)<.variable.*//;
 push(@blacklists,uc($_));
}
# push @blacklists, "MIME_HTML_MOSTLY";
# push @blacklists,"MISSING_HEADERS";
# push @blacklists,"URIBL";
# add any other Spam matches you'd like to count

chdir($KERIOSTORE) or die "No $KERIOSTORE $!";
foreach $domain (<*>) {
 foreach $user (<$domain/*>) { 
# if you have a very large number of  messages, this glob will break 
   foreach $eml (<$user/$folder/#msgs/*.eml>) {
     open(EML,$eml);
     $inspam=0;
     $messages++;
     while (<EML>) {
      last if /X-Spam-Flag:/;
      $inspam=1 if /X-Spam-Status:/;
      next if not $inspam;
      print if $DEBUG;
      $line=$_;
      foreach  (@blacklists) {
        s/ +//g;
        if ($line =~ /$_/) {
          $blcount{$_}++ ;
          $bluniq{$_}=1;
        }
      }
     }
    $multiples="";
    $mcount=0;
    foreach(@blacklists) {
      s/ +//g;
      $multiples .= "$_/"  if $bluniq{$_};
      $mcount++ if $bluniq{$_};;
    }
    $blcount{$multiples}++ if ($mcount > 1);
    foreach (@blacklists) {
      s/ +//g;
      $bluniq{$_}=0;
    }
   }
  }
}

foreach (sort keys %blcount) {
  print "$_ : $blcount{$_}\n";
}
 


Got something to add? Send me email.





(OLDER) <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> -> Checking DNS Black List Usage on Kerio Connect




Increase ad revenue 50-250% with Ezoic


More Articles by

Find me on Google+

© Anthony Lawrence



Kerio Connect Mailserver

Kerio Samepage

Kerio Control Firewall

Have you tried Searching this site?

Unix/Linux/Mac OS X support by phone, email or on-site: Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us





The difference between e-mail and regular mail is that computers handle e-mail, and computers never decide to come to work one day and shoot all the other computers. (Jamais Cascio)

All this modern technology just makes people try to do everything at once. (Bill Watterson)








This post tagged: