Quick and simple web log grepper in Perl

Some material is very old and may be incorrect today

© December 2010 Anthony Lawrence

I often want quick answers to very simple questions. How many pageviews and unique visitors were there for a certain page this month or yesterday?

Sure, Google Analytics can tell me that, though Analytics won't count visitors with Javascript disabled and sometimes misses even then. More importantly, if I don't already have it open, it takes several steps to get what I want.

Grepping raw log files always works, and is fast, but a Perl program is effective and gives me what I usually want very quickly.

You might need o have to adjust how the log is read - mine get compressed daily so I have to zcat them to get what I want.

Usage is simple enough: countuniq arg1 arg2 ...

Example "countuniq sample.html 16/Dec"

This ignores any GET of jpg's etc. Of course you could modify this to add all kinds of switches and features, but for me, it's a daily tool that is quick and simple.

#!/usr/bin/perl
chdir("/home/cluster1/data/g/q/1863165/logs") or die "No logs!";
@stuff=`zcat tr*.gz | cat - trans*[0-9]`;

READING: foreach (@stuff) {
 next if not /html *HTTP/;
 $line=$_;
 foreach (@ARGV) {
   $arg=$_;
   next READING if ( $line !~ /$arg/i);
}
 #print;
 $pviews++;
 s/ .*//;
 $uniques{$_}++;
}

$args=join " ", @ARGV;
print  "$args:  Unique IP: " . scalar (keys %uniques);
print  " Pageviews:  $pviews \n";
 

If you found something useful today, please consider a small donation.

Got something to add? Send me email.

---

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us