Lightsquid Stop Working in 2021

Publish on: 2021-01-03 Classify at: Linux Read:≈ 5min Views: Comments:

Lightsquid is a handy log analyzer for squid proxy. However, the project is not maintained since 2009.

Today, I found lightsquid doesn’t work in 2021. Why?

Lightsquid Error

When I run lightparser, the output looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
$ ./lightparser.pl access.log.66666
>>> use file :: /var/log/squid/access.log.66666
run TIME: 1 sec
LightSquid parser statistic report

                 3312 lines processed (average 3312.00 lines per second)
                    0 lines parsed
                    0 lines recovered
                    0 lines notrecovered
                    0 lines skiped by bad year
                 3312 lines skiped by date filter
                    0 lines skiped by Denied filter
                    0 lines skiped by skipURL filter

WARNING !!!!, parsed 0 lines from total : 3312
please check confiuration !!!!
may be wrong log format selected ?

Seems that all lines are filtered by the date filter. I double checked the log format and make sure it is correct and the same with previous days.

Analyze the Problem

Look at this piece of code (I am a Perl layman): here

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
my $filterdatestop =timelocal(59,59,23,31,12-1,2020-1900)+1000;

$fToday=1 if ($ARGV[0] eq "today");
$fToday=1 if ($ARGV[0] eq "yesterday");

if ($fToday) {
   ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;

   $filterdate=sprintf("%04d%02d%02d",$year+1900,$mon+1,$mday);;
   $filterdatestart=timelocal( 0, 0, 0,$mday,$mon,$year);
   $filterdatestop =timelocal(59,59,23,$mday,$mon,$year);
   print ">>> filter today: $filterdate\n" if ($debug);
}

if ($ARGV[0] eq "yesterday") {
   $filterdatestart=$filterdatestart-(24*60*60);
   $filterdatestop =$filterdatestop -(24*60*60);
   ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($filterdatestart);
   $filterdate=sprintf("%04d%02d%02d",$year+1900,$mon+1,$mday);;
   print ">>> filter yesterday: $filterdate\n" if ($debug);
}

if ($ARGV[0] =~ m/^(\d\d\d\d)(\d\d)(\d\d)$/) {
   $filterdate=$ARGV[0];
   $filterdatestart=timelocal( 0, 0, 0,$3,$2-1,$1);
   $filterdatestop =timelocal(59,59,23,$3,$2-1,$1);
   print ">>> filter date:	$filterdate\n" if ($debug);
}

if ($ARGV[0] =~ m/access\.log\.(\d)/) {
   $filename=$ARGV[0];
   $catname="zcat" if ($ARGV[0] =~ m/\.gz$/);
   $catname="bzcat" if ($ARGV[0] =~ m/\.bz2$/);
}

print ">>> use file :: $logpath/$filename\n" if ($debug);
...

	   #speed optimization for FILTERDATE mode
	   $Ltimestamp=substr $_,0,11;
	   if ($Ltimestamp<$filterdatestart or $Ltimestamp>$filterdatestop) {
		  print ">>>> skipDafteFilter URL $Lurl\n$_" if ($debug2 >= 2 );
		  $skipfilterdatecntr++;
		  next;
	   };

Found two issues:

A hardcode 2020 in $filterdatestop which makes lightsquid not work in 2021
today argument cannot work with customized file at the same time

First Issue

Obviously, $filterdatestop is the culprit:

1
my $filterdatestop =timelocal(59,59,23,31,12-1,2020-1900)+1000;

A quick fix is to change the hardcoded 2020 to 2100. It seems that the month and year parameter are weird…

I read the document of Perl’s “Time::Local” and found that the design is really complex and hard to understand (personally I think it is anti-pattern, why not make it a simple design which supports only absolute year?):

Years greater than 999 are interpreted as being the actual year, rather than the offset from 1900. Thus, 1964 would indicate the year Martin Luther King won the Nobel prize, not the year 3864.
Years in the range 100..999 are interpreted as offset from 1900, so that 112 indicates 2012. This rule also applies to years less than zero (but see note below regarding date range).
Years in the range 0..99 are interpreted as shorthand for years in the rolling “current century,” defined as 50 years on either side of the current year. Thus, today, in 1999, 0 would refer to 2000, and 45 to 2045, but 55 would refer to 1955. Twenty years from now, 55 would instead refer to 2055. This is messy, but matches the way people currently think about two digit dates. Whenever possible, use an absolute four digit year instead.

Then I try to understand what these lines means:

1
2
3
4
5
6
7
8
9
use Time::Local;

my $date1 = timelocal(59,59,23,31,12-1,2020-1900);
my $date2 = timelocal(59,59,23,31,12-1,2020-1900)+1000;
my $date3 = timelocal(59,59,23,31,11,2020);

print "$date1\n";
print "$date2\n";
print "$date3\n";

Output:

The solution is easy, change this line to(note that the month is zero indexed (0..11)), and now it works until 2100:

1
my $filterdatestop =timelocal(59,59,23,31,11,2100)+1000;

Second Issue

Another problem is that today argument cannot work with customized filename:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$fToday=1 if ($ARGV[0] eq "today");
$fToday=1 if ($ARGV[0] eq "yesterday");

if ($fToday) {
...
}

if ($ARGV[0] eq "yesterday") {
...
}

if ($ARGV[0] =~ m/^(\d\d\d\d)(\d\d)(\d\d)$/) {
   $filterdate=$ARGV[0];
   $filterdatestart=timelocal( 0, 0, 0,$3,$2-1,$1);
   $filterdatestop =timelocal(59,59,23,$3,$2-1,$1);
   print ">>> filter date:	$filterdate\n" if ($debug);
}

if ($ARGV[0] =~ m/access\.log\.(\d)/) {
   $filename=$ARGV[0];
   $catname="zcat" if ($ARGV[0] =~ m/\.gz$/);
   $catname="bzcat" if ($ARGV[0] =~ m/\.bz2$/);
}

print ">>> use file :: $logpath/$filename\n" if ($debug);

Apparently, only one argument ARGV[0] is applied. If you wrote the command like this and not fixed the first issue:

`1`	`$./lightparser.pl access.log.66666 today`

`1`	`$./lightparser.pl today access.log.66666`

You will encounter the “parsed 0 lines” error.

Summary

Just fix the first issue and you will make lightsquid works longer :-)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
$ ./lightparser.pl access.log.66666
>>> use file :: /var/log/squid/access.log.66666
>>> Make Report 20210102 (1577 - log line parsed)
>>> Make Report 20210103 (1572 - log line parsed)
run TIME: 1 sec
LightSquid parser statistic report

                 3312 lines processed (average 3312.00 lines per second)
                 3149 lines parsed
                    0 lines recovered
                    0 lines notrecovered
                    0 lines skiped by bad year
                    0 lines skiped by date filter
                  163 lines skiped by Denied filter
                    0 lines skiped by skipURL filter

Fixed version: https://github.com/finisky/lightsquid-1.8.1