Zimbra

LMStone · #1 (**permalink**) 07-17-2007, 09:21 PM

/tmp/logswatch.out and /tmp/zmlogger.out are at 1.3GB and 750MB in size, respectively.

Are these files required to be kept around or can I periodically delete them?

Will Zimbra recreate them if they are not there? For example, is this OK:

zmcontrol stop
rm /tmp/logswatch.out
rm /tmp/zmlogger.out
zmcontrol start

Thanks!
Mark

jholder · #2 (**permalink**) 07-17-2007, 09:23 PM

did you apply the patch for solving the bug for the logger stopping?

If so, you may wish to revert back

john

LMStone · #3 (**permalink**) 07-17-2007, 09:28 PM

Quote:

Originally Posted by jholder

did you apply the patch for solving the bug for the logger stopping?

If so, you may wish to revert back

john

Hi John,

I did indeed apply the patch, but didn't think the patch would survive the upgrade to 4.5.6 from 4.5.5.

I'll go back out the patch, and then may I assume it is safe to delete the files after stopping Zimbra?

Thanks John,
Mark

jholder · #4 (**permalink**) 07-17-2007, 09:33 PM

hmm. good point.

I'm gonna have to ask Brian to chime in on this.

LMStone · #5 (**permalink**) 07-17-2007, 09:43 PM

Quote:

Originally Posted by jholder

hmm. good point.

I'm gonna have to ask Brian to chime in on this.

OK, so I backed out the patch but haven't run 'zmlogswatchctl restart' yet.

Is this helpful?:

viognier:/opt/zimbra/bin # ls -al zmlogsw*
-rwxr-xr-x 1 root root 2019 Jul 17 23:36 zmlogswatchctl
-rwxr-xr-x 1 root root 2019 Jul 17 23:36 zmlogswatchctl.456
-rwxr-xr-x 1 root root 2019 Jun 7 10:10 zmlogswatchctl.orig
viognier:/opt/zimbra/bin # diff zmlogswatchctl.456 zmlogswatchctl
viognier:/opt/zimbra/bin #

viognier:/opt/zimbra/libexec # ls -al zmlogprocess*
-rwxr-xr-x 1 root root 37108 Jul 17 23:35 zmlogprocess
-rwxr-xr-x 1 root root 37365 Jul 17 23:35 zmlogprocess.456
-rwxr-xr-x 1 root root 37108 Jun 7 10:10 zmlogprocess.orig
viognier:/opt/zimbra/libexec # diff zmlogprocess.456 zmlogprocess
554,559d553
< $sth = sqlExec("delete from disk_aggregate where ".
< "period_start='".$periodStart."' and ".
< "period_end='".$periodEnd."' and ".
< "device='".$dev."' and ".
< "host='".$host."'");
<
viognier:/opt/zimbra/libexec #

viognier:/opt/zimbra/libexec # ls -al zmlogger*
-rwxr-xr-x 1 root root 4448 Jul 17 23:35 zmlogger
-rwxr-xr-x 1 root root 6065 Jul 17 23:34 zmlogger.456
-rwxr-xr-x 1 root root 4448 Jun 7 10:11 zmlogger.orig
-rwxr-xr-x 1 root root 3871 Jun 27 20:05 zmloggerinit
-rwxr-xr-x 1 root root 1936 Jun 27 20:05 zmloggerprune
viognier:/opt/zimbra/libexec # diff zmlogger.456 zmlogger
29,31c29
< my $zimbra_home = "/opt/zimbra";
<
< use lib "${zimbra_home}/zimbramon/lib";
---
> use lib "/opt/zimbra/zimbramon/lib";
37,39c35,37
< #$SIG{QUIT} = \&quit(0, "Received sigQUIT");
< #$SIG{KILL} = \&quit(0, "Received sigKILL");
< #$SIG{INT} = \&quit(0, "Received sigINT");
---
> my $baseDir = "/opt/zimbra";
>
> my $data_source="dbi:mysql:database=zimbra_logger;mys ql_read_default_file=/opt/zimbra/conf/my.logger.cnf;mysql_socket=/opt/zimbra/logger/db/mysql.sock";
41,42d38
< my $debug = 1;
< my $data_source="dbi:mysql:database=zimbra_logger;mys ql_read_default_file=${zimbra_home}/conf/my.logger.cnf;mysql_socket=${zimbra_home}/logger/db/mysql.sock";
44c40
< my $password = `${zimbra_home}/bin/zmlocalconfig -s -m nokey zimbra_logger_mysql_password`;
---
> my $password = `/opt/zimbra/bin/zmlocalconfig -s -m nokey zimbra_logger_mysql_password`;
46,47d41
< my $platform=`${zimbra_home}/libexec/get_plat_tag.sh`;
< chomp $platform;
49c43,54
< my $dbh = dbConnect($data_source,$username,$password);
---
> my $platform=`/opt/zimbra/libexec/get_plat_tag.sh`;
> chomp $platform;
> my $timeout=0;
> while (! isSqlRunning() && ($timeout <= 120) ) {
> $timeout += sleep 5;
> }
> my $dbh = DBI->connect($data_source, $username, $password);
>
> if (!$dbh) {
> print STDERR "DB: Can't connect to $data_source: $DBI::errstr\n";
> exit 1;
> }
52,64c57,69
< 'Jan' => '01',
< 'Feb' => '02',
< 'Mar' => '03',
< 'Apr' => '04',
< 'May' => '05',
< 'Jun' => '06',
< 'Jul' => '07',
< 'Aug' => '08',
< 'Sep' => '09',
< 'Oct' => '10',
< 'Nov' => '11',
< 'Dec' => '12'
< );
---
> 'Jan' => '01',
> 'Feb' => '02',
> 'Mar' => '03',
> 'Apr' => '04',
> 'May' => '05',
> 'Jun' => '06',
> 'Jul' => '07',
> 'Aug' => '08',
> 'Sep' => '09',
> 'Oct' => '10',
> 'Nov' => '11',
> 'Dec' => '12'
> );
65a71,81
> sub logdate_to_sqldate {
> my $ts = shift;
> if ($ts eq "" || !defined($ts)) {
> return '';
> }
> # Sep 14 18:30:02
>
> my @p = split (' ', $ts);
> my @s= localtime();
> my $y = $s[5]+1900;
> my $m = $MON{$p[0]};
67,69c83,85
< my $statement =
< "insert into raw_logs(log_date, loghost, app, pid, msg, postfix_qid) values (?,?,?,?,?,?)";
< my $sth = $dbh->prepare($statement);
---
> my $dt = sprintf ("%4d-%02d-%02d %s",$y,$m,$p[1],$p[2]);
> return $dt;
> }
71,73d86
< my $status_statement =
< "insert into service_status(server, service, time, status, loghostname) values (?,?,?,?,?)";
< my $s_sth = $dbh->prepare($status_statement);
74a88,89
> sub get_last_timestamp {
> my $ts;
76,79c91
< open FOO, ">/tmp/zmlogger.out" if $debug;
< select FOO;
< $|=1;
< logMsg("INFO", "zmlogger starting up");
---
> my $statement = "select max(time) from server_stat";
81c93
< my $logregex = qr/(^.{15}) ((\d+\.\d+\.\d+\.\d+) \S+|(\S+)) ([^[]+)\[(\d+)\]: (.*)$/o;
---
> my $sth = $dbh->prepare($statement);
83,84c95,98
< while (<>) {
< next if (/last message repeated/);
---
> if (!$sth->execute) {
> Zimbra::Logger::Log ("err", "DB: $sth->errstr");
> return undef;
> }
86,95c100
< # attempt to reconnect if mysql dropped
< if (!$dbh->ping) {
< logMsg("DEBUG", "DB Connection dropped. Attempting to reconnect");
< undef $dbh;
< $dbh = dbConnect($data_source,$username,$password);
<
< # prepare the statements with the new connection
< $sth = $dbh->prepare($statement);
< $s_sth = $dbh->prepare($status_statement);
< }
---
> my @ary = $sth->fetchrow_array;
97,99c102,104
< # parse the input
< my ($log_date, $host, $ip, $name, $app, $pid, $msg) = ($_ =~ m/$logregex/);
< $host = (($ip ne "") ? $ip : $name);
---
> return (datetime_to_timestamp($ary[0]));
>
> }
101,145c106,107
< # skip slapd and master
< if ($app eq 'slapd' || $app eq 'master') {
< logMsg("DEBUG", "Skipping update for app $app");
< next;
< } else {
< logMsg("DEBUG", "hostname:$host app:$app name:$name log_date:$log_date");
< }
<
< next if ($host eq "");
< my $qid = undef;
<
< $qid = $1 if ($app =~ /^postfix/ && $msg =~ /^(\S{8,12}): /);
<
< if ($msg =~ /info: ([^,]+), STATUS: (\S+): (\S+): (Running|Stopped).*$/) {
< my $tm = $1;
< my $hostname = $2;
< my $service = $3;
< my $status = ($4 eq "Running")?1:0;
<
< # delete previous entries
< my $statement = "delete from service_status ".
< "where server=\'".$hostname."\' and service=\'".$service."\'";
< my $dsth = $dbh->prepare($statement);
< if (!$dsth->execute() ) {
< logMsg("ERROR", $dsth->errstr);
< next;
< }
< # insert new status entries
< if (!$s_sth->execute($hostname, $service, $tm, $status, $host)) {
< logMsg("ERROR", "Status update failed for service:$service host:$host status:$status");
< logMsg("ERROR", $s_sth->errstr);
< } else {
< logMsg("INFO", "Status update for service:$service host:$host status:$status tm:$tm");
< }
< next;
< }
<
< if (!$sth->execute(logdate_to_sqldate($log_date), $host, $app, $pid, $msg, $qid) ) {
< logMsg("ERROR", "Error inserting $log_date, $host, $app, $pid, $msg");
< logMsg("DEBUG", "From $_");
< logMsg("DEBUG", "DBI errstr: ",$sth->errstr);
< #logMsg("ERROR", "zmlogger insert failed $sth->errstr");
< } else {
< logMsg("INFO", "zmlogger insert completed: $host $app $pid $msg $qid");
< }
---
> my $statement =
> "insert into raw_logs(log_date, loghost, app, pid, msg, postfix_qid) values (?,?,?,?,?,?)";
147,148c109
< }
< quit(1, "Loop failed this should never be reached");
---
> my $sth = $dbh->prepare($statement);
150c111,112
< # Sub Routines
---
> my $status_statement =
> "insert into service_status(server, service, time, status, loghostname) values (?,?,?,?,?)";
152,155c114
< sub isSqlRunning {
< `${zimbra_home}/bin/logmysqladmin status > /dev/null 2>&1`;
< return(($? == 0) ? 1 : 0);
< }
---
> my $s_sth = $dbh->prepare($status_statement);
157,188c116
< sub quit($$) {
< my ($rc,$msg) = @_;
< logMsg("INFO", "$msg");
< logMsg("INFO", "zmlogger shutting down");
< close FOO if $debug;
< exit $rc;
< }
<
< sub logMsg {
< my ($lvl, $msg) = @_;
< print FOO localtime(time), " $msg\n" if $debug;
< print "$msg\n" if ($lvl eq "INFO" || $lvl eq "ERROR");
< }
<
< sub dbConnect($$$) {
< my ($db_src,$db_user,$db_pass) = @_;
< my $timeout=0;
<
< while (! isSqlRunning() && ($timeout <= 3600) ) {
< logMsg("INFO", "database not running. retrying connection");
< $timeout += sleep 5;
< }
< quit(1, "zmlogger database isn't running") unless isSqlRunning();
<
< my $dbh = DBI->connect($db_src,$db_user,$db_pass);
< $dbh->{PrintError} = 0 unless $debug;
< $dbh->{RaiseError} = 0 unless $debug;
<
< if (!$dbh) {
< logMsg("ERROR", "DB: Can't connect to $db_src: $DBI::errstr");
< quit(1, "Can not connect to database $DBI::errstr");
< }
---
> my $DEBUG = 0;
190,191c118
< return $dbh;
< }
---
> ($DEBUG) && open FOO, ">/tmp/zmlogger.out";
193,203c120
< sub logdate_to_sqldate {
< my $ts = shift;
< if ($ts eq "" || !defined($ts)) {
< return '';
< }
< # Sep 14 18:30:02
<
< my @p = split (' ', $ts);
< my @s= localtime();
< my $y = $s[5]+1900;
< my $m = $MON{$p[0]};
---
> ($DEBUG) && print FOO localtime()," zmlogger starting up\n";
205,207c122
< my $dt = sprintf ("%4d-%02d-%02d %s",$y,$m,$p[1],$p[2]);
< return $dt;
< }
---
> $SIG{QUIT} = \&bye;
208a124
> my $logregex = qr/(^.{15}) ((\d+\.\d+\.\d+\.\d+) \S+|(\S+)) ([^[]+)\[(\d+)\]: (.*)$/o;
210,211c126,127
< sub get_last_timestamp {
< my $ts;
---
> while (<>) {
> (/last message repeated/) && next;
213c129,130
< my $statement = "select max(time) from server_stat";
---
> my ($log_date, $host, $ip, $name, $app, $pid, $msg) = ($_ =~ m/$logregex/);
> $host = (($ip ne "") ? $ip : $name);
215c132
< my $sth = $dbh->prepare($statement);
---
> if ($app eq 'slapd' || $app eq 'master') {next;}
217,220c134,135
< if (!$sth->execute) {
< Zimbra::Logger::Log ("err", "DB: $sth->errstr");
< return undef;
< }
---
> if ($host eq "") { next;}
> my $qid = undef;
222c137,184
< my @ary = $sth->fetchrow_array;
---
> if ($app =~ /^postfix/ && $msg =~ /^(\S{8,12}): /) {
> $qid = $1;
> }
>
> if ($msg =~ /info: ([^,]+), STATUS: (\S+): (\S+): (Running|Stopped).*$/) {
> my $tm = $1;
> my $hostname = $2;
> my $service = $3;
> my $status = ($4 eq "Running")?1:0;
> my $statement = "delete from service_status ".
> "where server=\'".$hostname."\' and service=\'".$service."\'";
>
> my $dsth = $dbh->prepare($statement);
>
> #print "Executing $statement with @args\n\n";
>
> eval {
> if (!$dsth->execute() ) {
> warn $dsth->errstr;
> next;
> }
> };
>
>
> eval {
> $s_sth->execute($hostname, $service, $tm, $status, $host);
> };
>
> next;
> }
>
> eval {
> if (!$sth->execute(logdate_to_sqldate($log_date), $host,
> $app, $pid, $msg, $qid) ) {
> ($DEBUG) && print FOO localtime(),
> " zmlogger insert failed: ",$sth->errstr,"\n";
> } else {
> #($DEBUG) && print FOO localtime(),
> #" zmlogger insert completed: $host $app $pid $msg $qid\n";
> }
>
> };
> if ($@) {
> print "Error inserting $log_date, $host, $app, $pid, $msg\n";
> print "From $_\n";
> print $sth->errstr,"\n";
> print "$@\n";
> }
224,225d185
< return (datetime_to_timestamp($ary[0]));
<
226a187,198
> ($DEBUG) && close FOO;
>
> sub isSqlRunning {
> `logmysqladmin status > /dev/null 2>&1`;
> return(($? == 0) ? 1 : 0);
> }
>
> sub bye {
> ($DEBUG) && close FOO;
> exit (0);
> }
>
viognier:/opt/zimbra/libexec #

All the best,
Mark

brian · #6 (**permalink**) 07-17-2007, 10:11 PM

The patch was mostly rolled into 4.5.6 but unfortunately the debug mode got left on in a couple of the scripts so these files will continue to be large. 4.5.7 will lower the logging level back to the normal level.

4.5.7 will also move the files from /tmp to /opt/zimbra/log and they will be part of the regular log rotation. In the meantime it is safe to truncate the files after stopping logger. They will be recreated upon restart.

Code:

% zmlogswatchctl stop
% rm -f /tmp/zmlogger.out /tmp/logswatch.out
% zmlogswatchctl start

LMStone · #7 (**permalink**) 07-18-2007, 07:00 AM

Quote:

Originally Posted by brian

The patch was mostly rolled into 4.5.6 but unfortunately the debug mode got left on in a couple of the scripts so these files will continue to be large. 4.5.7 will lower the logging level back to the normal level.

4.5.7 will also move the files from /tmp to /opt/zimbra/log and they will be part of the regular log rotation. In the meantime it is safe to truncate the files after stopping logger. They will be recreated upon restart.

Code:

% zmlogswatchctl stop
% rm -f /tmp/zmlogger.out /tmp/logswatch.out
% zmlogswatchctl start

Thanks Brian for the clarification on this.

I put the 4.5.6 files back in place instead of the 4.5.5 files (some debug code) and ran the above three commands just fine.

All the best,
Mark