Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: logswatch.out and zmlogger.out BIG!

  1. #1
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,373
    Rep Power
    10

    Default logswatch.out and zmlogger.out BIG!

    /tmp/logswatch.out and /tmp/zmlogger.out are at 1.3GB and 750MB in size, respectively.

    Are these files required to be kept around or can I periodically delete them?

    Will Zimbra recreate them if they are not there? For example, is this OK:

    zmcontrol stop
    rm /tmp/logswatch.out
    rm /tmp/zmlogger.out
    zmcontrol start

    Thanks!
    Mark

  2. #2
    jholder's Avatar
    jholder is offline Former Zimbran
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    did you apply the patch for solving the bug for the logger stopping?

    If so, you may wish to revert back

    john

  3. #3
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,373
    Rep Power
    10

    Default

    Quote Originally Posted by jholder View Post
    did you apply the patch for solving the bug for the logger stopping?

    If so, you may wish to revert back

    john
    Hi John,

    I did indeed apply the patch, but didn't think the patch would survive the upgrade to 4.5.6 from 4.5.5.

    I'll go back out the patch, and then may I assume it is safe to delete the files after stopping Zimbra?

    Thanks John,
    Mark

  4. #4
    jholder's Avatar
    jholder is offline Former Zimbran
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    hmm. good point.

    I'm gonna have to ask Brian to chime in on this.

  5. #5
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,373
    Rep Power
    10

    Default

    Quote Originally Posted by jholder View Post
    hmm. good point.

    I'm gonna have to ask Brian to chime in on this.
    OK, so I backed out the patch but haven't run 'zmlogswatchctl restart' yet.

    Is this helpful?:

    viognier:/opt/zimbra/bin # ls -al zmlogsw*
    -rwxr-xr-x 1 root root 2019 Jul 17 23:36 zmlogswatchctl
    -rwxr-xr-x 1 root root 2019 Jul 17 23:36 zmlogswatchctl.456
    -rwxr-xr-x 1 root root 2019 Jun 7 10:10 zmlogswatchctl.orig
    viognier:/opt/zimbra/bin # diff zmlogswatchctl.456 zmlogswatchctl
    viognier:/opt/zimbra/bin #

    viognier:/opt/zimbra/libexec # ls -al zmlogprocess*
    -rwxr-xr-x 1 root root 37108 Jul 17 23:35 zmlogprocess
    -rwxr-xr-x 1 root root 37365 Jul 17 23:35 zmlogprocess.456
    -rwxr-xr-x 1 root root 37108 Jun 7 10:10 zmlogprocess.orig
    viognier:/opt/zimbra/libexec # diff zmlogprocess.456 zmlogprocess
    554,559d553
    < $sth = sqlExec("delete from disk_aggregate where ".
    < "period_start='".$periodStart."' and ".
    < "period_end='".$periodEnd."' and ".
    < "device='".$dev."' and ".
    < "host='".$host."'");
    <
    viognier:/opt/zimbra/libexec #

    viognier:/opt/zimbra/libexec # ls -al zmlogger*
    -rwxr-xr-x 1 root root 4448 Jul 17 23:35 zmlogger
    -rwxr-xr-x 1 root root 6065 Jul 17 23:34 zmlogger.456
    -rwxr-xr-x 1 root root 4448 Jun 7 10:11 zmlogger.orig
    -rwxr-xr-x 1 root root 3871 Jun 27 20:05 zmloggerinit
    -rwxr-xr-x 1 root root 1936 Jun 27 20:05 zmloggerprune
    viognier:/opt/zimbra/libexec # diff zmlogger.456 zmlogger
    29,31c29
    < my $zimbra_home = "/opt/zimbra";
    <
    < use lib "${zimbra_home}/zimbramon/lib";
    ---
    > use lib "/opt/zimbra/zimbramon/lib";
    37,39c35,37
    < #$SIG{QUIT} = \&quit(0, "Received sigQUIT");
    < #$SIG{KILL} = \&quit(0, "Received sigKILL");
    < #$SIG{INT} = \&quit(0, "Received sigINT");
    ---
    > my $baseDir = "/opt/zimbra";
    >
    > my $data_source="dbi:mysql:database=zimbra_logger;mys ql_read_default_file=/opt/zimbra/conf/my.logger.cnf;mysql_socket=/opt/zimbra/logger/db/mysql.sock";
    41,42d38
    < my $debug = 1;
    < my $data_source="dbi:mysql:database=zimbra_logger;mys ql_read_default_file=${zimbra_home}/conf/my.logger.cnf;mysql_socket=${zimbra_home}/logger/db/mysql.sock";
    44c40
    < my $password = `${zimbra_home}/bin/zmlocalconfig -s -m nokey zimbra_logger_mysql_password`;
    ---
    > my $password = `/opt/zimbra/bin/zmlocalconfig -s -m nokey zimbra_logger_mysql_password`;
    46,47d41
    < my $platform=`${zimbra_home}/libexec/get_plat_tag.sh`;
    < chomp $platform;
    49c43,54
    < my $dbh = dbConnect($data_source,$username,$password);
    ---
    > my $platform=`/opt/zimbra/libexec/get_plat_tag.sh`;
    > chomp $platform;
    > my $timeout=0;
    > while (! isSqlRunning() && ($timeout <= 120) ) {
    > $timeout += sleep 5;
    > }
    > my $dbh = DBI->connect($data_source, $username, $password);
    >
    > if (!$dbh) {
    > print STDERR "DB: Can't connect to $data_source: $DBI::errstr\n";
    > exit 1;
    > }
    52,64c57,69
    < 'Jan' => '01',
    < 'Feb' => '02',
    < 'Mar' => '03',
    < 'Apr' => '04',
    < 'May' => '05',
    < 'Jun' => '06',
    < 'Jul' => '07',
    < 'Aug' => '08',
    < 'Sep' => '09',
    < 'Oct' => '10',
    < 'Nov' => '11',
    < 'Dec' => '12'
    < );
    ---
    > 'Jan' => '01',
    > 'Feb' => '02',
    > 'Mar' => '03',
    > 'Apr' => '04',
    > 'May' => '05',
    > 'Jun' => '06',
    > 'Jul' => '07',
    > 'Aug' => '08',
    > 'Sep' => '09',
    > 'Oct' => '10',
    > 'Nov' => '11',
    > 'Dec' => '12'
    > );
    65a71,81
    > sub logdate_to_sqldate {
    > my $ts = shift;
    > if ($ts eq "" || !defined($ts)) {
    > return '';
    > }
    > # Sep 14 18:30:02
    >
    > my @p = split (' ', $ts);
    > my @s= localtime();
    > my $y = $s[5]+1900;
    > my $m = $MON{$p[0]};
    67,69c83,85
    < my $statement =
    < "insert into raw_logs(log_date, loghost, app, pid, msg, postfix_qid) values (?,?,?,?,?,?)";
    < my $sth = $dbh->prepare($statement);
    ---
    > my $dt = sprintf ("%4d-%02d-%02d %s",$y,$m,$p[1],$p[2]);
    > return $dt;
    > }
    71,73d86
    < my $status_statement =
    < "insert into service_status(server, service, time, status, loghostname) values (?,?,?,?,?)";
    < my $s_sth = $dbh->prepare($status_statement);
    74a88,89
    > sub get_last_timestamp {
    > my $ts;
    76,79c91
    < open FOO, ">/tmp/zmlogger.out" if $debug;
    < select FOO;
    < $|=1;
    < logMsg("INFO", "zmlogger starting up");
    ---
    > my $statement = "select max(time) from server_stat";
    81c93
    < my $logregex = qr/(^.{15}) ((\d+\.\d+\.\d+\.\d+) \S+|(\S+)) ([^[]+)\[(\d+)\]: (.*)$/o;
    ---
    > my $sth = $dbh->prepare($statement);
    83,84c95,98
    < while (<>) {
    < next if (/last message repeated/);
    ---
    > if (!$sth->execute) {
    > Zimbra::Logger::Log ("err", "DB: $sth->errstr");
    > return undef;
    > }
    86,95c100
    < # attempt to reconnect if mysql dropped
    < if (!$dbh->ping) {
    < logMsg("DEBUG", "DB Connection dropped. Attempting to reconnect");
    < undef $dbh;
    < $dbh = dbConnect($data_source,$username,$password);
    <
    < # prepare the statements with the new connection
    < $sth = $dbh->prepare($statement);
    < $s_sth = $dbh->prepare($status_statement);
    < }
    ---
    > my @ary = $sth->fetchrow_array;
    97,99c102,104
    < # parse the input
    < my ($log_date, $host, $ip, $name, $app, $pid, $msg) = ($_ =~ m/$logregex/);
    < $host = (($ip ne "") ? $ip : $name);
    ---
    > return (datetime_to_timestamp($ary[0]));
    >
    > }
    101,145c106,107
    < # skip slapd and master
    < if ($app eq 'slapd' || $app eq 'master') {
    < logMsg("DEBUG", "Skipping update for app $app");
    < next;
    < } else {
    < logMsg("DEBUG", "hostname:$host app:$app name:$name log_date:$log_date");
    < }
    <
    < next if ($host eq "");
    < my $qid = undef;
    <
    < $qid = $1 if ($app =~ /^postfix/ && $msg =~ /^(\S{8,12}): /);
    <
    < if ($msg =~ /info: ([^,]+), STATUS: (\S+): (\S+): (Running|Stopped).*$/) {
    < my $tm = $1;
    < my $hostname = $2;
    < my $service = $3;
    < my $status = ($4 eq "Running")?1:0;
    <
    < # delete previous entries
    < my $statement = "delete from service_status ".
    < "where server=\'".$hostname."\' and service=\'".$service."\'";
    < my $dsth = $dbh->prepare($statement);
    < if (!$dsth->execute() ) {
    < logMsg("ERROR", $dsth->errstr);
    < next;
    < }
    < # insert new status entries
    < if (!$s_sth->execute($hostname, $service, $tm, $status, $host)) {
    < logMsg("ERROR", "Status update failed for service:$service host:$host status:$status");
    < logMsg("ERROR", $s_sth->errstr);
    < } else {
    < logMsg("INFO", "Status update for service:$service host:$host status:$status tm:$tm");
    < }
    < next;
    < }
    <
    < if (!$sth->execute(logdate_to_sqldate($log_date), $host, $app, $pid, $msg, $qid) ) {
    < logMsg("ERROR", "Error inserting $log_date, $host, $app, $pid, $msg");
    < logMsg("DEBUG", "From $_");
    < logMsg("DEBUG", "DBI errstr: ",$sth->errstr);
    < #logMsg("ERROR", "zmlogger insert failed $sth->errstr");
    < } else {
    < logMsg("INFO", "zmlogger insert completed: $host $app $pid $msg $qid");
    < }
    ---
    > my $statement =
    > "insert into raw_logs(log_date, loghost, app, pid, msg, postfix_qid) values (?,?,?,?,?,?)";
    147,148c109
    < }
    < quit(1, "Loop failed this should never be reached");
    ---
    > my $sth = $dbh->prepare($statement);
    150c111,112
    < # Sub Routines
    ---
    > my $status_statement =
    > "insert into service_status(server, service, time, status, loghostname) values (?,?,?,?,?)";
    152,155c114
    < sub isSqlRunning {
    < `${zimbra_home}/bin/logmysqladmin status > /dev/null 2>&1`;
    < return(($? == 0) ? 1 : 0);
    < }
    ---
    > my $s_sth = $dbh->prepare($status_statement);
    157,188c116
    < sub quit($$) {
    < my ($rc,$msg) = @_;
    < logMsg("INFO", "$msg");
    < logMsg("INFO", "zmlogger shutting down");
    < close FOO if $debug;
    < exit $rc;
    < }
    <
    < sub logMsg {
    < my ($lvl, $msg) = @_;
    < print FOO localtime(time), " $msg\n" if $debug;
    < print "$msg\n" if ($lvl eq "INFO" || $lvl eq "ERROR");
    < }
    <
    < sub dbConnect($$$) {
    < my ($db_src,$db_user,$db_pass) = @_;
    < my $timeout=0;
    <
    < while (! isSqlRunning() && ($timeout <= 3600) ) {
    < logMsg("INFO", "database not running. retrying connection");
    < $timeout += sleep 5;
    < }
    < quit(1, "zmlogger database isn't running") unless isSqlRunning();
    <
    < my $dbh = DBI->connect($db_src,$db_user,$db_pass);
    < $dbh->{PrintError} = 0 unless $debug;
    < $dbh->{RaiseError} = 0 unless $debug;
    <
    < if (!$dbh) {
    < logMsg("ERROR", "DB: Can't connect to $db_src: $DBI::errstr");
    < quit(1, "Can not connect to database $DBI::errstr");
    < }
    ---
    > my $DEBUG = 0;
    190,191c118
    < return $dbh;
    < }
    ---
    > ($DEBUG) && open FOO, ">/tmp/zmlogger.out";
    193,203c120
    < sub logdate_to_sqldate {
    < my $ts = shift;
    < if ($ts eq "" || !defined($ts)) {
    < return '';
    < }
    < # Sep 14 18:30:02
    <
    < my @p = split (' ', $ts);
    < my @s= localtime();
    < my $y = $s[5]+1900;
    < my $m = $MON{$p[0]};
    ---
    > ($DEBUG) && print FOO localtime()," zmlogger starting up\n";
    205,207c122
    < my $dt = sprintf ("%4d-%02d-%02d %s",$y,$m,$p[1],$p[2]);
    < return $dt;
    < }
    ---
    > $SIG{QUIT} = \&bye;
    208a124
    > my $logregex = qr/(^.{15}) ((\d+\.\d+\.\d+\.\d+) \S+|(\S+)) ([^[]+)\[(\d+)\]: (.*)$/o;
    210,211c126,127
    < sub get_last_timestamp {
    < my $ts;
    ---
    > while (<>) {
    > (/last message repeated/) && next;
    213c129,130
    < my $statement = "select max(time) from server_stat";
    ---
    > my ($log_date, $host, $ip, $name, $app, $pid, $msg) = ($_ =~ m/$logregex/);
    > $host = (($ip ne "") ? $ip : $name);
    215c132
    < my $sth = $dbh->prepare($statement);
    ---
    > if ($app eq 'slapd' || $app eq 'master') {next;}
    217,220c134,135
    < if (!$sth->execute) {
    < Zimbra::Logger::Log ("err", "DB: $sth->errstr");
    < return undef;
    < }
    ---
    > if ($host eq "") { next;}
    > my $qid = undef;
    222c137,184
    < my @ary = $sth->fetchrow_array;
    ---
    > if ($app =~ /^postfix/ && $msg =~ /^(\S{8,12}): /) {
    > $qid = $1;
    > }
    >
    > if ($msg =~ /info: ([^,]+), STATUS: (\S+): (\S+): (Running|Stopped).*$/) {
    > my $tm = $1;
    > my $hostname = $2;
    > my $service = $3;
    > my $status = ($4 eq "Running")?1:0;
    > my $statement = "delete from service_status ".
    > "where server=\'".$hostname."\' and service=\'".$service."\'";
    >
    > my $dsth = $dbh->prepare($statement);
    >
    > #print "Executing $statement with @args\n\n";
    >
    > eval {
    > if (!$dsth->execute() ) {
    > warn $dsth->errstr;
    > next;
    > }
    > };
    >
    >
    > eval {
    > $s_sth->execute($hostname, $service, $tm, $status, $host);
    > };
    >
    > next;
    > }
    >
    > eval {
    > if (!$sth->execute(logdate_to_sqldate($log_date), $host,
    > $app, $pid, $msg, $qid) ) {
    > ($DEBUG) && print FOO localtime(),
    > " zmlogger insert failed: ",$sth->errstr,"\n";
    > } else {
    > #($DEBUG) && print FOO localtime(),
    > #" zmlogger insert completed: $host $app $pid $msg $qid\n";
    > }
    >
    > };
    > if ($@) {
    > print "Error inserting $log_date, $host, $app, $pid, $msg\n";
    > print "From $_\n";
    > print $sth->errstr,"\n";
    > print "$@\n";
    > }
    224,225d185
    < return (datetime_to_timestamp($ary[0]));
    <
    226a187,198
    > ($DEBUG) && close FOO;
    >
    > sub isSqlRunning {
    > `logmysqladmin status > /dev/null 2>&1`;
    > return(($? == 0) ? 1 : 0);
    > }
    >
    > sub bye {
    > ($DEBUG) && close FOO;
    > exit (0);
    > }
    >
    viognier:/opt/zimbra/libexec #

    All the best,
    Mark

  6. #6
    brian is offline Project Contributor
    Join Date
    Jul 2006
    Posts
    623
    Rep Power
    10

    Default

    The patch was mostly rolled into 4.5.6 but unfortunately the debug mode got left on in a couple of the scripts so these files will continue to be large. 4.5.7 will lower the logging level back to the normal level.

    4.5.7 will also move the files from /tmp to /opt/zimbra/log and they will be part of the regular log rotation. In the meantime it is safe to truncate the files after stopping logger. They will be recreated upon restart.

    Code:
    % zmlogswatchctl stop
    % rm -f /tmp/zmlogger.out /tmp/logswatch.out
    % zmlogswatchctl start
    Bugzilla - Wiki - Downloads - Before posting... Search!

  7. #7
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,373
    Rep Power
    10

    Default

    Quote Originally Posted by brian View Post
    The patch was mostly rolled into 4.5.6 but unfortunately the debug mode got left on in a couple of the scripts so these files will continue to be large. 4.5.7 will lower the logging level back to the normal level.

    4.5.7 will also move the files from /tmp to /opt/zimbra/log and they will be part of the regular log rotation. In the meantime it is safe to truncate the files after stopping logger. They will be recreated upon restart.

    Code:
    % zmlogswatchctl stop
    % rm -f /tmp/zmlogger.out /tmp/logswatch.out
    % zmlogswatchctl start
    Thanks Brian for the clarification on this.

    I put the 4.5.6 files back in place instead of the 4.5.5 files (some debug code) and ran the above three commands just fine.

    All the best,
    Mark

  8. #8
    jameztcc is offline Active Member
    Join Date
    May 2007
    Posts
    26
    Rep Power
    8

    Default

    Hi,

    in regards to this issue, is there a way to 'automate' the logging size?


    thanks,
    James

  9. #9
    jholder's Avatar
    jholder is offline Former Zimbran
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    Install 4.5.7

  10. #10
    jameztcc is offline Active Member
    Join Date
    May 2007
    Posts
    26
    Rep Power
    8

    Default

    Quote Originally Posted by jholder View Post
    Install 4.5.7
    Would 4.5.7 solve the problem?

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •