[SOLVED] Stats on 6.0.5 cant be started - /opt/zimbra/libexec/zmstat-fd - Multi-serve

[sugiggs]

I'm testing to upgrade from 6.0.3 to 6.0.5
Our setup is multi-server setup

All stats on our servers, except mailbox servers can't be started
then when I check zmstatctl status, only zmstat-fd is not running

after scratching head for half a day, I find out that the script on 6.0.5 and 6.0.3 are different.

Looks like (sorry not a perl programmer) the 6.0.5 script assumes that it is running on mailbox server.
I replace 6.0.5 script with 6.0.3 one then stats magically can be started again...

any comment??

6.0.3 /opt/zimbra/libexec/zmstat-fd script

#!/usr/bin/perl -w
#
# ***** BEGIN LICENSE BLOCK *****
# Zimbra Collaboration Suite Server
# Copyright (C) 2008, 2009 Zimbra, Inc.
#
# The contents of this file are subject to the Yahoo! Public License
# Version 1.0 ("License"); you may not use this file except in
# compliance with the License.  You may obtain a copy of the License at
# http://www.zimbra.com/license.
#
# Software distributed under the License is distributed on an "AS IS"
# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.
# ***** END LICENSE BLOCK *****
#

# Periodically print FD open stats obtained from /proc/sys/fs/file-nr
# The first column of /proc/sys/fs/file-nr gives an instantaneous reading
# of all file descriptors open throughout all processes.  The third column
# gives a maximum file descriptor count for the entire system.

use strict;
use Getopt::Long;
use lib "/opt/zimbra/zimbramon/lib";
use Zimbra::Mon::Zmstat;
use Zimbra::Mon::Logger;
use vars qw($LOGFH $CONSOLE $LOGFILE $ROTATE_NOW $ROTATE_DEFER);

zmstatInit();

my $STAT = '/proc/sys/fs/file-nr';
my $HEADING = 'timestamp, fd_count';

sub get_fd_stat() {
    open(STAT, "<$STAT") || die "Can't open $STAT: $!";
    my $line = <STAT>;
    $line =~ /^(\d+)\s+/;
    my $stat = $1;
    close(STAT);
    $stat;
}
sub sighup {
    if (!$CONSOLE) {
        $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING);
    } else {
        $ROTATE_NOW = 1;
    }
}
$SIG{HUP} = \&sighup;

sub usage {
    print STDERR << '_USAGE_';
Usage: zmstat-fd [options]
Monitor system filedescriptor usage
-i, --interval: output a line every N seconds
-l, --log:      log file (default is /opt/zimbra/zmstat/fd.csv)
-c, --console:  output to stdout
If logging to a file, rotation occurs when a HUP signal is sent or when
the date changes.  The current log is renamed to <dir>/YYY-MM-DD/fd.csv
and a new file is created.
_USAGE_
    exit(1);
}

$| = 1;

$LOGFILE = getLogFilePath('fd.csv');
my $interval = getZmstatInterval();
my $opts_good = GetOptions(
    'interval=i' => \$interval,
    'log=s' => \$LOGFILE,
    'console' => \$CONSOLE,
);
if (!$opts_good) {
    print STDERR "\n";
    usage();
}
createPidFile('zmstat-fd.pid');

my $date = getDate();
if ($CONSOLE) {
    $LOGFH = \*STDOUT;
    $LOGFH->print($HEADING . "\n");

} else {
    $LOGFH = openLogFile($LOGFILE, $HEADING);
}
waitUntilNiceRoundSecond($interval);
while (1) {
    my $stat = get_fd_stat();
    my $tstamp = getTstamp();
    my $currDate = getDate();
    if ($currDate ne $date && !$CONSOLE) {
        $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING, $date);
        $date = $currDate;
    }
    $ROTATE_DEFER = 1;
    $LOGFH->print("$tstamp, $stat\n");
    Zimbra::Mon::Logger::LogStats( "info", "zmstat fd.csv: ${HEADING}:: $tstamp, $stat");
    $LOGFH->flush();
    $ROTATE_DEFER = 0;
    if ($ROTATE_NOW) {
        $ROTATE_NOW = 0;
        $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING);
    }
    sleep($interval);
}

6.0.5 /opt/zimbra/libexec/zmstat-fd script

#!/usr/bin/perl -w
# 
# ***** BEGIN LICENSE BLOCK *****
# Zimbra Collaboration Suite Server
# Copyright (C) 2008, 2009, 2010 Zimbra, Inc.
# 
# The contents of this file are subject to the Zimbra Public License
# Version 1.3 ("License"); you may not use this file except in
# compliance with the License.  You may obtain a copy of the License at
# http://www.zimbra.com/license.
# 
# Software distributed under the License is distributed on an "AS IS"
# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.
# ***** END LICENSE BLOCK *****
# 

# Periodically print FD open stats obtained from /proc/sys/fs/file-nr
# The first column of /proc/sys/fs/file-nr gives an instantaneous reading
# of all file descriptors open throughout all processes.  The third column
# gives a maximum file descriptor count for the entire system.

use strict;
use Getopt::Long;
use lib "/opt/zimbra/zimbramon/lib";
use Zimbra::Mon::Zmstat;
use Zimbra::Mon::Logger;
use vars qw($LOGFH $CONSOLE $LOGFILE $ROTATE_NOW $ROTATE_DEFER);

Zimbra::Mon::Zmstat::osCheck();
Zimbra::Mon::Zmstat::getLocalConfig('zimbra_home', 'zimbra_user', 'zimbra_server_hostname',
               'zmstat_log_directory', 'zmstat_interval',
               'zmstat_disk_interval');
my $zuser = $Zimbra::Mon::Zmstat::LC{zimbra_user};
my ($zuid,$zgid) = (getpwnam($zuser))[2,3];

if (@ARGV > 0 && $ARGV[0] eq 'stop') {
	my $dir = getPidFileDir();
	my $pidFile = "$dir/zmstat-fd-real.pid";
	my $pid = readPidFile($pidFile);
    if (!kill(0, $pid)) {
        unlink($pidFile);
    } elsif (kill(15, $pid) == 1) {  # SIGTERM
        unlink($pidFile);
    } elsif (kill(9, $pid) == 1) {
    	unlink($pidFile);
    }
	exit;
}
if (@ARGV > 0 && $ARGV[0] eq 'rotate') {
	my $dir = getPidFileDir();
	my $pidFile = "$dir/zmstat-fd-real.pid";
	my $pid = readPidFile($pidFile);
    kill(1, $pid);
}

if ($< != 0) {
    Zimbra::Mon::Zmstat::userCheck();
    createPidFile('zmstat-fd.pid');
    $SIG{TERM} = sub {
    	system("sudo /opt/zimbra/libexec/zmstat-fd stop");
    };
    $SIG{INT} = sub {
    	system("sudo /opt/zimbra/libexec/zmstat-fd stop");
    };
    $SIG{HUP} = sub {
    	system("sudo /opt/zimbra/libexec/zmstat-fd rotate");
    };
    my $args = "";
    if (@ARGV > 0) {
    	$args = " " . join(' ', @ARGV);
    }
    system("sudo /opt/zimbra/libexec/zmstat-fd$args");
    exit;
}

my $STAT = '/proc/sys/fs/file-nr';
my $HEADING = 'timestamp, fd_count, mailboxd_fd_count';

sub get_fd_stat() {
    open(STAT, "<$STAT") || die "Can't open $STAT: $!";
    my $line = <STAT>;
    $line =~ /^(\d+)\s+/;
    my $stat = $1;
    close(STAT);
    $stat;
}
sub get_mbox_stat() {
    my $mbox_pid = get_mboxd_pid();
    return 0 if ($mbox_pid == 0);
    
    opendir(PROC, "/proc/$mbox_pid/fd") || warn $!;
    my @ents = grep { /^[0-9]+/ } readdir(PROC);
    closedir(PROC);
    return scalar @ents;
}

my $MBOX_PID_FILE = '/opt/zimbra/log/zmmailboxd_java.pid';
sub get_mboxd_pid() {
    my $pid = 0;
    if (-f $MBOX_PID_FILE) {
        eval {
            open(PID_FILE, "<$MBOX_PID_FILE") || die $!;
            $pid = <PID_FILE>;
            chomp($pid); chomp($pid);
            close PID_FILE;
        };
        if ($@) {
            print STDERR "Unable to find pid file, falling back: $@\n";
        }
    }
    if (!$pid) {
        eval {
            open(PS, "ps -ef | grep [j]etty.xml | awk '{ print \$2 }' |")
                    || die $!;
            $pid = <PS>;
            chomp($pid); chomp($pid);
            close(PS);
        };
        if ($@) {
            print STDERR "Unable to manually obtain mailboxd pid: $@\n";
        } elsif (!$pid) {
            print STDERR "Unable to determine mailboxd pid\n";
        }
    }
    return $pid;
}

sub sighup {
    if (!$CONSOLE) {
        $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING);
    } else {
        $ROTATE_NOW = 1;
    }
}
$SIG{HUP} = \&sighup;

sub usage {
    print STDERR << '_USAGE_';
Usage: zmstat-fd [options]
Monitor system filedescriptor usage
-i, --interval: output a line every N seconds
-l, --log:      log file (default is /opt/zimbra/zmstat/fd.csv)
-c, --console:  output to stdout

If logging to a file, rotation occurs when a HUP signal is sent or when
the date changes.  The current log is renamed to <dir>/YYY-MM-DD/fd.csv
and a new file is created.
_USAGE_
    exit(1);
}

$| = 1;

$LOGFILE = getLogFilePath('fd.csv');
my $interval = getZmstatInterval();
my $opts_good = GetOptions(
    'interval=i' => \$interval,
    'log=s' => \$LOGFILE,
    'console' => \$CONSOLE,
);
if (!$opts_good) {
    print STDERR "\n";
    usage();
}
createPidFile('zmstat-fd-real.pid');

my $date = getDate();
if ($CONSOLE) {
    $LOGFH = \*STDOUT;
    $LOGFH->print($HEADING . "\n");
    
} else {
    $LOGFH = openLogFile($LOGFILE, $HEADING);
    chown $zuid, $zgid, $LOGFILE;
}
waitUntilNiceRoundSecond($interval);

while (1) {
    my $stat = get_fd_stat();
    my $mbox_stat = get_mbox_stat();
    my $tstamp = getTstamp();
    my $currDate = getDate();
    if ($currDate ne $date && !$CONSOLE) {
        $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING, $date);
        chown $zuid, $zgid, $LOGFILE;
        $date = $currDate;
    }
    $ROTATE_DEFER = 1;
    $LOGFH->print("$tstamp, $stat, $mbox_stat\n");
    Zimbra::Mon::Logger::LogStats( "info", "zmstat fd.csv: ${HEADING}:: $tstamp, $stat, $mbox_stat"); 
    $LOGFH->flush();
    $ROTATE_DEFER = 0;
    if ($ROTATE_NOW) {
        $ROTATE_NOW = 0;
        $LOGFH = rotateLogFile($LOGFH, $LOGFILE, $HEADING);
    }
    sleep($interval);
}

[uxbod]

Bug 44638 - zmstat-fd should require no password when sudo is executed

[sugiggs]

Ah.. Voted.. !

.
.

Originally Posted by uxbod

Bug 44638 - zmstat-fd should require no password when sudo is executed

[uxbod]

And you can just make the change yourself to /etc/sudoers so that your stats will work.

[dunkirk]

I had a similar problem on Release 6.0.5_GA_2213.UBUNTU8 UBUNTU8 FOSS edition. I upgraded from 6.0.4 to 6.0.5 several days ago. Last night, at 12:01 am, I got an email saying that service stats was stopped. All attempts to revive it failed. I have but a single server, so all services run locally, and I already had the line in my /etc/sudoers file:

%zimbra ALL=NOPASSWD:/opt/zimbra/libexec/zmstat-fd *

I looked on my company's Release 6.0.5_GA_2213.UBUNTU8 UBUNTU8 NETWORK edition to see what the status processes should look like, ran `/usr/bin/perl -w /opt/zimbra/libexec/zmstat-cpu' by hand, and got notice that it couldn't jigger log files around. I see that root wound up owning the stat logfile directory in /opt/zimbra/zmstat/2010-02-23. I changed the ownership back to zimbra, and things seem to be working again.

Although this seems to be a different issue than having a split server setup, the errors looked exactly the same. I don't know why root took over that directory. I'm new to running Zimbra, but my personal server had been purring along for a week when this happened, and I can't think of what I must have done to cause this to happen. Anyway, I thought I'd leave this here to help someone else. It may or may not be completely appropriate. ;-)

UPDATE: I keep coming back to reference this because it seems to be happening about once a week. I changed the perms on /opt/zimbra/zmstat/ to g+w and guid. Hopefully this will force any new directories created there to be group-owned by zimbra, and forced to be group writable, and this will avoid the service from getting hung.