server stats are not working

hi.

after upgrading to network-edition server stats in admin-interface are not working any more.

Code:

zmcontrol -v Release 8.0.1_GA_5438.SLES11_64_20121105164455 SLES11_64 NETWORK edition.

i searched the forum but could not solve the problem. here are some information:

Code:

cat /etc/SuSE-release SUSE Linux Enterprise Server 11 (x86_64) VERSION = 11 PATCHLEVEL = 2

Code:

cat /etc/profile.local ulimit -v unlimited ulimit -m unlimited

Code:

zmcontrol status Host zimbraserver.com antispam Running antivirus Running convertd Running ldap Running logger Running mailbox Running memcached Running mta Running proxy Running snmp Running spell Running stats Running zmconfigd Running

Code:

cat /etc/syslog-ng/syslog-ng.conf | grep zimbra filter(f_iptables) and not filter(zimbra_local0) and not filter(zimbra_local1) and not filter(zimbra_auth) and not filter(zimbra_mail); }; not filter(zimbra_local0) and not filter(zimbra_local1) and not filter(zimbra_auth) and not filter(zimbra_mail); }; filter zimbra_local0 { facility(local0); }; # zimbra filter zimbra_local1 { facility(local1); }; # zimbra filter zimbra_auth { facility(auth); }; # zimbra filter zimbra_mail { facility(mail); }; # zimbra destination zimbra_mail { udp("1.1.1.1" port(514) ); }; # zimbra destination zimbra_local1 { udp("1.1.1.1" port(514) ); }; # zimbra destination zimbra_local0 { udp("1.1.1.1" port(514) ); }; # zimbra destination zimbra_auth { udp("1.1.1.1" port(514) ); }; # zimbra log { source(src); filter(zimbra_mail); destination(zimbra_mail); }; # zimbra log { source(src); filter(zimbra_local0); destination(zimbra_local0); }; # zimbra log { source(src); filter(zimbra_local1); destination(zimbra_local1); }; # zimbra log { source(src); filter(zimbra_auth); destination(zimbra_auth); }; # zimbra

some hints what might be wrong

Code:

/opt/zimbra/libexec/zmrrdfetch -f zmmtastats DBD::SQLite::db prepare failed: no such table: rrds at /opt/zimbra/libexec/zmrrdfetch line 57. Can't call method "bind_param" on an undefined value at /opt/zimbra/libexec/zmrrdfetch line 62.

Code:

/var/log/zimbra.log is empty

Code:

tail /var/log/messages Dec 6 16:11:28 vvx-mail-01 syslog-ng[58510]: I/O error occurred while writing; fd='4', error='Connection refused (111)' Dec 6 16:11:28 vvx-mail-01 syslog-ng[58510]: Connection broken; time_reopen='60' Dec 6 16:11:45 vvx-mail-01 syslog-ng[58510]: I/O error occurred while writing; fd='4', error='Connection refused (111)' Dec 6 16:11:45 vvx-mail-01 syslog-ng[58510]: Connection broken; time_reopen='60' Dec 6 16:11:49 vvx-mail-01 syslog-ng[35193]: I/O error occurred while writing; fd='5', error='Connection refused (111)' Dec 6 16:11:49 vvx-mail-01 syslog-ng[35193]: Connection broken; time_reopen='60' Dec 6 16:11:49 vvx-mail-01 syslog-ng[35193]: I/O error occurred while writing; fd='5', error='Connection refused (111)' Dec 6 16:11:49 vvx-mail-01 syslog-ng[35193]: Connection broken; time_reopen='60' Dec 6 16:11:53 vvx-mail-01 syslog-ng[58510]: I/O error occurred while writing; fd='4', error='Connection refused (111)' Dec 6 16:11:53 vvx-mail-01 syslog-ng[58510]: Connection broken; time_reopen='60'

You shouldn't have memcached or the proxy running on a single server, search the forums for details on how to stop & disable those services and reset the ports. You could also reinitialize the logger, you might also have a problem with syslog-ng or you could try some of the solutions mentioned in these forum posts: site:zimbra.com +"/opt/zimbra/libexec/zmrrdfetch line" - Yahoo! Search Results

bill, thank you for your feedback.

but I could not solve it yet.

1) You shouldn't have memcached or the proxy running on a single server

i disable memcached - but when I disable proxy imap is not working anymore. so i enable proxy again.

Code:

zmmemcachedctl stop zmprov ms `zmhostname` -zimbraServiceEnabled memcached zmproxyctl stop zmprov ms `zmhostname` -zimbraServiceEnabled proxy -> imap not working anymore zmproxyctl start zmprov ms `zmhostname` +zimbraServiceEnabled proxy

2) You could also reinitialize the logger

Code:

/opt/zimbra/libexec/zmsyslogsetup updateSyslogNG: Updating /etc/syslog-ng/syslog-ng.conf...done.

3) you might also have a problem with syslog-ng

Code:

tail -n 20 /etc/syslog-ng/syslog-ng.conf # # Warnings (except iptables) in one file: # destination warn { file("/var/log/warn" fsync(yes)); }; log { source(src); filter(f_warn); destination(warn); }; filter zimbra_local0 { facility(local0); }; # zimbra filter zimbra_local1 { facility(local1); }; # zimbra filter zimbra_auth { facility(auth); }; # zimbra filter zimbra_mail { facility(mail); }; # zimbra destination zimbra_mail { udp("91.195.95.131" port(514) ); }; # zimbra destination zimbra_local1 { udp("91.195.95.131" port(514) ); }; # zimbra destination zimbra_local0 { udp("91.195.95.131" port(514) ); }; # zimbra destination zimbra_auth { udp("91.195.95.131" port(514) ); }; # zimbra log { source(src); filter(zimbra_mail); destination(zimbra_mail); }; # zimbra log { source(src); filter(zimbra_local0); destination(zimbra_local0); }; # zimbra log { source(src); filter(zimbra_local1); destination(zimbra_local1); }; # zimbra log { source(src); filter(zimbra_auth); destination(zimbra_auth); }; # zimbra

4) could try some of the solutions mentioned in these forum posts:

I found this posts which seems to be the same problem:
http://www.zimbra.com/forums/adminis...tml#post181241

but it already failed at step 2:

Code:

zimbra@vvx-mail-01:/root> /opt/zimbra/libexec/zmlogger& [1] 11261 zimbra@vvx-mail-01:/root> zmrrdfetch server pid: 11295 DBD::SQLite::db selectrow_array failed: no such table: config at /opt/zimbra/libexec/zmlogger line 409. Schema version not found, upgrading to version 4 at /opt/zimbra/libexec/zmlogger line 412. [1]+ Stopped /opt/zimbra/libexec/zmlogger

5) troubleshooting logs
Logger (ZCS 6.0.x and later) - Zimbra :: Wiki

Code:

zmloggerctl status; echo $?; ps ax | grep zmlogger 0 20605 pts/1 S+ 0:00 grep zmlogger /opt/zimbra/libexec/zmrrdfetch -f zmmtastats no structure found for zmmtastats at /opt/zimbra/libexec/zmrrdfetch line 239. /opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats no structure found for zmmtastats at /opt/zimbra/libexec/zmrrdfetch line 239. grep -w MTA /var/log/zimbra-stats.log 0 tail /var/log/zimbra-stats.log 0 zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20 ERROR: service.FAILURE (system failure: Unable to read logger stats) zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50 ERROR: service.FAILURE (system failure: Unable to read logger stats)

so I checked this posts without success:
http://www.zimbra.com/forums/adminis...log-error.html
http://www.zimbra.com/forums/adminis...s-running.html
http://www.zimbra.com/forums/adminis...er-status.html
http://www.zimbra.com/forums/install...-services.html

ok - now logging to /var/log/zimbra.log and /var/log/zimbra-stats.log works again - but not the rest.

what have I done?

when I run following command:

Code:

/opt/zimbra/libexec/zmsyslogsetup

then /etc/syslog-ng/syslog-ng.conf looks like this:

Code:

cat syslog-ng.conf | grep zimbra filter(f_iptables) and not filter(zimbra_local0) and not filter(zimbra_local1) and not filter(zimbra_auth) and not filter(zimbra_mail); }; not filter(zimbra_local0) and not filter(zimbra_local1) and not filter(zimbra_auth) and not filter(zimbra_mail); }; filter zimbra_local0 { facility(local0); }; # zimbra filter zimbra_local1 { facility(local1); }; # zimbra filter zimbra_auth { facility(auth); }; # zimbra filter zimbra_mail { facility(mail); }; # zimbra destination zimbra_mail { udp("X.X.X.X" port(514) ); }; # zimbra destination zimbra_local1 { udp("X.X.X.X" port(514) ); }; # zimbra destination zimbra_local0 { udp("X.X.X.X" port(514) ); }; # zimbra destination zimbra_auth { udp("X.X.X.X" port(514) ); }; # zimbra log { source(src); filter(zimbra_mail); destination(zimbra_mail); }; # zimbra log { source(src); filter(zimbra_local0); destination(zimbra_local0); }; # zimbra log { source(src); filter(zimbra_local1); destination(zimbra_local1); }; # zimbra log { source(src); filter(zimbra_auth); destination(zimbra_auth); }; # zimbra

restarting syslog tells me this:

Code:

rcsyslog restart Shutting down syslog services done Starting syslog servicesDuplicate stats counter; counter='udp(AF_INET(X.X.X.X:514))' Duplicate stats counter; counter='udp(AF_INET(X.X.X.X:514))' Duplicate stats counter; counter='udp(AF_INET(X.X.X.X:514))'

I changed syslog-ng.conf to this:

Code:

cat syslog-ng.conf | grep zimbra filter(f_iptables) and not filter(zimbra_local0) and not filter(zimbra_local1) and not filter(zimbra_auth) and not filter(zimbra_mail); }; not filter(zimbra_local0) and not filter(zimbra_local1) and not filter(zimbra_auth) and not filter(zimbra_mail); }; filter zimbra_local0 { facility(local0); }; # zimbra filter zimbra_local1 { facility(local1); }; # zimbra filter zimbra_auth { facility(auth); }; # zimbra filter zimbra_mail { facility(mail); }; # zimbra destination zimbra_mail { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra destination zimbra_local1 { file("/var/log/zimbra-stats.log" owner("zimbra")); }; # zimbra destination zimbra_local0 { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra destination zimbra_auth { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra log { source(src); filter(zimbra_mail); destination(zimbra_mail); }; # zimbra log { source(src); filter(zimbra_local0); destination(zimbra_local0); }; # zimbra log { source(src); filter(zimbra_local1); destination(zimbra_local1); }; # zimbra log { source(src); filter(zimbra_auth); destination(zimbra_auth); }; # zimbra

restarting syslog also is better now:

Code:

rcsyslog restart Shutting down syslog services done Starting syslog services done

now troubleshooting-logs give this:

Code:

zmloggerctl status; echo $?; ps ax | grep zmlogger 0 45492 ? R 0:00 /usr/bin/perl /opt/zimbra/libexec/zmlogger 45527 ? S 0:00 zmlogger: zmrrdfetch: server 45582 pts/1 S+ 0:00 grep zmlogger /opt/zimbra/libexec/zmrrdfetch -f zmmtastats no structure found for zmmtastats at /opt/zimbra/libexec/zmrrdfetch line 239. /opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats no structure found for zmmtastats at /opt/zimbra/libexec/zmrrdfetch line 239. grep -w MTA /var/log/zimbra-stats.log -> nothing tail /var/log/zimbra-stats.log Dec 10 12:13:30 vvx-mail-01 zimbramon[6959]: 6959:info: :::A102CD9E-42BA-11E2-A3DC-F4F86A583D9E:::0,0,0,0,0,0,0,0,0,0,0,0,79562,0,0,0,0,0,0,0,0,0,0,0,0,8988,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,21232,4807,0,0,0,0,0,0,0,0,0,0,0,0,0,12935,0,0,0,0,0,80369,0,0,72876,241098,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,130,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,593,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,43408,3,0,0,0,0,0,0,OFF,602,3,5,546,0,0,0,1,243279,4627,0,0,63,394660,0,3794951,23935,132854,180774,661,0,0,61208,203670,38739,0,158550,0,876,39615,0,3885,41,15322115,40398,0,785130,16198,0,0,0,733302784,41495,64244,2648286208,79275,1564,ON,0,113809,4220,4962,0,0,50223104,16384,1277,44623,79275,0,0,0,0,1,7479,33205,4148566,61208,0,0,6698,1,0,0,2,0,0.000000,8,0,17,0,475,525,537,482,532,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,566540,566540,AUTH_MASTER,0,0,9712,0,585,0.000,0,0,0,OFF,0,52,0,1318,12967,1,0,0,:::A102D424-42BA-11E2-A3DC-F4F86A583D9E::: Dec 10 12:13:30 vvx-mail-01 zimbramon[6959]: :::A102D424-42BA-11E2-A3DC-F4F86A583D9E:::0,,,0,0,0,0,0,0,0,0,0,NONE,0,0,0,0,0,0,0,,175824,0,0,0,0,0,8,8,1,7869,7869 Dec 10 12:13:31 vvx-mail-01 zimbramon[60376]: 60376:info: zmstat fd.csv: timestamp, fd_count, mailboxd_fd_count:: 12/10/2012 12:13:31, 5408, 1725 Dec 10 12:13:34 vvx-mail-01 zimbramon[59914]: 59914:info: zmstat convertd.csv: timestamp, utime, stime, cputime, rchar, wchar, read_bytes, write_bytes, rss, processes, threads:: 12/10/2012 12:13:34, 0, 0, 0, 0, 0, 0, 0, 62284, 6, 6 Dec 10 12:13:35 vvx-mail-01 zimbramon[6953]: 6953:info: zmstat io.csv: timestamp, xvda:tps, xvda:kB_read/s, xvda:kB_wrtn/s, xvda:kB_read, xvda:kB_wrtn:: 12/10/2012 12:13:35, 4.57, 0.13, 45.87, 4, 1376 Dec 10 12:13:35 vvx-mail-01 zimbramon[6949]: 6949:info: zmstat io-x.csv: timestamp, xvda:rrqm/s, xvda:wrqm/s, xvda:r/s, xvda:w/s, xvda:rkB/s, xvda:wkB/s, xvda:avgrq-sz, xvda:avgqu-sz, xvda:await, xvda:svctm, xvda:%util:: 12/10/2012 12:13:35, 0.00, 3.70, 0.03, 4.30, 0.13, 43.60, 20.18, 0.04, 8.49, 1.51, 0.65 Dec 10 12:13:36 vvx-mail-01 zimbramon[59916]: 59916:info: zmstat mtaqueue.csv: timestamp, KBytes, requests:: 12/10/2012 12:13:36, 146, 3 Dec 10 12:13:42 vvx-mail-01 zimbramon[6965]: 6965:info: zmstat nginx.csv: timestamp, utime, stime, cputime, rchar, wchar, read_bytes, write_bytes, rss, processes, threads:: 12/10/2012 12:13:42, 1, 3, 4, 52503, 86904, 0, 4096, 34092, 5, 5 Dec 10 12:13:44 vvx-mail-01 zimbramon[6947]: 6947:info: zmstat vm.csv: timestamp, r, b, swpd, free, buff, cache, si, so, bi, bo, in, cs, us, sy, id, wa, st, MemTotal, MemFree, Buffers, Cached, SwapCached, Active, Inactive, Active(anon), Inactive(anon), Active(file), Inactive(file), Unevictable, Mlocked, SwapTotal, SwapFree, Dirty, Writeback, AnonPages, Mapped, Shmem, Slab, SReclaimable, SUnreclaim, KernelStack, PageTables, NFS_Unstable, Bounce, WritebackTmp, CommitLimit, Committed_AS, VmallocTotal, VmallocUsed, VmallocChunk, HardwareCorrupted, DirectMap4k, DirectMap2M, loadavg:: 12/10/2012 12:13:44, 3, 0, 3340, 380736, 258488, 4470692, 0, 0, 0, 48, 604, 876, 4, 1, 95, 0, 0, 8404788, 380736, 258488, 4256568, 20, 4088444, 3341192, 2377752, 535668, 1710692, 2805524, 0, 0, 2103292, 2099952, 576, 0, 2912980, 60420, 572, 239008, 214104, 24904, 3944, :::A8EEE9FC-42BA-11E2-AAA1-E8F86A583D9E::: Dec 10 12:13:44 vvx-mail-01 zimbramon[6947]: :::A8EEE9FC-42BA-11E2-AAA1-E8F86A583D9E:::0, 0, 0, 0, 6305684, 4828440, 34359738367, 5196, 34359731872, 0, 8388608, 0, 0.47 zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20 <GetLoggerStatsResponse xmlns="urn:zimbraAdmin"/> zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50 <GetLoggerStatsResponse xmlns="urn:zimbraAdmin"/>

ooooooooooooooohhh - after restarting zmloggerctl now also the services in admin-interface are shown green again :-)
but I am wondering why /opt/zimbra/libexec/zmrrdfetch is not working.....
so zimbra is doing the syslog-ng.conf not correct?!

bill what do you think about it?