Statistics : No data available / no structure found for zmmtastats ??

[gruad23]

In my admin-console when I want to display statistics there are not data available.

First there was only a graph for 365-days statistic that ended in january and now (because I deleted all rrd-files) there is no statistic at all.

I searched many forum-entries and wiki's but did not find the solution yet.

What I did:

* check if all log-services are running
* deleted the whole rrd-structure in /opt/zimbra/logger/db/data/ in case there was something wrong. The files have been created by zimbra again.
* checked logfiles for troubles
* restart zimbra
* looked at Logger (ZCS 6.0.x and later) - Zimbra :: Wiki where say one should run some commands and post the output here (See below)

Doing the last I found the problem:
no structure found for zmmtastats

There seems to be many people having similar problems that I do but I didnt found many solutions.

There were also people mentioning that syslogd-parameters should be set different in Ubuntu to make it listen on all interfaces. I run ubuntu 10.04 and its rsyslogd now. It seems that my syslog is listening on all interfaces

# netstat -anp | grep syslog
udp        0      0 0.0.0.0:43020           0.0.0.0:*                           20747/rsyslogd  
udp        0      0 0.0.0.0:50840           0.0.0.0:*                           20747/rsyslogd  
udp        0      0 0.0.0.0:34339           0.0.0.0:*                           20747/rsyslogd  
udp        0      0 0.0.0.0:51499           0.0.0.0:*                           20747/rsyslogd

My host has two IP-Adresses and the zimbra-hostname zimba.goldfisch.at resolves to one of them. And the other hostname voy.goldfisch.at resolves to the other IP. Could this be a problem?

zimbra@voy:~$ zmloggerhostmap 
Zimbra Logger Service Hostname Map

---------------------    ---------------------
DNS Hostname             Zimbra Hostname
---------------------    ---------------------
zimbra.goldfisch.at      zimbra.goldfisch.at
voy                      zimbra.goldfisch.at

and now the commands recommended in zimbra-wiki to post here:

zimbra@voy:~$     zmloggerctl status; echo $?; ps ax | grep zmlogger
0
20994 ?        S      0:05 /usr/bin/perl /opt/zimbra/libexec/zmlogger
21092 ?        S      0:00 zmlogger: zmrrdfetch: server
28759 pts/9    S+     0:00 grep zmlogger

zimbra@voy:~$     /opt/zimbra/libexec/zmrrdfetch -f zmmtastats
no structure found for zmmtastats at /opt/zimbra/libexec/zmrrdfetch line 239.

zimbra@voy:~$     /opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats
no structure found for zmmtastats at /opt/zimbra/libexec/zmrrdfetch line 239.

zimbra@voy:~$     grep -w MTA /var/log/zimbra-stats.log


zimbra@voy:~$     tail /var/log/zimbra-stats.log
Nov  5 09:23:00 voy zimbramon[3367]: 3367:info: :::5F716F80-0787-11E1-B277-C4B04DA02999:::, Threads_created, Threads_running, Uptime, Uptime_since_flush_status:: 11/05/2011 09:23:00,0,0,0,0,38422616,11773976103,5,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1354,0,0,0,57135,0,0,0,0,0,0,0,0,0,0,0,0,20632,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,141,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,57,114,0,0,0,0,88489,0,0,90732,269688,0,0,0,0,4,0,0,103,0,0,0,1251,0,1,0,0,380,0,0,1251,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1340,0,1251,103,1251,4,0,0,0,0,0,0,0,0,0,0,0,0,1,21467,74,0,0,0,0,0,0,OFF,1347,2576,5,4803,0,0,0,1,188285,288,0,0,1330,11796257,51473094,1645051,4513,1629849,30280,0,0,21511,236769,14540,3,17393,1,243,14784,24527,1638,120244001,27997,0,129393,31911,0,0,0,864145408,30071,41502,587520000,17393,2871,ON,0,12189,20824,22827,0,0,16560640,16384,10,52609,17393,0,0,0,0,0,309,183,60135386,21510,0,6672,22,82,22,0,0,0.:::5F71725A-0787-11E1-B277-C4B04DA02999:::
Nov  5 09:23:00 voy zimbramon[3367]: :::5F71725A-0787-11E1-B277-C4B04DA02999:::000000,6,0,83,0,1181,1200,13085,1289,1309,0,0,0,0,0,0,0,0,0,557592,557592,NULL,0,0,1091,0,5854,0,0,OFF,0,1228,0,40889,3772,25,0,0,0,,,0,0,0,0,0,0,0,0,0,NONE,0,0,0,0,0,0,0,,156378,0,0,0,0,1,5,6,1,22870,22870
Nov  5 09:23:00 voy zimbramon[3361]: 3361:info: zmstat io.csv: timestamp, sda:tps, sda:kB_read/s, sda:kB_wrtn/s, sda:kB_read, sda:kB_wrtn, sdb:tps, sdb:kB_read/s, sdb:kB_wrtn/s, sdb:kB_read, sdb:kB_wrtn:: 11/05/2011 09:23:00, 13.47, 1.07, 101.60, 32, 3048, 0.00, 0.00, 0.00, 0, 0
Nov  5 09:23:00 voy zimbramon[3351]: 3351:info: zmstat proc.csv: timestamp, system, user, sys, idle, iowait, mailbox, mailbox-total-cpu, mailbox-utime, mailbox-stime, mailbox-totalMB, mailbox-rssMB, mailbox-sharedMB, mailbox-process-count, mysql, mysql-total-cpu, mysql-utime, mysql-stime, mysql-totalMB, mysql-rssMB, mysql-sharedMB, mysql-process-count, convertd, convertd-total-cpu, convertd-utime, convertd-stime, convertd-totalMB, convertd-rssMB, convertd-sharedMB, convertd-process-count, ldap, ldap-total-cpu, ldap-utime, ldap-stime, ldap-totalMB, ldap-rssMB, ldap-sharedMB, ldap-process-count, postfix, postfix-total-cpu, postfix-utime, postfix-stime, postfix-totalMB, postfix-rssMB, postfix-sharedMB, postfix-process-count, amavis, amavis-total-cpu, amavis-utime, amavis-stime, amavis-totalMB, amavis-rssMB, amavis-sharedMB, amavis-process-:::5FB80292-0787-11E1-A801-B4B04DA02999:::
Nov  5 09:23:00 voy zimbramon[3351]: :::5FB80292-0787-11E1-A801-B4B04DA02999:::count, clam, clam-total-cpu, clam-utime, clam-stime, clam-totalMB, clam-rssMB, clam-sharedMB, clam-process-count, zmstat, zmstat-total-cpu, zmstat-utime, zmstat-stime, zmstat-totalMB, zmstat-rssMB, zmstat-sharedMB, zmstat-process-count:: 11/05/2011 09:23:00, system, 1.7, 0.3, 96.7, 1.3, mailbox, 0.1, 0.1, 0.0, 2125.1, 749.9, 14.3, 1, mysql, 0.1, 0.1, 0.0, 535.5, 327.6, 4.7, 1, convertd, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0, ldap, 0.0, 0.0, 0.0, 380.5, 51.0, 6.2, 1, postfix, 0.0, 0.0, 0.0, 46.9, 2.3, 1.7, 3, amavis, 0.0, 0.0, 0.0, 192.3, 86.1, 0.9, 11, clam, 0.0, 0.0, 0.0, 177.3, 128.2, 5.5, 2, zmstat, 0.0, 0.0, 0.0, 4.0, 0.8, 0.7, 13
Nov  5 09:23:00 voy zimbramon[3355]: 3355:info: zmstat vm.csv: timestamp, r, b, swpd, free, buff, cache, si, so, bi, bo, in, cs, us, sy, id, wa, MemTotal, MemFree, Buffers, Cached, SwapCached, Active, Inactive, Active(anon), Inactive(anon), Active(file), Inactive(file), Unevictable, Mlocked, SwapTotal, SwapFree, Dirty, Writeback, AnonPages, Mapped, Shmem, Slab, SReclaimable, SUnreclaim, KernelStack, PageTables, NFS_Unstable, Bounce, WritebackTmp, CommitLimit, Committed_AS, VmallocTotal, VmallocUsed, VmallocChunk, HardwareCorrupted, HugePages_Total, HugePages_Free, HugePages_Rsvd, HugePages_Surp, Hugepagesize, DirectMap4k, DirectMap2M, loadavg:: 11/05/2011 09:23:00, 0, 0, 66600, 112940, 465204, 658332, 0, 0, 1, 102, 483, 809, 2, 0, 97, 1, 4046736, 112940, 465204, 658332, 14296, 2352820, 1248972, 1784860, 716072, 567960, 532900, 0, 0, 993:::5FBBADA2-0787-11E1-97B4-B8B04DA02999:::
Nov  5 09:23:00 voy zimbramon[3355]: :::5FBBADA2-0787-11E1-97B4-B8B04DA02999:::3816, 9867216, 776, 0, 2468244, 67148, 22708, 190852, 165440, 25412, 3136, 23416, 0, 0, 0, 11957184, 4991020, 34359738367, 288412, 34359445952, 0, 0, 0, 0, 0, 2048, 6756, 4177920, 0.08
Nov  5 09:23:00 voy zimbramon[3797]: 3797:info: zmstat fd.csv: timestamp, fd_count, mailboxd_fd_count:: 11/05/2011 09:23:00, 6496, 331
Nov  5 09:23:01 voy zimbramon[3353]: 3353:info: zmstat cpu.csv: timestamp, cpu:user, cpu:nice, cpu:sys, cpu:idle, cpu:iowait, cpu:irq, cpu:softirq, cpu0:user, cpu0:nice, cpu0:sys, cpu0:idle, cpu0:iowait, cpu0:irq, cpu0:softirq, cpu1:user, cpu1:nice, cpu1:sys, cpu1:idle, cpu1:iowait, cpu1:irq, cpu1:softirq, cpu2:user, cpu2:nice, cpu2:sys, cpu2:idle, cpu2:iowait, cpu2:irq, cpu2:softirq, cpu3:user, cpu3:nice, cpu3:sys, cpu3:idle, cpu3:iowait, cpu3:irq, cpu3:softirq:: 11/05/2011 09:23:01, 1.7, 0.0, 0.3, 96.7, 1.3, 0.0, 0.1, 2.1, 0.0, 0.6, 92.5, 4.7, 0.0, 0.1, 0.6, 0.0, 0.3, 98.7, 0.4, 0.0, 0.0, 2.5, 0.0, 0.1, 97.1, 0.2, 0.0, 0.1, 1.5, 0.0, 0.1, 98.3, 0.0, 0.0, 0.1
Nov  5 09:23:07 voy zimbramon[3369]: 3369:info: zmstat mtaqueue.csv: timestamp, KBytes, requests:: 11/05/2011 09:23:07, 0, 0

zimbra@voy:~$     zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20
<GetLoggerStatsResponse xmlns="urn:zimbraAdmin"/>

zimbra@voy:~$     zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50 
<GetLoggerStatsResponse xmlns="urn:zimbraAdmin"/>

any help greatly appreatiated
thnx,
p

[phoenix]

Originally Posted by gruad23

any help greatly appreatiated

Which of the forums threads that cover this problem/solution have you tried? Have you also tried reinitialising the stats service? Has this problem just started happening? Does your forum profile reflect the actual version of ZCS you're using? Is this a single server or multi-server installation?

[gruad23]

its a single-server installation and the version in my status-line is correct. Its zcs-7.1.3_GA_3346.UBUNTU10_64

I searched many forum-entries and the solutions I found were:

* delete rrd-files as mentioned in my original post
* mess with syslogd which I did not do (see below)
* restart the services
* following the instructions here that are recommended as solution for the 48hour-statistic-bug. This is about recreating logger.sqlitedb and the rrd-files which sounds quite close to my problem but it didnt help. There is also syslog involved and I - beside the other recommendations - ran /opt/zimbra/libexec/zmsyslogsetup and restarted rsyslogd on my system ... and waited.

Actually most posts are for older zimbra-version and did not contain any information that seems useful to me.

My personal favorites for the problem are at the moment:

* The notexisting structure for zmmtastats is the problem. Then I need a way to reconstruct the "structure for zmmtastats". If I only knew which structure and where. sqlite-db? rrd?
* syslogd has something to do with it.

you mentioned reinitialising the stats-service. Sounds great, but how is it done? What does it do?


for the syslog-stuff: One user recommends (for a centOS-system) that rsyslogd is the wrong package for zimbra, but that installing syslog-ng instead solved his problem. I did not try that cause I think that any decent linux will install a working syslog-daemon by default. Thats a very old linux-tool and I dont think that zimbra would work with one and not with the other.
There are bugreports from 2009 that correlate problems to rsyslogd and logrotate, but I somehow think old problems like this are fixed now.

some "solutions" for the problem here in the forum go into changing some code which I also didnt risk to do.

I also found that my zmstat/zmstat.out exclusively contains the following lines (many many MBs of this line)

Use of uninitialized value in subtraction (-) at /opt/zimbra/libexec/zmstat-allprocs line 118

which also seems ... weird. People recommend changing the code of zmstat-allprocs, but as beeing a perlmonk by myself I dont think this would solve any problem cause this message merely shows a problem or is just ignorable. Changing the substraction-code so it does not show an error would not anything.

So I still dont have any idea how to solve my problem.

Any additional hints are highly welcome including help on tracking the problem down. Feeling lost here. Reading many postings that mostly are 1 or 2 years old but still no solid ground on where I could really start tracking my problem down.

peter

[ccelis5215]

Have the same problem since october 1.

Again, any help will be preciated.

[rgv151]

same problem here.

[myriad]

Add me to the list... Don't want to monkey with a working installation and mess something up!

[mickier]

yup me too -

I've read all the posts I can find related to stats/uninitialized value... on this board before posting...

hope I can offer some more info to help troubleshooting...
For me, stats dies occasionally (after several days) and when I restart it - it floods my zmstat/zmstat.out with the "Use of uninitialized value in subtraction (-) at /opt/zimbra/libexec/zmstat-allprocs line 118." that many complain about.

I'm running a new install ZCS 7.1.3 on a new HP server running Centos6 x64, with 8cores and 32 gb ram.

currently we have only 5 active users - although I'm setting this server up for 2000 user.

Almost all posts on this topic seem to be relating to Zimbra 6 and have attributed the problem to an UPGRADE. my server is a new server and a new install. most other postsers have no stats or it dies every day.
I actually have stats - no problem - as long as the stats module's running

I did setperms and followed the suggestions re: syslog vs rsyslog - looks like zimbra has that one fixed in 7.1.3 - it calls rsyslog and terminates by calling syslog.pid - correctly.
Since stats is working, I have not done the "delete your logger.sqlitedb & rrds and build the config..."

Not sure where to go from here - but am concerned that it might become a bigger problem later once my users really start hammering this server(!)

[rgv151]

I wanna confirm that the problem is gone after I upgraded Zimbra to 7.1.4

[pacsteel]

Same problem here, on 7.1.4 NE. Like mickier, This is on a fresh install, not an
upgrade

Originally Posted by rgv151

I wanna confirm that the problem is gone after I upgraded Zimbra to 7.1.4

Not for me. My zmstat/zmstat.out also contains bazillions of:

Use of uninitialized value in subtraction (-) at /opt/zimbra/libexec/zmstat-allprocs line 118.