High cpu caused by zmstatuslog

[iniazi]

Hello,

New to ZCS (OSS version). Using zcs-6.0.2 (although same issue with 6.0.1, 6.0 beta, and 5.0.18. The server is also running bind and is our primary dns server. Before zcs install, the load avg was 0.01, very low IO. Actually IO continues to be low, even during high cpu issues described below.

Machine spec: dual cpu 3.4 ghz p4 xeons (hyperthreaded) with 4gb ram, hardware Raid1 (hardware) with lvm on top. os is centos 5.3

I see some java processes (ProvUtil?) that run one after another, with slight variations in their command line args:

136 23056 23055 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l gacf
79.5 23377 23376 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l gs zimbra.example.com
99.6 23375 23231 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil gas mta
12.0 23755 9085 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l garpu
45.5 24240 9085 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l garpb

(results from `ps -eo pcpu,pid,ppid,user,args`)

They peg one of the two cpus for about 40 or 50 seconds before going quiet for 1 min and 10 or 20 seconds. I think they are invocations of zmstatuslog. I've searched and read several high cpu load posts, but they seem to be talking about zmlogger issues. I did see those issues as well on the older releases, but on this release, zmlogger is quiet. Also during the high cpu/load, I/O load is pretty low. During idle, i see about 0 to 30 blks/sec and during load its 0 to 70 blk/s but (although sometimes for a second or two its 100 or 150 blk/s).

This is a new installation, enabled zimbra proxy/memcached, enabled mixed http/https for proxy and webmail. Just one domain (example.com, for example which was configured in the install) which currently has primary mx going to mail.example.com (current live postfix box) and secondary mx going to zimbra.example.com (new zimbra install). Haven't done any other configuration or migration yet.

It seems like its kind of a high load for an idle machine. I'm also kind of unclear what its exactly doing. If it is the zmstatuslog, if I change this to every hour (or more or less), what effect will this have? Traffic on our current server is very light. Accept about 1500 messages per day for delivery (which is actually 5% of total delivery attempts; the rest being rejected by rbl/anti-relaying rules etc). Oh also our avg message size is 13k. We do have a lot of domains (between 400 and 500), of which couple of addresses are aliased to a handfull of account (which are parsed by an trouble ticketing system).

I understand that there will be overhead for tomcat, java, and of course some processes that occasionally do cleanup etc. But on a fresh install, is this high load normal? Or is there some type of misconfiguration (of the zcs, possibly OS)? And although our mail volume will be very low, what can I expect the load profile, in rough terms, when do migrate over?

Sorry for making this post long; just was trying to cover everything relevant. But let me know if I left anything out.

I really appreciate you reading and any input you can give me.

Thanks,
Imran

[phoenix]

Originally Posted by iniazi

This is a new installation, enabled zimbra proxy/memcached....

That should not be enabled or installed on a single server, you should remove it (details in several forum posts by me) if this isn't a multi-server installation.

[iniazi]

Originally Posted by phoenix

That should not be enabled or installed on a single server, you should remove it (details in several forum posts by me) if this isn't a multi-server installation.

Ok. I'll reinstall without that. I forgot that I originally installed that to see what exactly the proxy module did. So it only for multi-server install? The wiki page isn't clear about the why.

Going to reinstall without proxy/memcached.

Thanks.

[iniazi]

And that didn't change the high cpu problem, btw. I guess I just need to figure out why the ProvUtil invocations create high load.

[stringi]

I have the same problem.Both on Debian 5 and CenOS 5.3/5.4. Every 2 minutes high CPU load. I didn't install proxy/memcached.

[Des]

Originally Posted by stringi

I have the same problem.Both on Debian 5 and CenOS 5.3/5.4. Every 2 minutes high CPU load. I didn't install proxy/memcached.

I think that might be down to zmmtaconfig. I turned that off using zmmtaconfigctl and the peaks went away. zmmtaconfig is a perl script that seems to check processes, copy settings from the LDAP config into the variety of of config files that make up Zimbra*.

*expert advice appreciated!! It would be nice to have a detailed description of the process architecture.

Having said that, I don't think you can leave zmmtaconfig turned off without understanding the consequences - I don't, so I won't. My experiments were only to find out what was causing the spikes. I get a bit fidgety when I see a functionally idle system stabbing the CPU for some reason.

In the end, you just have to trial it and keep a close eye on it until you begin to trust it.

[iniazi]

I'm not sure of zmmtaconfig but "com.zimbra.cs.account.ProvUtil" as I said above is the main high cpu culprit. Same as calling zmprov to list all accounts, config etc. I changed zmmtaconfig_interval (zmlocalconfig -e zmmtaconfig_interval=300). and now the high cpu load that lasts for 40 or 50 seconds happens every 5min instead of 2. There is a small spike that lasts for 12 to 15 seconds that still happens every 2 minutes.

Anyway, I think I had determined that what it was doing is calling zmprov multiple times. As Des said, it seems to be reading ldap configs and generating local configs. Possible that only certain machines are affected with slow zmprov invocation. running 'zmprov -l gaa' takes about 5.7 seconds:

real 0m5.731s
user 0m9.792s
sys 0m0.468s

Only accounts I have (since its a fresh installation) is the admin, wiki, spam and ham. Wonder how long it takes to run "zmprov -l gaa" for others... with or without the same problem...

[stringi]

Originally Posted by iniazi

... Wonder how long it takes to run "zmprov -l gaa" for others... with or without the same problem...

With 59 accounts:

real 0m4.957s
user 0m1.282s
sys 0m1.586s

I turned zmmtaconfigctl off and now CPU usage looks like this:

[stringi]

With 59 accounts time is:
real 0m4.957s
user 0m1.282s
sys 0m1.586s

I turned zmmtaconfigctl off and now CPU usage is smaller and it takes less time. See attachment...

[liverpoolfcfan]

Wow - what a difference. I just tried turning it off on my server, and CPU usage dropped right off.

Found this old article on web

Zimbra antivirus not started | Papandut.Com: like IT, blog IT, share IT, help with IT

which talked about the schedule interval for zmmtaconfig.

I took a look inside zmmtaconfig on 6.0.2 and found the following

# get some of our config from localconfig
getLocalConfig();
#print Dumper $state->{localconfig};

$config{loglevel} = $state->{localconfig}->{zmmtaconfig_log_level} || 3;
$config{interval} = $state->{localconfig}->{zmmtaconfig_interval} || 60;
$config{debug} = $state->{localconfig}->{zmmtaconfig_debug} || 0;
$config{restartconfig} =
  ($state->{localconfig}->{zmmtaconfig_enable_config_restarts} eq "FALSE") ? 0 : 1;

so it appears that if you don't specify a local configuration setting of zmmtaconfig_interval then it will default to running every 60 seconds.

On my system this value did not exist in localconfig.

Perhaps with the interval so short, the process has not finished one execution before it is being kicked off again.

I ran

zmlocalconfig -e zmmtaconfig_interval=300
zmmtaconfig restart

and watched TOP for a while, and saw no more spikes.


From the zmmtaconfig log file, it seems to be just checking for the existence of the anti virus - but maybe it is doing other stuff too. I didn't enable debug to try to figure out.

The original poster of the web article mentioned had set an interval of 180 so perhaps that would work too.

It would be interesting to know if there is a good reason for defaulting to 60 seconds as an interval.