Page 1 of 2 12 LastLast
Results 1 to 10 of 15

Thread: High cpu caused by zmstatuslog

  1. #1
    iniazi is offline Member
    Join Date
    Sep 2008
    Location
    Southern California
    Posts
    10
    Rep Power
    6

    Default High cpu caused by zmstatuslog

    Hello,

    New to ZCS (OSS version). Using zcs-6.0.2 (although same issue with 6.0.1, 6.0 beta, and 5.0.18. The server is also running bind and is our primary dns server. Before zcs install, the load avg was 0.01, very low IO. Actually IO continues to be low, even during high cpu issues described below.

    Machine spec: dual cpu 3.4 ghz p4 xeons (hyperthreaded) with 4gb ram, hardware Raid1 (hardware) with lvm on top. os is centos 5.3

    I see some java processes (ProvUtil?) that run one after another, with slight variations in their command line args:

    136 23056 23055 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l gacf
    79.5 23377 23376 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l gs zimbra.example.com
    99.6 23375 23231 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil gas mta
    12.0 23755 9085 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l garpu
    45.5 24240 9085 zimbra /opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common:/opt/zimbra/lib/ext/clamscanner com.zimbra.cs.account.ProvUtil -l garpb

    (results from `ps -eo pcpu,pid,ppid,user,args`)

    They peg one of the two cpus for about 40 or 50 seconds before going quiet for 1 min and 10 or 20 seconds. I think they are invocations of zmstatuslog. I've searched and read several high cpu load posts, but they seem to be talking about zmlogger issues. I did see those issues as well on the older releases, but on this release, zmlogger is quiet. Also during the high cpu/load, I/O load is pretty low. During idle, i see about 0 to 30 blks/sec and during load its 0 to 70 blk/s but (although sometimes for a second or two its 100 or 150 blk/s).

    This is a new installation, enabled zimbra proxy/memcached, enabled mixed http/https for proxy and webmail. Just one domain (example.com, for example which was configured in the install) which currently has primary mx going to mail.example.com (current live postfix box) and secondary mx going to zimbra.example.com (new zimbra install). Haven't done any other configuration or migration yet.

    It seems like its kind of a high load for an idle machine. I'm also kind of unclear what its exactly doing. If it is the zmstatuslog, if I change this to every hour (or more or less), what effect will this have? Traffic on our current server is very light. Accept about 1500 messages per day for delivery (which is actually 5% of total delivery attempts; the rest being rejected by rbl/anti-relaying rules etc). Oh also our avg message size is 13k. We do have a lot of domains (between 400 and 500), of which couple of addresses are aliased to a handfull of account (which are parsed by an trouble ticketing system).

    I understand that there will be overhead for tomcat, java, and of course some processes that occasionally do cleanup etc. But on a fresh install, is this high load normal? Or is there some type of misconfiguration (of the zcs, possibly OS)? And although our mail volume will be very low, what can I expect the load profile, in rough terms, when do migrate over?

    Sorry for making this post long; just was trying to cover everything relevant. But let me know if I left anything out.

    I really appreciate you reading and any input you can give me.

    Thanks,
    Imran

  2. #2
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,470
    Rep Power
    56

    Default

    Quote Originally Posted by iniazi View Post
    This is a new installation, enabled zimbra proxy/memcached....
    That should not be enabled or installed on a single server, you should remove it (details in several forum posts by me) if this isn't a multi-server installation.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  3. #3
    iniazi is offline Member
    Join Date
    Sep 2008
    Location
    Southern California
    Posts
    10
    Rep Power
    6

    Default

    Quote Originally Posted by phoenix View Post
    That should not be enabled or installed on a single server, you should remove it (details in several forum posts by me) if this isn't a multi-server installation.
    Ok. I'll reinstall without that. I forgot that I originally installed that to see what exactly the proxy module did. So it only for multi-server install? The wiki page isn't clear about the why.

    Going to reinstall without proxy/memcached.

    Thanks.

  4. #4
    iniazi is offline Member
    Join Date
    Sep 2008
    Location
    Southern California
    Posts
    10
    Rep Power
    6

    Default

    And that didn't change the high cpu problem, btw. I guess I just need to figure out why the ProvUtil invocations create high load.

  5. #5
    stringi is offline New Member
    Join Date
    Oct 2009
    Posts
    4
    Rep Power
    5

    Default

    I have the same problem.Both on Debian 5 and CenOS 5.3/5.4. Every 2 minutes high CPU load. I didn't install proxy/memcached.

  6. #6
    Des
    Des is offline Member
    Join Date
    Sep 2009
    Posts
    12
    Rep Power
    5

    Default

    Quote Originally Posted by stringi View Post
    I have the same problem.Both on Debian 5 and CenOS 5.3/5.4. Every 2 minutes high CPU load. I didn't install proxy/memcached.
    I think that might be down to zmmtaconfig. I turned that off using zmmtaconfigctl and the peaks went away. zmmtaconfig is a perl script that seems to check processes, copy settings from the LDAP config into the variety of of config files that make up Zimbra*.

    *expert advice appreciated!! It would be nice to have a detailed description of the process architecture.

    Having said that, I don't think you can leave zmmtaconfig turned off without understanding the consequences - I don't, so I won't. My experiments were only to find out what was causing the spikes. I get a bit fidgety when I see a functionally idle system stabbing the CPU for some reason.

    In the end, you just have to trial it and keep a close eye on it until you begin to trust it.

  7. #7
    iniazi is offline Member
    Join Date
    Sep 2008
    Location
    Southern California
    Posts
    10
    Rep Power
    6

    Default

    I'm not sure of zmmtaconfig but "com.zimbra.cs.account.ProvUtil" as I said above is the main high cpu culprit. Same as calling zmprov to list all accounts, config etc. I changed zmmtaconfig_interval (zmlocalconfig -e zmmtaconfig_interval=300). and now the high cpu load that lasts for 40 or 50 seconds happens every 5min instead of 2. There is a small spike that lasts for 12 to 15 seconds that still happens every 2 minutes.

    Anyway, I think I had determined that what it was doing is calling zmprov multiple times. As Des said, it seems to be reading ldap configs and generating local configs. Possible that only certain machines are affected with slow zmprov invocation. running 'zmprov -l gaa' takes about 5.7 seconds:

    real 0m5.731s
    user 0m9.792s
    sys 0m0.468s

    Only accounts I have (since its a fresh installation) is the admin, wiki, spam and ham. Wonder how long it takes to run "zmprov -l gaa" for others... with or without the same problem...

  8. #8
    stringi is offline New Member
    Join Date
    Oct 2009
    Posts
    4
    Rep Power
    5

    Default

    Quote Originally Posted by iniazi View Post
    ... Wonder how long it takes to run "zmprov -l gaa" for others... with or without the same problem...
    With 59 accounts:

    real 0m4.957s
    user 0m1.282s
    sys 0m1.586s

    I turned zmmtaconfigctl off and now CPU usage looks like this:

  9. #9
    stringi is offline New Member
    Join Date
    Oct 2009
    Posts
    4
    Rep Power
    5

    Default

    With 59 accounts time is:
    real 0m4.957s
    user 0m1.282s
    sys 0m1.586s

    I turned zmmtaconfigctl off and now CPU usage is smaller and it takes less time. See attachment...
    Attached Images Attached Images

  10. #10
    liverpoolfcfan's Avatar
    liverpoolfcfan is offline Outstanding Member
    Join Date
    Oct 2009
    Location
    Dublin, IRELAND
    Posts
    710
    Rep Power
    6

    Default

    Wow - what a difference. I just tried turning it off on my server, and CPU usage dropped right off.

    Found this old article on web

    Zimbra antivirus not started | Papandut.Com: like IT, blog IT, share IT, help with IT

    which talked about the schedule interval for zmmtaconfig.

    I took a look inside zmmtaconfig on 6.0.2 and found the following

    Code:
    # get some of our config from localconfig
    getLocalConfig();
    #print Dumper $state->{localconfig};
    
    $config{loglevel} = $state->{localconfig}->{zmmtaconfig_log_level} || 3;
    $config{interval} = $state->{localconfig}->{zmmtaconfig_interval} || 60;
    $config{debug} = $state->{localconfig}->{zmmtaconfig_debug} || 0;
    $config{restartconfig} =
      ($state->{localconfig}->{zmmtaconfig_enable_config_restarts} eq "FALSE") ? 0 : 1;
    so it appears that if you don't specify a local configuration setting of zmmtaconfig_interval then it will default to running every 60 seconds.

    On my system this value did not exist in localconfig.

    Perhaps with the interval so short, the process has not finished one execution before it is being kicked off again.

    I ran

    zmlocalconfig -e zmmtaconfig_interval=300
    zmmtaconfig restart

    and watched TOP for a while, and saw no more spikes.


    From the zmmtaconfig log file, it seems to be just checking for the existence of the anti virus - but maybe it is doing other stuff too. I didn't enable debug to try to figure out.

    The original poster of the web article mentioned had set an interval of 180 so perhaps that would work too.

    It would be interesting to know if there is a good reason for defaulting to 60 seconds as an interval.
    Last edited by liverpoolfcfan; 11-18-2009 at 09:51 AM.

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. High CPU usage on server and client
    By jack-Z in forum Administrators
    Replies: 10
    Last Post: 01-07-2009, 10:12 PM
  2. High CPU usage, Massive slow down
    By deoriger in forum Administrators
    Replies: 5
    Last Post: 11-20-2008, 09:31 AM
  3. Loggerd Causing High CPU and Memory
    By sujithmk in forum Administrators
    Replies: 2
    Last Post: 06-26-2008, 07:12 PM
  4. HIGH CPU use
    By brwatters in forum Administrators
    Replies: 3
    Last Post: 07-05-2006, 08:47 AM
  5. 3.1 on FC4 problems
    By cohnhead in forum Installation
    Replies: 8
    Last Post: 05-26-2006, 11:16 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •