zcs 7.1.2 high CPU loading used by java process

[cadi-its]

Hi all,

The following java process using almost 100% CPU resource, and I can't find a correct solution, any one can help? Thanks a lot.

zimbra    7882 54.8 19.1 1471388 392812 ?      Sl   09:03 224:47 /opt/zimbra/java/bin/java -Dfile.encoding=UTF-8 -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xss256k -Xms512m -Xmx512m -Xmn128m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar /opt/zimbra/mailboxd/etc/jetty.properties /opt/zimbra/mailboxd/etc/jetty-setuid.xml /opt/zimbra/mailboxd/etc/jetty.xml

[phoenix]

Originally Posted by cadi-its

The following java process using almost 100% CPU resource, and I can't find a correct solution, any one can help?

This has been covered in the forums, you need to upgrade to the most recent Zimbra release.

[bofh]

same issue here - specially with bigger attachments about 1-2mb
and it is the latest realease already so its not solved

[cadi-its]

it seems the problem is fixed by 7.1.3 upgrade

[Klug]

There's a patch for 7.1.3 (not just 7.1.3) that helps fixing this issue.

[ulaaser]

Hi,

i have installed newest ZCS with the patch in Centos 6.0 64bit inside Esxi 4.0 with 2vCPU and 6GB RAM.

I still getting high cpu loads... and after a few hours,server becomes unresponsive and i have to connect trough esxi to restart the server, after that the server works fine again for a few hours...

Release 7.1.3_GA_3346.RHEL6_64_20110928134428 CentOS6_64 FOSS edition, Patch 7.1.3_P1.

executing "top" i have (more and more) multiple instances of /opt/zimbra/java/bin/java -version (each of them trying to maxout cpu usage!)

Can i stop this processes manually... kill -procid?

[ulaaser]

Well i see that there are being created several java processes (using top)

all of them maxing out cpu usage permanently...
in normal situation the java proceses runs only seconds and are closed normally, but some processes are not closed....
specially the ones calles java -version
so i created a crontab using
crontab -e as root.

and put inside the following line:
* */2 * * * kill -9 `ps -ef | grep 'java -version' | grep -v grep | awk '{print $2}'`

so in my case im killing this processes every 2 hours...
and because i am not so overloaded with users, i also do a reboot at 3:50am

50 3 * * * /sbin/reboot

i also was trying to kill another cpu hungry thread with

kill -9 `ps -ef | grep 'java -XX' | grep -v grep | awk '{print $2}'`

but this causes the zmconfigd to stop...

so the first 2 lines are enough for me...

[Brad46526]

I have this exact same problem, running what I believe is the latest version of ZCS on Centos 6 - version 7.1.3 with the 7.1.3_P1 patch.

'zmcontrol -v' shows the version of ZCS that I am running:

Release 7.1.3_GA_3346.RHEL6_64_20110928134428 CentOS6_64 FOSS edition, Patch 7.1.3_P1.

I find that, every few hours, I'll have a java process in an infinite loop, taking 100% of a CPU. Usually the process is simply running java with a '-version' option (called by zmlocalconfig); once or twice it's been a java process kicked off by something else.

Sending a SIGHUP or SIGTERM to the java process doesn't affect it; I have to slay it with a SIGKILL. Which kills the process and returns my machine's CPU usage to normal ... until the next time.

Right after killing the runaway process I can manually run the command pipeline that zmlocalconfig invokes:

/opt/zimbra/java/bin/java -version 2>&1 | grep "java version" | sed -e 's/"//g' | awk '{print $NF}' | awk -F_ '{print $1}'

But it runs fine (printing 1.6.0 to stdout).

Is there a fix for this problem? Thanks!

EDITED TO ADD:

Just 30 mins or so later another runaway java process has taken off on my ZCS server; this time it's been kicked off by zmstatctl; here's a line of output from 'pstree -p':

-java(13587)-+-zmstatctl(29602)---java(29605)-+-{java}(29625)

In this case the java process has been passed more than just the '-version' option:

/opt/zimbra/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -cp /opt/zimbra/lib/jars/zimbracommon.jar:/opt/zimbra/lib/jars/zimbrasoap ...

It seems to be an intermittent problem with basic java itself?

I'm running Centos 6; this is the output of 'uname -a':

Linux centos6.test.com 2.6.32-71.el6.x86_64 #1 SMP Fri May 20 03:51:51 BST 2011 x86_64 x86_64 x86_64 GNU/Linux

Thanks for any help!

[Klug]

Might also be related to this (fixed in 7.1.4) : Bug 65408 – mime parser: no upper bound in ByteArrayOutputStream may lead to tmp memory explosion for some mal-formed message

[Brad46526]

Originally Posted by Klug

Might also be related to this (fixed in 7.1.4) : Bug 65408 – mime parser: no upper bound in ByteArrayOutputStream may lead to tmp memory explosion for some mal-formed message

Hmm. My ZCS server is totally quiescent; it's a test server with no message traffic, no users, at all at the moment. Just its own internal processes, which includes the periodic stats collection and other things which run and occasionally kick off an infinite-looping java process.

Still, I didn't know that 7.1.4 is out (novice Zimbra admin here); I'll have to aggressively look for and apply upgrades if the problem persists.