CMS: abort preclean due to time/concurrent mode failure

[jocx]

Hello.
With Release 6.0.1_GA_1816.UBUNTU8 UBUNTU8 NETWORK edition.
we run several days just fine, and then
the processor time charged to the zimbra java process jumps up.
User experience with the web interface suffers.

in zmmailboxd.out I've noticed the incidence of two particular message
increase: "CMS: abort preclean due to time" and "concurrent mode failure"

615284.919: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 615289.931: [CMS-concurrent-abortable-preclean: 0.275/5.012 secs] [Times: user=0.30 sys=0.06, real=5.02 secs]
615289.932: [GC[YG occupancy: 386350 K (554816 K)]615289.933: [Rescan (parallel) , 1.0865790 secs]615291.019: [weak refs processing, 0.0000270 secs] [1 CMS-remark: 1850959K(1851392K)] 2237310K(2406208K), 1.0871770 secs] [Times: user=0.48 sys=0.39, real=1.09 secs]
Total time for which application threads were stopped: 1.0882570 seconds

This has occurred 2100 times in the last almost 11+ hours. Yet only
0-3 times in earlier days after a restart.

and the second:

587640.400: [GC 587640.400: [ParNew: 554815K->554815K(554816K), 0.0000810 secs]587640.401: [CMS587640.401: [CMS-concurrent-abortable-preclean: 0.236/4.157 secs] [Times: user=0.31 sys=0.23, real=4.16 secs]
(concurrent mode failure)[Unloading class sun.reflect.GeneratedSerializationConstructorAcces sor11]
[Unloading class sun.reflect.GeneratedConstructorAccessor190]
[Unloading class sun.reflect.GeneratedMethodAccessor469]
[Unloading class sun.reflect.GeneratedMethodAccessor456]
[Unloading class sun.reflect.GeneratedMethodAccessor468]
[Unloading class sun.reflect.GeneratedConstructorAccessor191]
: 1837879K->1851392K(1851392K), 12.7999000 secs] 2392695K->1899378K(2406208K), [CMS Perm : 48919K->48900K(131072K)], 12.8004090 secs] [Times: user=10.79 sys=0.08, real=12.80 secs]
Total time for which application threads were stopped: 12.8018620 seconds


We have 5 GB memory (tuned up as a vm from 3GB after the 6.0 upgrade)
and are the zimbra java process is running with:
/opt/zimbra/java/bin/java -server -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+UseConcMarkSweepGC -XX:PermSize=128m -Xss256k -Xms2409m -Xmx2409m -Xmn602m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar /opt/zimbra/mailboxd/etc/jetty.properties /opt/zimbra/mailboxd/etc/jetty-setuid.xml /opt/zimbra/mailboxd/etc/jetty.xml

Can I restart this process with minimal/no effect to the user's web experience?
Can someone share insight into what's going on, and what action
I might take?

thanks,
John

[ArcaneMagus]

With times like [Times: user=0.30 sys=0.06, real=5.02 secs] where it only used .36 seconds of processor time, but 5.02 seconds of total time you either have a different process using a ton of cpu, or this process took forever waiting on disk access.

You might want to check the disk subsystem this is running on.

That or something else entirely is wrong...

[ArcaneMagus]

Also as that process is the main mailbox process it will have an affect on user experience, and restarting just it might cause issues with other services that are depending on it.

You should be able to restart it with "zmmailboxdctl stop; zmmailboxdctl start" but as I said... I have no idea what the effect of this on the rest of the parts of the server will be.

[jocx]

Hi.
It got back to normal after a restart, but I expect it will bog down
again sometime. Expanded thoughts:
I can't account for the wall clock delay, and I appreciate the
consideration of processor bound server or confounding disk subsystem.
A review of the CPU history shows things are nominal except
for the java process going heavy into CPU and the disk subsystem
seems just fine.
Tonight I scheduled a short outage to zmmailboxdctl stop,
wait (verify process ends, etc), and a subsequent zmmailboxdctl start.
zmmailboxd.out looks normal again, without the CMS preclean or concurrent
mode failures. And the processor load is back to normal. So seems the
zmmailboxd restart cleared up the issue. I'm expecting to see the
issue again, as this is now twice I've seen this condition arise.
I ran zmmailboxdmgr threaddump per
King0770-Notes - Zimbra :: Wiki
while it was bogged down, and after the restart. I'll attach the
log segments for two threaddumps.
Thanks for any thoughts.
-John

[jocx]

I created a bugzilla entry for this issue...
Bug 42854 – Parnew promotion failed, Java garbage issues, CMS: abort preclean due to time/concurrent mode failure