zimbra server becomes freezes (cpu load >7)

[fisch09]

Hi all,

I have a ZCS NE 4.5.6 evaluation on Ubuntu 6.06 and since yesterday (no changes at all) it locks up after a while. One java process just eats CPU and keeps going. I do not know where to start troubleshooting. Any help appreciated.

Kind Regards
Andre

[fisch09]

some more details:

it takes about 20min from a zimbra restart until the CPU load starts to increase

top:

top - 20:56:24 up  3:46,  2 users,  load average: 2.71, 1.70, 1.19
Tasks: 102 total,   2 running, 100 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.3% us,  0.7% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   1034656k total,   908240k used,   126416k free,    21944k buffers
Swap:  2096376k total,      112k used,  2096264k free,   254524k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                       
 9875 zimbra    17   0  720m 247m  13m S 99.7 24.5   4:34.57 java                                                                                                                                                          
14297 root      16   0  2196 1116  856 R  0.3  0.1   0:00.03 top                                                                                                                                                           
    1 root      16   0  1468  472  412 S  0.0  0.0   0:01.11 init

and this is the process:

zimbra    9875 16.7 34.0 737852 352584 ?       Sl   20:18   6:45  \_ /opt/zimbra/jdk1.5.0_08/bin/java -Xms404m -Xmx404m -client -XX:NewRatio=2 -Djava.awt.headless=true -Dcatalina.base=/opt/zimbra/apache-tomcat-5.5.15 -Dcatalina.home=/opt/zimbra/apache-tomcat-5.5.15 -Djava.io.tmpdir=/opt/zimbra/apache-tomcat-5.5.15/temp -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/apache-tomcat-5.5.15/common/endorsed -classpath /opt/zimbra/apache-tomcat-5.5.15/bin/bootstrap.jar:/opt/zimbra/apache-tomcat-5.5.15/bin/commons-logging-api.jar:/opt/zimbra/lib/jars/zimbra-launcher.jar com.zimbra.cs.launcher.TomcatLauncher

and 15min later top looks like this:

top - 21:11:23 up  4:01,  2 users,  load average: 8.46, 5.00, 2.96
Tasks: 104 total,   3 running, 101 sleeping,   0 stopped,   0 zombie
Cpu(s): 94.3% us,  5.7% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   1034656k total,   987752k used,    46904k free,    14460k buffers
Swap:  2096376k total,      248k used,  2096128k free,   201964k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                       
 9875 zimbra    17   0  721m 352m  13m S 93.7 34.9  18:56.38 java                                                                                                                                                          
15713 zimbra    25   0  9144 5408 2140 R  2.0  0.5   0:01.31 zmgengraphs                                                                                                                                                   
    1 root      15   0  1468  472  412 S  0.0  0.0   0:01.11 init

I am watching the logs but can't see anything unusual there

please help.....

[KevinH]

What is the server like? CPU? Disk? Mem?

[fisch09]

hardware:

CPU: AMD Athlon 64 3700+
MEM: 1GB
HDD: 2x160GB SATA as soft raid 1

OS: 32bit Ubuntu 6.06 LTS (up2date)

Usage: about 10mailboxes, this happens even if the server not actively used by users and there is less then 1 email per minute (in/out combined). It was working fine until 1.5days ago - there have been not software/harware/config changes at all (no new mailboxes, nothing)

If you need any logs, or more details let me know

Kind Regards
Andre

[fisch09]

I restarted now 4 times and it is always the same about 25-30min after restart the zimbra java process takes 99% CPU and the load is increasing until it freezes (not the whole system, I am still able to send a "reboot" via ssh)
The only connected user has been myself for the last 2 times.
I had open: web interface, isync client, entourage (imap).

I am reading logs for hours now - i can't see anything that is unusual.........

Cheers
Andre

[fisch09]

Hi all,

after a long night troubleshooting and finding absolutely nothing in logs or any other indication a "lsof -i -P" gave the important hint. It took ages to list the ipv6 connections that belong to the zimbra java process...
Ok - I do not need ipv6, so I disabled it and rebooted the server and now everything looks fine again.

Summary:
32bit Ubuntu 6.06LTS on 64Bit hardware + ZCS 4.5.6 NE was running without any trouble for 30days. Then the reproduceable problem - continuously increasing CPU load - started. There have been no hardware/software changes.
Disabling ipv6 - solved the issue.

Kind Regards
Andre

[jooray]

The same thing happens here. I run under VMWare with 1.7GB RAM allocated to Zimbra instance (open source edition). It operates about 70 mailboxes.

Sometimes, load goes very high (>4) for unknown reason. The ram is not eaten up.

This is how "top" looks:

top - 13:12:05 up 1 day, 12 min, 1 user, load average: 3.88, 1.53, 0.66
Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.3% us, 4.3% sy, 0.0% ni, 92.8% id, 0.4% wa, 0.0% hi, 0.2% si
Mem: 1815948k total, 1544348k used, 271600k free, 67580k buffers
Swap: 939760k total, 0k used, 939760k free, 456360k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17820 zimbra 15 0 256m 14m 8752 S 56 0.8 0:01.09 java
8878 zimbra 17 0 1008m 357m 13m S 53 20.2 7:09.78 java
17848 zimbra 18 0 59516 48m 1732 S 13 2.7 0:00.14 amavisd
11914 zimbra 25 0 9232 5464 2140 R 5 0.3 0:20.15 zmgengraphs

[fisch09]

I run ZCS opensource on identical hardware and have more mailboxes there and ipv6 is still enabled - running for 9month now without any issues.

Cheers
Andre