Zimbra responds slowly from time to time

[jooray]

Hello,

we have a problem with Zimbra network edition. From time to time, "java" process takes 90% of CPU time and 33% of memory.

I did jps and it's this process:

31825 com.zimbra.cs.launcher.TomcatLauncher -Xms709m -Xmx709m -XX:NewRatio=2 -Djava.awt.headless=true -Dcatalina.base=/opt/zimbra/apache-tomcat-5.5.15 -Dcatalina.home=/opt/zimbra/apache-tomcat-5.5.15 -Djava.io.tmpdir=/opt/zimbra/apache-tomcat-5.5.15/temp -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/apache-tomcat-5.5.15/common/endorsed

Version is 4.5.6, running on Ubuntu.

We have 2GB of memory, 15 users. This "downtime" and high load takes
about five minutes and then continues.

How can I debug this?


Juraj.

[jooray]

Top:

top - 00:04:51 up 9 days, 18:01,  4 users,  load average: 3.47, 1.76, 0.91
Tasks: 133 total,   6 running, 127 sleeping,   0 stopped,   0 zombie
Cpu(s): 46.7% us, 51.5% sy,  0.0% ni,  0.2% id,  0.0% wa,  1.1% hi,  0.5% si
Mem:   1815948k total,  1723604k used,    92344k free,    90164k buffers
Swap:   939760k total,      108k used,   939652k free,   410928k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                            
31825 zimbra    16   0 1025m 585m  14m S   62 33.0  84:10.62 java
20315 zimbra    25   0  251m 7920 5776 R   10  0.4   0:00.30 java
20320 postfix   16   0  9104 3388 2676 S    7  0.2   0:00.21 smtpd
17597 zimbra    25   0  9156 5384 2140 R    7  0.3   0:09.30 zmgengraphs
20278 zimbra    22   0  5380 2708 1524 S    6  0.1   0:00.19 zmstatuslog

Dstat output:

----total-cpu-usage---- -disk/total -net/total- ---paging-- ---system--
usr sys idl wai hiq siq|_read write|_recv _send|__in_ _out_|_int_ _csw_
  3   5  92   0   0   0|4980B   45k|   0     0 |   0   0.1 | 125   278 
 50  49   0   0   0   1|   0   268k|2700B 9430B|   0     0 | 157   243 
 51  44   0   0   3   1|   0     0 |7970B   20k|   0     0 | 243   343 
 48  50   0   0   1   1|   0  5120B|4922B   56k|   0     0 | 227   315 
 48  51   0   0   0   0|   0     0 |1046B 1126B|   0     0 | 118   244 
 35  62   0   0   2   0|   0   272k|3248B 2575B|   0     0 | 163   460 
 60  39   0   0   1   0|   0   236k|1748B 9128B|   0     0 | 140   295

We have our /opt/zimbra/store, /opt/zimbra/index and /opt/zimbra/redolog
on NFS server (actually running Solaris 10). I ran a zpool iostat 1 check and there was absolutely no traffic to the remote storage over NFS, so Zimbra
did not touch any of these directories during that high load.

What is pretty interesting is, that there's a lot of time spent in system
(I saw 70% and also 100% few times in system cpu usage).

Anything else I can have a look at?

The Zimbra install is running under VMWare server.

[jholder]

Oh, NFS is bad.
I'm actually surprised you got it working.

Zimbra and NFS don't mix will (right now). My guess is that's where the problem is occurring.

[jooray]

Really?

Actually, when we switched to NFS, it was a lot better than before performance-wise.

Anyways, it did the same thing without NFS, I just switched to NFS recently, because we brought 4.7TB NAS, so we wanted to put all mails there. Anyways, we had the same problem before we switched to NFS. Also as
seen from the IOstat output, there was no NFS traffic in that particular time.

Now I tried this KB article from vmware:

VMware Knowledge Base - View Document

I switched VMWare to use only one CPU for each guest (we run three guests on that machine). I added noapic nolapic nosmp options. I will let it run
for a week and see if it helps.

[fisch09]

Hi all,

check this thread:
poor performance ZCS NE vs. ZCS OSS
I had the same issue (but local filesystems) and it was related to email stuck in the deferred queue.

Kind Regards
Andre

[hillman]

Originally Posted by jholder

Oh, NFS is bad.
I'm actually surprised you got it working.

This is a fairly bold statement - do you have benchmarks and configuration info to back this up?

I'm running a test implementation on NFS right now, primarily because it was an easy way to throw as much backed-up, snapshotted space into my environment as I needed. It's working fine.

It would be nice to know what has and has not been tested on NFS, and what configurations were used. For example, the storage requirements of the MySQL store are very different from the message store. Putting them both on the same type of storage for an installation of any substantial size seems likely to be less than optimal.

[bquinata]

I too am interested in utilizing an NFS resource to off load old email but using Zimbra's HSM function instead. Specifically adding a new volume declaring is a secondary volume and then mounting an nfs resource to that volume's path.

My biggest concern is Zimbra's ability to recover if network issues arise between our zimbra server and the nfs server. Whether it is a "glitch" or an outage.

[jholder]

NFS has some known issues with IO performance with Zimbra. We do not recommend using it.

If you can use something like netapp or symlinks, that's better.

[dNb]

Originally Posted by jholder

NFS has some known issues with IO performance with Zimbra. We do not recommend using it.

If you can use something like netapp or symlinks, that's better.

Sorry to dredge up this thread again, but I got a little curious this evening to see what options we had for further expanding the size of our mailstore and decided to see what was currently being said here about Zimbra and NFS.

Can I ask for a little more detail? Do you mean "some NFS client implementations" or "some NFS server implementations?" I ask because we have a netapp, and they, um, do mostly NFS for a living. Were you suggesting netapp storage doing CIFS or iSCSI when you mentioned it above or do they have an implementation that has worked well?

Are you at liberty to describe the setup where NFS had problems?

Thanks for any info you can provide, we just want to do the right thing by our little (but growing) Zimbra implementation.

-- dNb

[phoenix]

Well, there is this statement in the wiki. YMMV depending on the number of users on the system but, in general, it's really not a good idea.