[SOLVED] Zimbra v6.03 unusable on CentOS 5.4/KVM - java hanging on futex lock!

[hobleyd]

Hello,

We have been running Zimbra on a 32 bit CentOS 5.4 install (4 Gig of memory), on a physical machine. It has been working like a dream and we are very happy with it.

Having read Zimbra's intention to migrate to the 64 bit version exclusively at some point, we thought we ought to test out the 64 bit version and look at the migration path. So we installed a 64 bit CentOS virtual machine (8 Gig memory, using the KVM hypervisor) and tried to install Zimbra 6.03.

Unfortunately the 64 bit java version included with Zimbra is pretty much unusable out of the box - we are seeing extensive hangs with one of the virtual CPUs maxing out at 100%. When this happens, we strace the Java process and we see:

clone(child_stack=0x403e5250, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND| CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARE NT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x403e59d0, tls=0x403e5940, child_tidptr=0x403e59d0) = 5828
futex(0x403e59d0, FUTEX_WAIT, 5828, NULL

The end result is that any Zimbra process that gets kicked off and runs a Java program, there is a high likelihood that it will hang for ages (min 10s of minutes to several hours). Not good.

I haven't seen much discussion about this at all, but it is so reproducible for us, I am surprised that nothing has been mentioned. Is anyone else running Zimbra 64bit on a 64 bit CentOS installation? Or a RHEL 5.4 installation for that matter?

Cheers,
David

[Klug]

ZCS 64 on Centos works very well.

However, KVM is not supported at the moment.
So no clue if it will work under a KVM VM (or not)...

[hobleyd]

Originally Posted by Klug

ZCS 64 on Centos works very well.

However, KVM is not supported at the moment.
So no clue if it will work under a KVM VM (or not)...

Ok, thanks. Where can I find which Hypervisors are supported?

Cheers,
David

[phoenix]

Originally Posted by hobleyd

Ok, thanks. Where can I find which Hypervisors are supported?

VMware ESX & ESXi (Server & Workstation aren't recommended for production systems) and XEN 64bit.

[hobleyd]

All, the problems we were seeing with Java relate to this bug:

0004059: SUN java runs very long time on SMP KVM guests - CentOS Bug Tracker

Appears to be solved in CentOS 5.5!

[powrrrplay]

Originally Posted by hobleyd

Hello,

We have been running Zimbra on a 32 bit CentOS 5.4 install (4 Gig of memory), on a physical machine. It has been working like a dream and we are very happy with it.

Having read Zimbra's intention to migrate to the 64 bit version exclusively at some point, we thought we ought to test out the 64 bit version and look at the migration path. So we installed a 64 bit CentOS virtual machine (8 Gig memory, using the KVM hypervisor) and tried to install Zimbra 6.03.

Unfortunately the 64 bit java version included with Zimbra is pretty much unusable out of the box - we are seeing extensive hangs with one of the virtual CPUs maxing out at 100%. When this happens, we strace the Java process and we see:

clone(child_stack=0x403e5250, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND| CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARE NT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x403e59d0, tls=0x403e5940, child_tidptr=0x403e59d0) = 5828
futex(0x403e59d0, FUTEX_WAIT, 5828, NULL

The end result is that any Zimbra process that gets kicked off and runs a Java program, there is a high likelihood that it will hang for ages (min 10s of minutes to several hours). Not good.

I haven't seen much discussion about this at all, but it is so reproducible for us, I am surprised that nothing has been mentioned. Is anyone else running Zimbra 64bit on a 64 bit CentOS installation? Or a RHEL 5.4 installation for that matter?

Cheers,
David

David, I am running Fedora64 11/KVM and LVM with a few virtual machines. One VM is Centos64 5.4 w/ Zimbra 6.06.0. I originally installed my Centos64 as 5.2 and Zimbra 6.00 Beta.

The whole thing purrrrs. I chose Fedora64 11 over Ubuntu64 9.0 for my KVM machine because of hardware compatibility issues with my Adaptec Raid card. Otherwise I probably would be running Ubuntu today as my KVM. I also believe sticking with Centos or Redhat will keep you as safe as possible for hardware support. Other than that I was evaluating Fedora and Ubuntu because I was trying to get myself a more robust KVM than what was available at the time via Centos.

I consider myself a very aggressive maintainer, meaning that I install upgrades about 3 weeks after they have been released by the Zimbra team. This lets me watch what others go through after an upgrade and what I may have in store for myself.

I have worked through some bugs in Zimbra, and they always seem to get corrected along the way but if I could give anybody advice about upgrading these platforms I would suggest to always upgrade the OS then Zimbra in one shot - it will save you heartache. I must stress, in that order, upgrade the OS then upgrade Zimbra - never upgrade Zimbra then the OS. Of course you can upgrade just Zimbra.

One day I will probably change my Centos64 5.4 to Ubuntu64 10.4 for Zimbra