Virtualization Zimbra

[uxbod]

I have just (this weekend) switched from VMWare Server too Xen 3.1. Dom-U is 64 bit CentOS 5.1 and the guest is i386 CentOS 5.1. Even migrated my VM disk image to Qemu with no problems at all. Fingers crossed all is running sweet, and I have noticed a real improvement in performance. Load average has dropped from 1.2 down to 0.3 on both Dom-U and the guest

[whacker]

anyone tried it on VMWare ESX?
I could test it but it but atm I don't have enough time.

[uxbod]

If it works on VMWare server I am sure it will work on ESX. At the end of the day if you can install a supported distro then Zimbra should just work.

[whacker]

I'm sure it'll run but I would want to know what kind of resources to give it and if it performs almost equally as on a dedicated server.
Anyway, I'm trying it right now.

[uxbod]

From a resources perspective you would give the VM a similar setup to a normal server. Also depends on what processors you are running and whether they are AMD-V/Intel VT based one. This would then dictate if it were a virtualised or para-virtualised VM. These would offer different performance levels due to resource allocation and hardware addressing IMHO.

[lars_BE]

We use VMWARE ESX3 on three 2x CPU 2,4 Ghz Intel Xeon (Proliant DL380 G3) servers, now about 4 years old. There are about 15 virtual servers (mixed windows 2000 and 2003 and Linux) running on these 3 servers. Zimbra runs in a virtual server under RHEL 4.

Zimbra uses 30%-50% of one CPU. Have a look at the pictures.

We have 100 users, most of them use Outlook 2003 and a few the webinterface. The webinterface is not super fast, but fast enough and i have no complaints about speed. We are looking to replace hardware now.

[joshwest]

I have no intention of running a production Zimbra install under Xen any time soon, but I do want to run other production applications that depend on berkeley DB, so I care about the general issue.

Hey Rich,

We currently have Zimbra 5.0.4 running in multi-node fashion under Xen 3.1.3. We're using our own custom Xen RPM's (and therefore custom Xen 2.6.18.8 kernel) on RHEL 5.1 dom0's. Each Zimbra node also runs on RHEL 5.1 virtual machines. In total, there are 13 virtual machines running in our Xen cluster for our production mail setup:

• Two Zimbra MX Virtual Machine's
• Two split-DNS Nameserver Virtual Machine's (see Split DNS - Zimbra :: Wiki)
• Two Zimbra Proxy Virtual Machine's
• Two Zimbra LDAP Virtual Machine's (one master, one replica)
• One Zimbra Logger Virtual Machine
• Four Zimbra Mail Store Virtual Machine's

We run a 64-bit Xen hypervisor with 32-bit dom0's and domU's. The hypervisor was built 64-bit so we can utilize the full 32GB of memory on each of our 14 Dell PowerEdge 1950's within the Xen cluster. But, we stuck with 32-bit dom0's and 32-bit domU's because we require the ability to live migrate virtual machines from one dom0 to another. (You can boot a 32-bit domU on a 64-bit dom0, and vice versa, but you cannot migrate the domU; a limitation of the xc_save and xc_restore tools; see my post on the Xen-devel mailing list for details: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-b - Xen Source).

Back when we ran Xen 3.0.4-1 with RHEL 4.x dom0's and domU's, we did see issues with BDB and components which use it (OpenLDAP, MySQL). Even though we followed the common practice of moving /lib/tls to /lib/tls.disabled on our domU's, we still ran into BDB issues due to its heavy utilization of thread local storage (TLS) within the native posix threading library (NPTL) by those applications. (Just a note - to get around BDB woes with MySQL on a RHEL 4.x domU, we would enter "skip-bdb" within the /etc/my.cnf and just not use the BDB MySQL database engine).

However, since we've switched to Xen 3.1.3 and RHEL 5.1 for dom0's and domU's, provided we enter that same exact "hwcap 0 nosegneg" directive within /etc/ld.so.conf (and run ldconfig for changes to take effect), we've had zero issues with BDB and friends. From what I can tell, the "hwcap 0 nosegneg" just instructs the RHEL-specific (or stock, I'm not sure) dynamic linker to ignore and not execute the offending TLS code. From what I can see, its essentially the same as compiling one's entire system with the GCC flag "-mno-tls-direct-seg-refs", which is common for us Gentoo folks who use Xen and are compiling our entire system from scratch. I think that Red Hat preferred a more "toggleable" or dynamic way of dis/allowing the NTPL/TLS feature without fully turning it off or fully on, such that their users could leverage its features if they aren't using Xen.

For reference, from the GCC man page:

       -mno-tls-direct-seg-refs
           Controls whether TLS variables may be accessed with offsets from the TLS segment register (%gs for 32-bit, %fs for 64-bit), or whether the
           thread base pointer must be added.  Whether or not this is legal depends on the operating system, and whether it maps the segment to cover the
           entire TLS area.

           For systems that use GNU libc, the default is on.

Regarding 'quanah's post on executing:

Unfortunately, I don't know a specific test for this. You can try the following and see if it spews errors, on one of your zimbra Xen hosts:

su - zimbra
zmcontrol stop

(make sure slapd is stopped)

cd /opt/zimbra/openldap-data
/opt/zimbra/sleepycat/bin/db_recover

to test for BDB woes, we have executed such without any errors or issues. I didn't expect to see problems either, as OpenLDAP, MySQL, or Zimbra would probably not even start if TLS and Xen were conflicting (as is the case with RHEL 4.x and Xen 3.0.4-1). Additionally, Red Hat worked pretty hard to make RHEL 5.x support Xen and be Xen compatible, so I'd expect it to work (as it does).

For those folks interested in the performance of our system, it is quite outstanding. Because of the paravirtualized nature of Xen, we don't run into speed issues with the virtual processors or memory. We run many other applications within our Xen cluster that are quite busy, such as Oracle 10g, MySQL 4.x and 5.x databases, our entire Moodle infrastructure (see Welcome to LATTE - Brandeis University), PeopleSoft Financials 9.0 and Campus Solutions 9.0, RT, and so forth. I/O (network and block) performance is also quite good, as we've taken a unique approach with both:

First, the actual servers (dom0's) have both 1GB/s NIC's (eth0 and eth1) bonded (etherchannel for those Cisco folks out there) to become bond0, in order to provide high availability and up to 2GB/s aggregate throughput (we use LACP for bonding). A VLAN trunk is passed down the bond (bond0), and split out into separate bridges accordingly. Virtual machines (domU's) are then connected to any of the bridges they require access to; VM's generally need either one or two network connections out of the 20 VLAN's sent to each dom0.

Second, for block devices (storage), each virtual machine (domU) has its own set of volumes on our SAN. Xen servers (dom0's) are connected to the SAN via 4GB/s fibre channel links. We then pass those block devices directly into the virtual machines so there is no middle-layer such as the clustered file system approach like OCFS2, GFS, or VMFS. We do use multipath in order to abstract the volume names to those like /dev/mapper/vm_zimbra_store_1 instead of the real block devices names /dev/sdz (because while a volume will appear as sdz on one box, it could be sdm on another).

Rich (or anybody else), if you'd like to take a gander at our custom Xen RPM's, just let me know and I'll send them your way. We've made our own to enable quite a few additional features, such as XENFB support and a 64-bit Hypervisor, as well as for bug fixes (the Xen API in 3.1.3 is much improved over that in 3.1.0).

I'll also keep this forum updated as to our success with future versions of Xen and Zimbra.

Hope this helps.

[Klug]

Joshua, what made you choose Xen against a "commercial" solution (such as VirtualIron, commercial-Xen or VMWare) ?

[Rich Graves]

Unless you have a long-time talented Xen hacker like Joshua working for you, I would advise holding off until RHEL 5.2, expected mid-May (with CentOS 5.2 following a week or two later). Xen disk I/O under RHEL 5.1 is awful... a factor of 20 difference between RHEL 5.1 and Joshua's custom builds. The RHEL 5.2 beta is competitive with Joshua's kernels, usually only slightly slower than native hardware.

With any virtualized host, and certainly on a SAN, you want to set elevator=noop in grub.conf.

[Armelito]

Hi, I just joined the Forum and can't find anywhere any reference to ther VM environnement such as IBM z/VM. Any experience ? I am working on a server consolidation onto IBM Mainframe. The current testing environnement is zVM 5.3 and RHEL5. WE are not sure if Zimbra Network Edition is supported or how to have it supported?
Thanks.