Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Zimbra KVM locking up

  1. #1
    mattcen is offline Intermediate Member
    Join Date
    Mar 2010
    Location
    Melbourne, Australia
    Posts
    17
    Rep Power
    5

    Default Zimbra KVM locking up

    Hi all,

    I have an Ubuntu 10.04 physical host running KVM. This host runs several
    Ubuntu 10.04 VMs, one of which is running Zimbra
    (7.0.1_GA_3105.UBUNTU10_64 UBUNTU10_64 NETWORK).

    5 times since September last year I've had just this VM completely lock
    up at seemingly irregular intervals, and I can find no pattern as to why
    it's happening, but would like to either be able to confirm or discount
    Zimbra as a cause.

    When the lock-up occurs, I can't ping the VM, view its console (using
    the KVM VNC console; it just sits there not even loading a blank
    screen), or shutdown the VM. When I try to shut it down, nothing appears
    to happen, and if I then try to kill the VM, the dead (zombie) process
    stuck around, and KVM thought it was still running, so wouldn't let me
    start it back up without rebooting the physical host. In 4 out of 5
    cases, all other VMs and the host OS were unaffected. In the 5th, all
    VMs locked up.

    Has anybody seen anything like this before?

  2. #2
    Olorin is offline Senior Member
    Join Date
    May 2007
    Posts
    52
    Rep Power
    7

    Default

    I have seen such behaviour of KVM on Ubuntu and Debian.

    If there is room for testing, I would try to take this to RHEL or Fedora hosts and VMs.

    Having said that, can you elaborate on the host and VM configuration and resources available?

  3. #3
    friedmar is offline Loyal Member
    Join Date
    Nov 2005
    Location
    Germany
    Posts
    89
    Rep Power
    9

    Default

    I could not confirm this specific behavior. But I also found some irregular behavior of the Zimbra KVM guest. No problems with other guests!

    So from my point of view it is worth for Zimbra to look into this.
    ---
    Ubuntu 12.04 - Release 8.0.2.GA.5569.UBUNTU12.64 UBUNTU12_64 FOSS edition. - z-push

  4. #4
    Olorin is offline Senior Member
    Join Date
    May 2007
    Posts
    52
    Rep Power
    7

    Default

    Quote Originally Posted by friedmar View Post
    I could not confirm this specific behavior. But I also found some irregular behavior of the Zimbra KVM guest. No problems with other guests!
    Well, I know of a large (~5k users) Zimbra setup running on RHEL+KVM for years now, and no such issues were encountered.

    Zimbra seems to be a bit of a resource hog, probably due to all the Java stack it uses (still light as a feather compared to msexchange though). There might be some tweaks around the Java memory management side of things, but I'm not aware of those.

    BTW, my own setup is restarted every 24 hours, by the backup script, any chance you're running 24/7?

  5. #5
    Tripple is offline Elite Member
    Join Date
    Apr 2009
    Posts
    304
    Rep Power
    5

    Default

    Quote Originally Posted by Olorin View Post
    BTW, my own setup is restarted every 24 hours, by the backup script, any chance you're running 24/7?
    Do you mean Zimbra or your complete server?

  6. #6
    Olorin is offline Senior Member
    Join Date
    May 2007
    Posts
    52
    Rep Power
    7

    Default

    zimbra services

  7. #7
    mattcen is offline Intermediate Member
    Join Date
    Mar 2010
    Location
    Melbourne, Australia
    Posts
    17
    Rep Power
    5

    Default

    Hi all, thank you all for your responses, and sorry for the delay in
    getting back with more information.

    Zimbra is configured to do an incremental backup nightly (plus a full
    backup weekly), though I don't know whether that is supposed to restart
    Zimbra services in the process. If it doesn't, then yes, Zimbra not
    being restarted regularly to my knowledge (except for a reboot of the
    entire host once a month).

    More detail in my setup is:

    * I have an Ubuntu 10.04 host running KVM with libvirt.

    * Each VM's storage is a raw LV for the OS, and, for those that need it,
    another raw LV for data. The Zimbra VM has 4 LVs for it: / (2.0G of
    7.9G used, /opt (356G of 493G used), /opt/zimbra/backup (814G of 1.2T
    used), and swap (~9G).

    * The VMs themselves are also running Ubuntu 10.04. It's worth noting
    that these systems haven't had security patches applied for a long
    time, despite my best efforts (the customer is holding this up a bit).
    I'm hoping that once these updates happen, the problem will go away,
    but in the interim, I'm looking into other possible causes.

    * The Zimbra has 8GB RAM dedicated to it, as well as 4 cores of an AMD
    Opteron(tm) Processor 6134 dedicated to it. The whole server has 2 of
    these processors.

    * I'm not in a position to be able to test this setup on Redhat-based
    hosts.

    Hope this is enough information to evoke some new suggestions. Thanks
    again for your time.

  8. #8
    Olorin is offline Senior Member
    Join Date
    May 2007
    Posts
    52
    Rep Power
    7

    Default

    Quote Originally Posted by mattcen View Post
    * I have an Ubuntu 10.04 host running KVM with libvirt.
    This might be the issue in it's own right. Any chance of trying this VM on another machine, with a RHEL or derivative running kvm?


    * The VMs themselves are also running Ubuntu 10.04. It's worth noting
    that these systems haven't had security patches applied for a long
    time, despite my best efforts (the customer is holding this up a bit).
    I'm hoping that once these updates happen, the problem will go away,
    but in the interim, I'm looking into other possible causes.
    I have seen Ubuntu, especially 10 and 11 doing amazing blunders in conjunction with kvm and libvirt, both as hosts and guests. Internet forums are full of reports where VMs lock up, die and generally misbehave, that could not be reproduced on Fedora or RHEL. This is something very easy to prove, and not written in order to start yet another distro war (even though I am definitely not a fan of Ubuntu as a server of any kind)

    * The Zimbra has 8GB RAM dedicated to it, as well as 4 cores of an AMD
    Opteron(tm) Processor 6134 dedicated to it. The whole server has 2 of
    these processors.
    Without looking those CPUs up, does this mean the Zimbra VM has more virtual cores than the host has physical? If so, this is definitely a problem. But I guess those are multiple core CPUs and I am wrong here.

    * I'm not in a position to be able to test this setup on Redhat-based
    hosts.
    It shouldn't be hard really - get a /opt backup from your customer, and run it in your own local sandbox for a few days, but do it on RHEL or Centos, both as host and guest.

  9. #9
    Brad_C is offline Senior Member
    Join Date
    Apr 2012
    Posts
    73
    Rep Power
    3

    Default

    Quote Originally Posted by Olorin View Post
    I have seen Ubuntu, especially 10 and 11 doing amazing blunders in conjunction with kvm and libvirt, both as hosts and guests.
    Just a counter datapoint, but here we've run Zimbra on an Ubuntu 10.04 Guest on a Debian host with no issues at all for over 12 months now. I'm sure you can find plenty of evidence of it misbehaving, but then if you look you can probably find evidence of RH based VM's misbehaving also.

    If you have a fault that causes the guest to become an unkillable zombie, it's not the fault of the guest!

    I'd be looking very hard at the host, particularly the version of qemu-kvm that is being run.

    I keep ours updated from recently compiled kvm git trees. I certainly would not rely on any packaged version from any distribution. There are always tangible performance and stability increases running a more recent emulator and host kernel.

  10. #10
    Olorin is offline Senior Member
    Join Date
    May 2007
    Posts
    52
    Rep Power
    7

    Default

    Quote Originally Posted by Brad_C View Post
    Just a counter datapoint, but here we've run Zimbra on an Ubuntu 10.04 Guest on a Debian host with no issues at all for over 12 months now. I'm sure you can find plenty of evidence of it misbehaving, but then if you look you can probably find evidence of RH based VM's misbehaving also.
    I've been specializing in kvm based virtualization since 2008, and so far I've amassed quite a statistic. I really don't mean to start a flamewar, and would rather leave the discussion at this. Considering where kvm and libvirt development and QA happens, it is not a wonder Fedora and RHEL behave better.

    If you have a fault that causes the guest to become an unkillable zombie, it's not the fault of the guest!
    not exactly. This can be the fault of the guest kernel drivers. I've seen older virtio_serial and once - misconfigured in-guest IO scheduler to cause exactly this.

    I keep ours updated from recently compiled kvm git trees. I certainly would not rely on any packaged version from any distribution. There are always tangible performance and stability increases running a more recent emulator and host kernel.
    well, this explains why you are not seeing the typical ubuntu/debian behaviour. I wouldn't put anything from the upstream git in production though, not if I want to keep my job at least, but that's up to you of course.

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. What to clean on a Zimbra mail server?
    By tezarin in forum Administrators
    Replies: 11
    Last Post: 12-16-2011, 12:43 PM
  2. /tmp filling
    By Nutz in forum Administrators
    Replies: 8
    Last Post: 02-22-2008, 02:00 AM
  3. [SOLVED] Clamav problem ? What's happening ?
    By aNt1X in forum Installation
    Replies: 23
    Last Post: 02-14-2008, 05:43 AM
  4. Can't start Zimbra!
    By zibra in forum Administrators
    Replies: 5
    Last Post: 03-22-2007, 11:34 AM
  5. zmtlsctl give LDAP error
    By sourcehound in forum Administrators
    Replies: 5
    Last Post: 03-11-2007, 03:48 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •