[SOLVED] zcs-5.0.18_GA to zcs-6.0.2_GA - stats not running and CPU load spikes...

[mlcdigital]

I was running our Zimbra deployment on Debian 4 and zcs-5.0.18_GA as a VMware host with 2CPU and 1GB RAM (it is a crappy old server).
We were lucky enough to get funding for new server hardware (Dual Optron, 16GB RAM, 5TB RAID 50). It's running Debian 5 with VMware Server 2.0.2 smoothly. I slowly moved virtual machines over...
The last one I migrated was Zimbra. Once migrated I increased it's memory to 3.51GB (32bit build) turned it on and verified all was good. I then did Debian 4 updates and upgraded Zimbra to zcs-6.0.2_GA.
Everything seemed to be running smoothly for a couple days and then...
Stats stopped running. I could turn it back on without errors and view stats in the admin interface but in a few hours (cron job?) it would be off again. I'm also noticing CPU loads spike to 100% then drop down to 50-60%.
I've tail -f /var/log/zimbra.log during the spikes/stats-down but saw nothing unusual.

zimbraserver:~# more /var/log/zimbra.log | grep stats
Nov 16 07:52:01 zimbraserver zimbramon[320]: 320:err: Service status change: zimbraserver.domain.tld stats changed from stopped to running
Nov 16 12:20:15 zimbraserver zimbramon[320]: 320:err: Service status change: zimbraserver.domain.tld stats changed from running to stopped

It looks like this might be a VM issue. This is the VMware Server...

top - 14:45:33 up 17:12, 1 user, load average: 2.52, 2.29, 2.21
Tasks: 82 total, 1 running, 81 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6%us, 35.4%sy, 0.0%ni, 63.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16474388k total, 16421488k used, 52900k free, 102736k buffers
Swap: 6385796k total, 664k used, 6385132k free, 15576440k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3412 root 20 0 1909m 88m 75m S 138 0.5 1616:37 vmware-vmx
3578 root 20 0 1234m 1.1g 1.1g S 8 6.9 116:36.42 vmware-vmx
3399 root 20 0 790m 546m 536m S 5 3.4 41:57.94 vmware-vmx
3544 root 20 0 765m 507m 495m S 2 3.2 58:10.85 vmware-vmx

This is the Zimbra Server...

top - 14:54:29 up 17:20, 1 user, load average: 3.54, 3.76, 3.77
Tasks: 112 total, 4 running, 108 sleeping, 0 stopped, 0 zombie
Cpu(s): 27.4%us, 67.3%sy, 0.0%ni, 5.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3653788k total, 3509940k used, 143848k free, 57272k buffers
Swap: 2650684k total, 52k used, 2650632k free, 1721556k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17336 zimbra 24 0 205m 12m 7520 S 4 0.3 0:00.13 java
8068 zimbra 25 0 4616 876 464 R 4 0.0 73:25.13 nginx
320 zimbra 21 0 9396 7384 1128 S 1 0.2 0:41.02 zmmtaconfig
20038 root 15 0 2228 1168 864 R 0 0.0 0:02.13 top

If it's not a VMware Server issue would this be the next place to dig even if I'm going from Zimbra 5 to 6...?
Troubleshooting High CPU load average in ZCS 5.0.x as compared to 4.5.x - Zimbra :: Wiki

[mlcdigital]

I think I've pin-pointed the issue...
Debian 4 kernel 2.6.18-6-686 isn't 100% SMP compliant, at least running on a VMware host with compliant virtualization CPUs. The mail archive server is running Debian 4 kernel 2.6.18-6-686 but with one CPU and it runs smooth.
I'm setting up a new Zimbra VM running Debian 5 kernel 2.6.26-2-686 and will attempt to migrate /opt/zimbra over.
Are these the most recent and accurate directions to migrate Zimbra?
Moving ZCS to Another Server » Zimbra :: Blog

[phoenix]

Are these the most recent and accurate directions to migrate Zimbra?
Moving ZCS to Another Server » Zimbra :: Blog

Yes, they will get you to a new server.

[mlcdigital]

I tried following the steps located here a couple times with no luck.
Moving ZCS to Another Server » Zimbra :: Blog

I'd like to focus on the errors existing on the live server after the upgrade from ZCS 5.0.18 to ZCS 6.0.2...

1. Stats stops after 3 hours'ish and I can restart stats it with no errors.
2. Secure IMAP/POP isn't working.
3. I go into the Zimbra admin interface to verify IMAP/POP settings. The interface is slow and doesn't look correct.
4. When I hit save on Server -> Server.Address I get an error like...
   

   JavaScript error encountered in method ZaXFormViewController.prototype.fireChangeEvent
   name:TypeError message: 'this_toolbar' is null or not an object number: - 2146823281 description: 'this_toolbar' is null or not an object

5. I tail -f /var/log/zimbra.log during my tests and didn't see any errors.
6. Who knows what other problems also exist...?

I'm afraid if I do successfully migrate our /opt/zimbra over to a fresh server these issues will just follow...?
Is there some kind of repair or reinstall process to potentially rebuild everything and maintain the existing data?

[mlcdigital]

After a week of running on 6.0.2 I've come to realize it's not stable yet so I had to roll back to 5.

[emiro]

I have also run into that error. We just upgraded from 5.0.13 to 6.0.3.

JavaScript error encountered in method ZaXFormViewController.prototype.fireChangeEvent
-------------------------------------------------------------------
message: this._toolbar is undefined fileName: https://{adminIP:port}/zimbraAdmin/js/Admin_all.js.zgz?v=091118112000 lineNumber: 1658 stack: ()@https://{adminIP:port}/zimbraAdmin/js/Admin_all.js.zgz?v=091118112000:1658 ([object Object])@https://{adminIP: port}/zimbraAdmin/js/Admin_all.js.zgz?v=0911...


This happens when trying to save a "GAL Configuration" screen. Which, btw, is broken which is why I'm trying to edit it. After this error, opening up the config screen again reveals that the edits aren't saved. :( Can't try to resolve my GALsync issue till I can edit the config!

[emiro]

Also, in IE the "details" look the same as mlcdigital's last post. IE and firefox differ on the description of the problem. Also, I was able to make changes from "external" to "internal" but no details regarding galsync account name/datasource, etc are saved.