6.0.4 performance plunge (from 5.0.18) Downgrade path?

[Nutz]

Hello,

I upgraded from 5.0.18 on Christmas day. Upgrade went smoothly except for dropping my commercial cert. I've gotten used to that with major upgrades.

Come Monday morning, when everybody started working, emails started getting deferred with message:

(Names and IPs have been changed, but they are correct)
zimbra postfix/error[13201]: 5EB901798D11A: to=<someone@domain.tld>, relay=none, delay=0.3, delays=0.04/0.02/0/0.25, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to server.domain.tld[10.10.10.10]:7025: Connection refused)

When this happens all mail delivery stalls, and all mails continue to get deferred. Also the web client becomes unresponsive.

It seems to get better with a zmmtactl restart and then flushing the queues, however this requires human interaction, which defies the point of the whole thing...

Any help would be highly appreciated. I opened a ticket with support almost a week ago, and have had extremely little response.

At this point, if I could get back to 5.0.18 that would be fine, but I can't even find any documentation on that.

On top of all of this, my boss has strongly suggested I start pricing out Exchange.

Other performance problems: Accessing free/busy for more than a couple users never returns; Web client frequently announces that server is slow...

Thanks!
Nutz

ZCS Network 6.0.4
2x quad core Xeon 3.0 GHZ
72 Gigs of RAM
CentOS5 X64

[Nutz]

I have had a little contact with Support, who has suggested dropping the jvm heap size to try to avoid excessive CGs.

NOTE: this is only valid for 72 G RAM, check out the performance doc for the right settings for you: Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki

zmlocalconfig -e mailboxd_java_heap_memory_percent=6

[xeon]

Well to get back to 5.0.18 you will need to restore your server from the backup you did before the upgrade, you did that right? There are no downgrade paths, from my understanding. I don't have much advice on your performance problems but I think more information would be nice.

Hard drive usage?
CPU usage?
Any processes taking up a lot of CPU time and memory?
Have you rebooted the server? (I know this usually does not solve much, but there could be some zombie processes)

[Nutz]

Hard drive usage?
Definitely not full

CPU usage?
Load average of about 2-4, should be alright

Any processes taking up a lot of CPU time and memory?
Mysql and java, but nothing else really

Have you rebooted the server? (I know this usually does not solve much, but there could be some zombie processes)
Did that today out of sheer frustration, as I could not get the server back up otherwise.

Support has been in contact with me throughout the day, and I know that they are looking at it, but have had little in the way of suggestions so far.

Cheers,
Nutz

[adeelarifbhatti]

Hi all,
"Web client frequently announces that server is slow.." this issue can be related to network. From home the zimbra interface is quite slow, but from the local n/w it is quite fast, plus it is also dependent on the web browser u r using.
On one of my PC where I use Kaspersky AntiVirus and browser is Mozilla the Zimbra interface is quite slow. But on another system, where I am running with NOD antivirus and browser is same, zimbra is quite fast.

Regards
Adeel

[Nutz]

Yeah I'm seeing this across the board with Mac and PC and Linux Clients, across 400+ mailboxes. I'm pretty sure that the problem is server side, not client side.

Cheers,
Nutz

Update: Support has looked at my logs and we are looking as LDAP as being the bottleneck here. We have now bumped up the loglevel and turned off interprocess tls (for now, as it is all confined to 1 server anyway)

[FritzBrause]

Quad Core CPUs and 72 Gig RAM with a load of 2-4 looks like IO Bottleneck to me. If you're running top or htop and your system's load is above 2 without full CPU utilization there's a real disk IO problem.

Could you check with iostat -d 5 -x what your disks are doing. With 72 Gig RAM there may also be a problem with kernel dirty_ratio.

Without precise information it'll be guesswork to determine the reason for your performance issue.

You may test if these settings improve overall performance:

zmlocalconfig -e zmmtaconfig_interval=3600
zmlocalconfig -e zmstat_interval=300
zmlocalconfig -e mailboxd_java_heap_memory_percent=10
zmlocalconfig -e mailboxd_java_options="-server -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:PermSize=128m \
-XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1"

zmlocalconfig -e ldap_common_threads=16
zmlocalconfig -e ldap_common_toolthreads=8
zmlocalconfig -e ldap_db_cachesize=50000
zmlocalconfig -e ldap_db_idlcachesize=50000
zmlocalconfig -e ldap_db_dncachesize=0
zmprov mcf zimbraMessageCacheSize 5000

/opt/zimbra/conf/my.cnf

table_cache = 1200
innodb_open_files = 2710
innodb_buffer_pool_size = 16384M
innodb_max_dirty_pages_pct = 10
innodb_flush_method = O_DIRECT


Regards

Andreas