Sudden Zimbra slowness?

[lindsey]

I'm hoping that someone can offer me some advice.

About four days ago our Zimbra server suddenly started running *really* slow. Web users continually received pop-up warnings that their requests were timing out and remote IMAP users couldn't browse their folders.

The server load is higher than usual, hovering around 2-3 (usually it's at .5 or so). Killing Zimbra restores the server back to normal; starting it again causes problems. I've played with all kinds of system settings swapped out memory, checked the database integrity, etc. to no avail.

About a week before we had upgraded to zcs-NETWORK-5.0.11, but things had been running smoothly in the following week. No system updates were performed, nor were any other changes made to the system before it started 'acting up'.

How can I can track down what exactly is happening in Zimbra?

As far as the server configuration is concerned:

2 quad core 2.33GHz Xeon processors
32GB RAM
4.5TB disk (at about 12% capacity)
~ 370 active users
64-bit RHEL 5.2

Thanks for any leads.

[iway]

Have you checked memory consumption (top)? We have this issue sometimes: Zimbra on Ubuntu 8.0.4 - severe issues

[lindsey]

Yes, memory consumption is always topped by the java process. I've seen this since we first installed Zimbra (5.0.4) and, like you, see the memory usage continue to climb until the system crashes.

My temporary workaround was to run this nightly at 4am:

/opt/zimbra/bin/zmmailboxdctl restart

which seemed to free up memory to get us through the day.

I think it's a problem for us, but I don't think it explains the sudden slowdown that we're seeing now. The system gets bogged down and unresponsive even after a reboot.

The system acts like it's running out of threads, suddenly blocking on certain inputs. Open filehandles and kernel threads thresholds are OK though. I've followed many of the guidelines in the Zimbra Performance Tuning guide, although I don't know how to tell how many threads Java is actually using/needs, etc.

[iway]

Please vote for the bug: Bug 34040 – Zimbra java consumes all memory on Ubuntu 8.0.4 64bit

Perhaps someone at Zimbra will finally try to solve this severe issue! More and more people seem to have this java memory problem.

[DanielGentleman]

I too am having this problem. I put my information in the bug and don't care to play the "waiting game."

Have there been any strides made to determine if it can be fixed outside of Zimbra? Any active work-arounds? I'm not a programmer - just a systems guy - so I don't even know where to begin with Java memory leaks.

---D

[iway]

Well, Zimbra support was looking into it on our server, with no real success, unfortunately. Setting the java heap to 25% fixed some of the major problems on a short-term basis, but a long-lasting solution is still out to be found.

I hope some people at Zimbra start taking a look into it, since this is a really severe bug!

Could someone from Zimbra reply to this please?

[lindsey]

Although I do have the Java memory problem, I want to point out that it's tangential to the issue that I'm asking about here. We've had a workaround (1 minute outage daily during a restart) for the Java memory issues, but the extreme slowness that we're seeing has no workaround. The system is almost unuseable and I'm worried what will happen after the New Year when students and faculty return and the load increases.

[DanielGentleman]

I Emailed Zimbra asking if I could buy a support incident from them at whatever the price to get this resolved:

Hi Daniel,

I just spoke with my management, and was informed that support is only made available with the Network Edition. Have you considered a possible upgrade?

Sincerely,
Jin

[lindsey]

How does one use Zimbra support? We have a three-year educational contract with Zimbra, but our sales rep told me to use the forums for support issues. I wasn't aware that anything else was available.

/is technical, not social

[phoenix]

How does one use Zimbra support? We have a three-year educational contract with Zimbra, but our sales rep told me to use the forums for support issues. I wasn't aware that anything else was available.

I find it hard to believe that you weren't told how to contact support. If you are the person responsible for administering Zimbra then you should have been given details of how to log a support case via the Support Portal, you should also add your details to the bug report. When your organisation purchased Zimbra they would have been given details of how to get support (and login details to the support portal), can you ask your management what your procedure is for getting access? If you still have problems then fire off an email to the support address in the link I've given you wuith full details of your problem and reference this thread and the bug report.

BTW, can you try reducing the Java Heap Size to a value that would leave it at no more than 3GB and see if that helps your problem?