I'm having a little fun with a mailbox server and hoping someone can help me out.
This server literally has no users on it. We switched a BES over to use it and mailboxd keeps crashing reporting that Java heap was out of memory. I bounce mailboxd and everything is cool again for 10 minutes, an hour, two days.. who knows.
It started out with only 2GB of memory, but it was only handling LDAP traffic and BES traffic so we didn't think it would need all that much. We allotted it 40% memory for Java heap. After the first couple of times, I increased that to 50%. When that didn't help I added a GB of RAM. Then a second processor. Then another GB of RAM (it's on VMware ESX). Now, mailboxd is still crashing, but somehow magically recovering (seen in the logs and from our monitoring system). Shortly after the crash, I see messages about services starting again in /opt/zimbra/log/mailbox.log.
It seems to be happening when the BES is checking a certain mailbox, but I can't seem to find anything terribly different about that account. I started noticing after a few crashes that it all seemed too familiar and I was right. grep FATAL /opt/zimbra/log/mailbox.log revealed a bunch of these:
There were a bunch of those lines (without the grep they are followed by stack traces for out of memory) and no other users are shown in the logs on the same line as "FATAL".
2009-10-13 13:03:05,286 FATAL [btpool0-17] [firstname.lastname@example.org;email@example.com;ip=172.16.0.154;ua=ZimbraConnectorForBES/5.0.2711.18;] system - handler exception
The mailboxd process doesn't seem to be using all that much memory. When I watch it in top, Java typically takes up something like 15-20% all the time. Then suddenly, it will quickly climb to over 60% (larger than 50% which is allegedly the max), and it crashes.
The performance graphs in ESX seem to indicate the same thing. All of a sudden the VM starts consuming large amounts of memory until mailboxd crashes. Then slowly it lets go until the same thing happens again.
Is there anything else I can check? I should note.. this user does speak French, so it could be that there's some funky character in there, but I'm not seeing any problems viewing this person's mail or anything indicating that it's hitting a bad character.