We spent a while working closely with Zimbra support on a similar memory usage problem with a 6GB RAM box. We still see Zimbra periodically cause swap file usage, but it's fairly minor now (100MB-200MB each week). No one was ever able to identify with absolute certainty any single root cause, but we did review the following items which, collectively, made a difference and got us where we are now.
- Ensure Novell's AppArmor is turned off or not installed. SUSE installs this by default.
- Turn off the SUSE Firewall (assuming this is safe in your environment).
- Reduce Java memory usage percentage by 5% (to 25% in our case).
- Reduce MySQL memory usage by 5% (again, to 25% in our case).
- In YaST > System > Runlevel Editor, change the default runlevel from 5 to 3.
- Delete the "locate" package, if installed.
- Reboot the server and see what happens.
- Use a properly-sized RAM disk for Amavis's temp files, but you'll probably need to add more RAM to make this worthwhile.
To find out what the current percent of memory used by Java is, as the zimbra user run:
zmlocalconfig | grep mailboxd_java_heap
Now, to change the memory percentage used by Java, run the following command as the zimbra user:
zmlocalconfig -e mailboxd_java_heap_memory_percent=25
(but change the 25 above to be 5 less than what you are using now!)
To change the memory percentage used by MySQL, you need to do three things. First, find the amount of memory currently being used by MySQL:
zmlocalconfig | grep mysql_memory
The, run the following command as the zimbra user:
zmlocalconfig -e mysql_memory_percent=25
(but again, change the 25 above to be 5 less than what you are using now!)
Next, edit the /opt/zimbra/conf/my.cnf file to change the following line:
innodb_buffer_pool_size = xxx
where "xxx" equals the amount of memory used for the MySQL innodb buffer pool. If you are reducing MySQL memory percent usage from 30% to 25%, take the number that is there already and multiply it by 5/6ths. If what you have now is different than 30%, change the fraction accordingly. (I'd suggest commenting out the existing line and adding your new line, so you can easily revert back to from where you started if needed).
After the box is rebooted, as the zimbra user run:
mysql
and then (at the mysql command prompt), run:
mysql> show innodb status;
Look for the following lines toward the end of the report:
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 1740081304; in additional pool allocated 1048576
Buffer pool size 92160
Free buffers 37789
Database pages 52539
Modified db pages 132
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages read 51202, created 1337, written 765301
0.00 reads/s, 0.00 creates/s, 11.05 writes/s
Buffer pool hit rate 1000 / 1000
Re run this command a few times while the system is being used heavily; you are looking for two things:
First, make sure there is a "good" (highly subjective) supply of free buffers.
Second, watch the buffer pool hit rate. It should
rarely drop below 997/1000. If it does, you need more RAM.
To quit out of mysql, type:
quit;
and hit the Enter key.
Another easy thing to do is to run top, and watch the
"%wa" statistic. This represents the percentage of cpu time spent waiting for the system to give the cpu work to do. Typically, this means waiting for the disks. If the
%wa is consistently above 25%, in my experience your users will notice a performance hit.
To make this a little easier on the eyes, as root run in a big window:
vmstat -n 1
(Note that you may need to install the vmstat package.)
On a Zimbra system, the overwhelming majority of disk I/O we have found is Amavis's processing of email in its temp directory. So, if you have consistently high
%wa, either you need a faster disk subsystem or to put amavis's temp directory on a RAM disk.
The latter course requires only sufficient RAM, which is much less disruptive and cheaper usually than replacing a disk subsystem.
If you can add more RAM to your box and want to know how to configure a RAM disk for Zimbra on SUSE, let me know and I'll post that here as well.
Hope that helps,
Mark