[SOLVED] Java Memory Consumption Problem

[LMStone]

It seems Java memory usage keeps growing to the point where our system is beginning to use the swap file.

Should we adjust down somewhat Java's heap memory percentage to avoid growing swap file usage?

Anything else we should look at?

Thanks!
Mark

:::::: Java Local Config :::::::

zimbra@ourserver:~> zmlocalconfig -s |grep java
javamail_imap_debug = false
javamail_imap_enable_starttls = false
javamail_imap_timeout = 60
javamail_pop3_debug = false
javamail_pop3_enable_starttls = false
javamail_pop3_timeout = 60
javamail_smtp_debug = false
javamail_smtp_enable_starttls = false
javamail_smtp_timeout = 60
mailboxd_java_heap_memory_percent = 40
mailboxd_java_home = ${zimbra_java_home}
mailboxd_java_options = -client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1
mailboxd_truststore = /opt/zimbra/java/jre/lib/security/cacerts
zimbra_java_home = /opt/zimbra/java
zimbra@ourserver:~>

:::::::: Top Output After 12 Hours of Zimbra Uptime :::::::
top - 12:57:28 up 5 days, 11:07, 2 users, load average: 0.67, 0.71, 0.71
Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.4%sy, 0.0%ni, 98.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6050552k total, 5969908k used, 80644k free, 114096k buffers
Swap: 2104504k total, 147564k used, 1956940k free, 842788k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22752 zimbra 16 0 2947m 2.6g 12m S 1 44.7 21:57.92 java
22347 zimbra 16 0 2778m 843m 5264 S 0 14.3 3:43.56 mysqld
22927 zimbra 16 0 225m 164m 1044 S 0 2.8 1:25.97 clamd
22361 zimbra 25 0 562m 88m 12m S 0 1.5 0:39.96 java
29910 zimbra 16 0 167m 82m 3696 S 0 1.4 0:07.69 amavisd
27289 zimbra 16 0 166m 81m 3636 S 0 1.4 0:15.11 amavisd
26927 zimbra 15 0 166m 81m 3640 S 0 1.4 0:09.78 amavisd
2836 zimbra 16 0 164m 79m 3640 S 0 1.3 0:06.27 amavisd
31754 zimbra 16 0 163m 79m 3632 S 0 1.3 0:08.56 amavisd
26988 zimbra 16 0 163m 79m 3640 S 0 1.3 0:07.42 amavisd
4700 zimbra 16 0 163m 78m 3624 S 0 1.3 0:09.64 amavisd
525 zimbra 16 0 162m 77m 3620 S 0 1.3 0:05.48 amavisd
6469 zimbra 15 0 161m 76m 3596 S 0 1.3 0:01.93 amavisd
4127 zimbra 16 0 160m 75m 3604 S 0 1.3 0:01.93 amavisd
22812 zimbra 16 0 156m 53m 2472 S 0 0.9 0:06.48 amavisd
21435 zimbra 19 0 374m 50m 7824 S 1 0.9 3:28.80 slapd
20067 named 25 0 86288 34m 1568 S 0 0.6 35:05.07 named
21753 zimbra 16 0 127m 26m 4848 S 0 0.4 5:39.13 mysqld
4207 gdm 16 0 93824 9m 9536 S 0 0.2 0:25.88 gdmgreeter
23355 zimbra 15 0 23304 9.9m 1780 S 0 0.2 0:08.87 perl
21760 zimbra 15 0 23308 9.8m 1784 S 0 0.2 0:14.71 perl
18335 root 16 0 44504 9284 2692 S 0 0.2 0:03.58 snmpd
21784 zimbra 16 0 24972 6448 2688 S 0 0.1 0:49.26 zmlogger
23347 zimbra 23 0 15824 5772 1708 S 0 0.1 0:00.07 swatch
21698 zimbra 25 0 15824 5752 1708 S 0 0.1 0:00.09 logswatch
23359 zimbra 16 0 35376 5576 2916 S 0 0.1 0:00.07 httpd
23361 zimbra 16 0 35376 5576 2916 S 0 0.1 0:00.11 httpd
23362 zimbra 17 0 35376 5576 2916 S 0 0.1 0:00.09 httpd
10913 zimbra 16 0 35376 5560 2904 S 0 0.1 0:00.11 httpd
23357 zimbra 16 0 35376 5560 2904 S 0 0.1 0:00.06 httpd
23358 zimbra 17 0 35376 5560 2904 S 0 0.1 0:00.11 httpd
10536 zimbra 17 0 35376 5556 2904 S 0 0.1 0:00.06 httpd
22239 zimbra 17 0 22248 5428 1252 S 0 0.1 0:59.90 zmmtaconfig
22257 zimbra 16 0 14984 5100 1808 S 0 0.1 0:00.23 zmconvertdmon
23537 zimbra 17 0 13892 4928 1688 S 0 0.1 0:24.57 zmstat-proc
4381 ntp 16 0 16016 4876 3720 S 0 0.1 0:00.32 ntpd
25574 postfix 15 0 45972 4828 3620 S 0 0.1 0:00.29 smtpd
26952 postfix 15 0 46020 4800 3616 S 0 0.1 0:00.21 smtpd
7940 postfix 16 0 45852 4716 3620 S 0 0.1 0:00.06 smtpd
25570 postfix 16 0 45856 4712 3592 S 0 0.1 0:00.19 smtpd

[iway]

What did you do here? Did you get it solved? We're running into the same problem here...

[LMStone]

Originally Posted by interways

What did you do here? Did you get it solved? We're running into the same problem here...

What we are doing these days is setting vm.swappiness=0 as per the Performance Tuning Guidelines wiki article.

We found the later 2.6 kernels are very aggressive at moving things out to swap; that setting reduces that tendency considerably, with no observed ill effects.

Hope that helps,
Mark