Memory Management Techniques?

[LMStone]

This weekend past we upgraded from 32-bit 4.5.11 to 64-bit 5.0.4 on one of our ZCS systems.

Today I (and other users) noticed things like login screens taking 10 or more seconds to appear; same for compose windows.

During this period top showed perhaps peak 7%wa, but I did see that swap usage had grown to about 1GB (this server has a 2GB swap file) with about 4GB of memory used by java alone. This server has 6GB of RAM and 15K U320 SCSI RAID1 disks.

I did a zmcontrol stop, saw that swap usage went down to about 300K, and RAM usage dropped to 1.5GB with no zimbra-user owned processes running.

After doing the zmcontrol start, swap file usage remained the same and total RAM usage is about 4GB (Java is using about 750MB). System snappiness returned; at least login windows and compose windows appeared now near instantly.

Here's the "after" shot:
top - 22:25:27 up 3 days, 20:35, 2 users, load average: 0.46, 0.59, 0.56
Tasks: 252 total, 1 running, 251 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.6%us, 4.1%sy, 0.1%ni, 91.0%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 6050552k total, 4145356k used, 1905196k free, 165028k buffers
Swap: 2104504k total, 33944k used, 2070560k free, 1328952k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23838 zimbra 16 0 2099m 706m 10m S 0 11.9 1:01.02 java
23452 zimbra 16 0 1641m 396m 5216 S 0 6.7 0:07.06 mysqld
24209 zimbra 15 0 200m 179m 1032 S 0 3.0 0:10.24 clamd
24222 zimbra 16 0 163m 103m 3660 S 0 1.8 0:15.87 amavisd
24216 zimbra 15 0 162m 102m 3628 S 0 1.7 0:07.10 amavisd
24220 zimbra 15 0 162m 102m 3628 S 0 1.7 0:07.41 amavisd
24218 zimbra 15 0 162m 102m 3640 S 0 1.7 0:08.73 amavisd
24223 zimbra 15 0 162m 102m 3624 S 0 1.7 0:06.19 amavisd
24215 zimbra 15 0 162m 102m 3632 S 0 1.7 0:06.11 amavisd
24224 zimbra 15 0 162m 101m 3636 S 0 1.7 0:06.93 amavisd
24219 zimbra 15 0 161m 101m 3604 S 0 1.7 0:07.33 amavisd
24221 zimbra 15 0 161m 100m 3688 S 0 1.7 0:04.75 amavisd
24214 zimbra 16 0 161m 100m 3628 S 2 1.7 0:08.34 amavisd
24159 zimbra 16 0 156m 96m 2468 S 0 1.6 0:02.79 amavisd
22509 zimbra 19 0 292m 49m 7368 S 6 0.8 0:07.35 slapd
23455 zimbra 25 0 474m 30m 9376 S 0 0.5 0:01.83 java
22867 zimbra 16 0 126m 25m 4700 S 0 0.4 0:03.98 mysqld
24267 zimbra 15 0 23304 9.8m 1780 S 0 0.2 0:00.37 perl
22836 zimbra 15 0 23172 9.8m 1780 S 0 0.2 0:00.38 perl
23326 zimbra 17 0 22252 7580 1252 S 0 0.1 0:01.20 zmmtaconfig
22926 zimbra 16 0 24972 6448 2688 S 1 0.1 0:01.33 zmlogger
24259 zimbra 22 0 15828 5772 1708 S 0 0.1 0:00.07 swatch
22770 zimbra 25 0 15824 5756 1708 S 0 0.1 0:00.10 logswatch
23340 zimbra 16 0 14984 5104 1808 S 0 0.1 0:00.11 zmconvertdmon
24480 zimbra 17 0 13764 4828 1672 S 0 0.1 0:00.66 zmstat-proc
24502 zimbra 16 0 13504 4568 1664 S 0 0.1 0:00.15 zmstat-mysql
24268 zimbra 16 0 34020 4552 2000 S 0 0.1 0:00.02 httpd
24482 zimbra 16 0 13500 4516 1672 S 0 0.1 0:00.08 zmstat-cpu
24484 zimbra 15 0 13504 4488 1660 S 0 0.1 0:00.08 zmstat-vm
24505 zimbra 16 0 13372 4428 1660 S 0 0.1 0:00.08 zmstat-mtaqueue
24495 zimbra 16 0 13368 4412 1660 S 0 0.1 0:00.08 zmstat-fd
24270 zimbra 24 0 34020 3076 512 S 0 0.1 0:00.00 httpd
24271 zimbra 24 0 34020 3076 512 S 0 0.1 0:00.00 httpd
24272 zimbra 25 0 34020 3076 512 S 0 0.1 0:00.00 httpd
24273 zimbra 25 0 34020 3076 512 S 0 0.1 0:00.00 httpd
24274 zimbra 25 0 34020 3076 512 S 0 0.1 0:00.00 httpd
12391 zimbra 16 0 14208 2532 1652 S 0 0.0 0:00.12 bash
22765 zimbra 25 0 8380 1584 1132 S 0 0.0 0:00.01 mysqld_safe
23335 zimbra 25 0 8384 1584 1132 S 0 0.0 0:00.00 mysqld_safe
24210 zimbra 16 0 14240 1388 976 S 0 0.0 0:00.22 freshclam
12390 zimbra 17 0 23324 1336 1044 S 0 0.0 0:00.00 su
24401 zimbra 24 0 25740 1244 800 S 0 0.0 0:00.00 saslauthd
24403 zimbra 21 0 25740 784 340 S 0 0.0 0:00.00 saslauthd
24404 zimbra 21 0 25740 704 260 S 0 0.0 0:00.00 saslauthd


I know 64-bit systems use more memory than comparable 32-bit systems, but is there a way to tune memory management here, perhaps by limiting the amount of RAM used by java? I didn't expect we'd be using the swap file after only three days of uptime.

Thanks!
Mark

[uxbod]

May be worth having a read through this Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki if not already done so. Some good bits on Java tuning.

[LMStone]

Originally Posted by uxbod

May be worth having a read through this Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki if not already done so. Some good bits on Java tuning.

I'm very familiar with that document but it was written for the 4-series product, and with all the changes in 5 (jetty/tomcat for example) as well as updates to the OSS components, it's not clear to me the advice and counsel in that document remains accurate.

Probably a number of Zimbra documents could use a "refresh" to highlight the differences between version 4 and 5...

All the best,
Mark

[glitch23]

Mark,

I think this was the second time I ever read through that whole document but I notice that is has changed and now includes alot of ZCS 5.0.x changes too. Also the last modified date on the article is now 3/10/2008. FYI.

[LMStone]

Originally Posted by glitch23

Mark,

I think this was the second time I ever read through that whole document but I notice that is has changed and now includes alot of ZCS 5.0.x changes too. Also the last modified date on the article is now 3/10/2008. FYI.

That's terrific, thank you!

Mark

P.S. The article is loading already in another tab, and I'm headed there now...