6.04 Memory Leak - Fedora 11

[dhayes]

Hello. We are having a vexing issue with a memory leak on a Fedora 11 box virtual guest (fully patched) running under Hyper-V. I know this is not officially supported but with the exception of the memory leak it is functioning fine. What happens is that the server will run for a day or so and the memory usage (viewable via top and free) decreases to the point where it runs out and causes LDAP to fail. Lots of LDAP errors when it runs out. I wonder if it has something to do with an entry in cron for Zimbra but I did not mess with the file. We have 5 GB of RAM allocated to the VM.

Does anyone have an idea as to where to look for this problem? Are we looking at a Java issue?

Thanks again for any assistance

Dave

[ArcaneMagus]

Which process is the one using all the available memory? The LDAP server? Java?

[dhayes]

Hello...Thanks...Top reveals 2589M under the VIRT column for java running under the user Zimbra. MySQLd shows about 1671M under the VIRT column. When we reboot the box the java usage goes down to about 970M and creeps up from there throughout the day. It will get to about 2 days of uptime and the java process appears to be close to 3000M under the VIRT column. At that point there is not enough free memory and then LDAP fails and people can no longer log in.

Thanks for any assistance

Dave

[ArcaneMagus]

Well the numbers that you posted really aren't that high, for example here is what my server looks like:

top - 11:26:04 up 199 days,  2:34,  1 user,  load average: 1.19, 0.95, 0.85
Tasks: 116 total,   1 running, 115 sleeping,   0 stopped,   0 zombie
Cpu(s): 51.8%us,  5.8%sy,  0.0%ni, 42.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3991744k total,  3762852k used,   228892k free,   237020k buffers
Swap:  8388600k total,   317488k used,  8071112k free,   602892k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31362 zimbra    20   0 2400m 1.9g  12m S    1 50.3 357:13.02 java
17651 zimbra    20   0 1198m 124m 4512 S    0  3.2 188:58.93 mysqld

I think it's been around a month or two since the Zimbra services were last restarted.

In /var/log/zimbra-stats.log could you find a line for "zmstat vm.csv" around the time that the system was overloaded? This should tell you most of the vital statistics about the Java VM.

[dhayes]

Hello..Thanks much...Actually, you are correct...Shortly after it rebooted the java process was at the number I mentioned above and it was not leaking at the time. So I was incorrect. But, I rebooted it 24 hours ago and the memory viewable to top has decreased drastically. I wonder if it is something in cron that is causing this. I am not really sure how to find this memory leak. The memory is going down as I type this.

Perhaps an incompatibility with Hyper-V?

Thanks for any help..

Dave

[uxbod]

What are your stats now ? You could try setting in sysctl.conf

vm.swappiness = 10

to see if that helps. On my atom server I show as

top - 06:24:06 up 12 days, 15:38,  1 user,  load average: 1.02, 0.72, 0.64
Tasks: 139 total,   1 running, 138 sleeping,   0 stopped,   0 zombie
Cpu(s): 23.0%us,  3.2%sy,  0.0%ni, 73.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2065860k total,  1940764k used,   125096k free,   181372k buffers
Swap:  4128760k total,        0k used,  4128760k free,   862144k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                      
25478 zimbra    18   0  403m  18m 8712 S 17.2  0.9   0:00.54 java

[dhayes]

Also, one other thing. zimbra-stats.log is empty. I do not think stats is running properly. Note too that I have run zmfixperms many times.

Thanks

[dhayes]

Thank you very much. I changed the vm.swappiness setting to 10 and rebooted. I will monitor everything from this point forward.

Again, thanks again. I am not sure why this is leaking so bad.

Thanks
Dave

[uxbod]

Linux does allow memory to balloon on processes if it has sufficient resources (AFAIK). Setting the swapiness should throttle that back a bit.

[dhayes]

Thanks much. I seems like I was able to get another day out of it with the vm.swappiness option set but it eventually ran out of memory again causing LDAP to fail. Just FYI this is Fedora core 64-Bit version 11. System VM has 4GB allocated to it. I have no idea how to find this leak. After a reboot I am back to normal for a day or so.

Thanks
Dave