[SOLVED] Java Eating Up the CPU

[Chewie71]

Our students are back this past weekend. We have one mailbox server that is having lots of problems with the CPU spiking up to 18-30....and then the mailbox becomes unresponsive until we 'zmcontrol stop/start' or 'zmmailboxdctl restart'. We have been having to do this once every 2 hours or so during the day. It's possibly Java causing the problem but I can't seem to be able to nail it down. We have 4 other mailbox servers with similar numbers and types of users and none of them are having any problems.

Can anyone help me?

What should I be looking for to try and figure out why only this server keeps dying on us?

Thanks,
Matt

[uxbod]

Any difference in usage patterns on that server ie. just Ajax ZWC or IMAP ?? If you could post some stats ie. top, meminfo, vmstat

[Chewie71]

The usage patterns are very similar across all mailboxes. A mix of students and faculty/staff....most are using the ZWC, but there are a very small number of IMAP users.

top - 10:51:28 up 8 days, 15:18,  7 users,  load average: 11.72, 7.57, 6.50
Tasks: 185 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.5% us, 10.1% sy,  0.0% ni, 37.1% id, 42.4% wa,  0.1% hi,  0.9% si
Mem:   8167108k total,  8125612k used,    41496k free,     6176k buffers
Swap: 16386292k total,  2731492k used, 13654800k free,    52336k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                         
29498 zimbra    16   0 3809m 1.7g  34m S 45.2 22.0   7:53.79 java                                             
  107 root      15   0     0    0    0 D 37.3  0.0 149:06.50 kswapd1                                          
12889 zimbra    16   0 2228m 452m 3248 S 10.9  5.7   6:16.87 mysqld                                           
  108 root      15   0     0    0    0 S  8.3  0.0 126:18.94 kswapd0                                          
    9 root      34  19     0    0    0 S  0.3  0.0   2:00.02 ksoftirqd/3                                      
  742 root       0 -20     0    0    0 S  0.3  0.0  19:23.49 mpp_dcr                                          
 5185 zimbra    16   0  5444 1168  816 R  0.3  0.0   0:00.05 top                                              
11858 root      15   0     0    0    0 S  0.3  0.0   2:38.90 kjournald                                        
11869 root      15   0     0    0    0 S  0.3  0.0   1:44.30 kjournald                                        
11891 root      15   0     0    0    0 S  0.3  0.0   1:21.17 kjournald                                        
    1 root      16   0  4772  396  372 S  0.0  0.0   0:01.41 init                                             
    2 root      RT   0     0    0    0 S  0.0  0.0   0:03.13 migration/0                                      
    3 root      34  19     0    0    0 S  0.0  0.0   1:30.20 ksoftirqd/0                                      
    4 root      RT   0     0    0    0 S  0.0  0.0   0:02.86 migration/1                                      
    5 root      34  19     0    0    0 S  0.0  0.0   9:20.68 ksoftirqd/1                                      
    6 root      RT   0     0    0    0 S  0.0  0.0   0:03.04 migration/2                                      
    7 root      34  19     0    0    0 S  0.0  0.0   7:40.14 ksoftirqd/2                                      
    8 root      RT   0     0    0    0 S  0.0  0.0   0:01.92 migration/3

# cat /proc/meminfo
MemTotal:      8167108 kB
MemFree:         46328 kB
Buffers:          8124 kB
Cached:          58836 kB
SwapCached:     881964 kB
Active:        2159928 kB
Inactive:       153724 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      8167108 kB
LowFree:         46328 kB
SwapTotal:    16386292 kB
SwapFree:     13371872 kB
Dirty:            3132 kB
Writeback:           0 kB
Mapped:        2230484 kB
Slab:          5746512 kB
CommitLimit:  20469844 kB
Committed_AS:  6723140 kB
PageTables:      17940 kB
VmallocTotal: 536870911 kB
VmallocUsed:     15088 kB
VmallocChunk: 536855819 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  3 3127988  62256  13052  68452   88   42   499   290   10     0 15  2 76  7

[Chewie71]

There are a bunch of stacktrace logs being generated in /op/zimbra/log.

Looks to me like it is probably dumping the stacktrace before I restart the services. So maybe something in there will tell me what's going wrong?

Matt

[razametal]

Try the following solution :

[SOLVED] high cpu load java

[Chewie71]

It ended up being a process from our backup software that would hang on us while it was trying to scan a huge filesystem. We've resolved it,

[razametal]

Thank you for sharing the solution.