Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: Jetty High CPU load since July 1st 00:00:00 UTC

  1. #1
    cvidal is offline Project Contributor
    Join Date
    Dec 2005
    Posts
    150
    Rep Power
    9

    Default Jetty High CPU load since July 1st 00:00:00 UTC

    I woke up this morning with 5 of my customers in 3 different continents showing high CPU loads for jetty since 00:00:00 UTC. 2 of them have only 5 accounts ! I restarted Zimbra but nothing changes, Jetty goes up again to the same load avg. This is what I see with top (process 10223 is jetty). The other 5 servers (and counting) with the same problem have similar output. The mailbox.log shows nothing abnormal. Any clues ???

    All servers are running ZCS NE 7.1.4 on RHEL6.

    top - 10:31:52 up 6 days, 49 min, 2 users, load average: 17.59, 18.02, 18.09
    Tasks: 310 total, 2 running, 308 sleeping, 0 stopped, 0 zombie
    Cpu0 : 26.4%us, 69.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 4.4%si, 0.0%st
    Cpu1 : 28.0%us, 68.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.4%si, 0.0%st
    Cpu2 : 28.5%us, 68.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.5%si, 0.0%st
    Cpu3 : 42.5%us, 53.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 4.1%si, 0.0%st
    Cpu4 : 27.7%us, 68.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.4%si, 0.0%st
    Cpu5 : 35.4%us, 63.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.4%si, 0.0%st
    Cpu6 : 30.4%us, 66.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.3%si, 0.0%st
    Cpu7 : 33.0%us, 63.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.3%si, 0.0%st
    Mem: 8176384k total, 4315580k used, 3860804k free, 677332k buffers
    Swap: 3999736k total, 13888k used, 3985848k free, 1436244k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    10223 zimbra 20 0 6395m 502m 13m S 489.8 6.3 305:08.96 java
    8811 zimbra 20 0 2217m 252m 12m S 117.4 3.2 82:38.36 java
    13 root 20 0 0 0 0 S 13.6 0.0 36:30.47 ksoftirqd/3

  2. #2
    MTHead is offline Junior Member
    Join Date
    Jun 2008
    Location
    Sudbury, Canada
    Posts
    6
    Rep Power
    7

    Default

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    29733 zimbra 20 0 4075m 1.1g 14m S 353 14.2 2252:36 java
    28617 zimbra 20 0 820m 395m 12m S 171 4.9 914:14.79 java


    same here

  3. #3
    bku09 is offline Member
    Join Date
    Jan 2010
    Posts
    13
    Rep Power
    5

    Default

    got the same problem, nothing intresting in logfiles ...

    System: Ubuntu 10.04.4 LTS, ZCS 7.2.0, zimbra restart didn't solve the problem ...

    strace on 100% cpu jetty threads:
    ----------------------------------
    futex(0x41f7b328, FUTEX_WAKE_PRIVATE, 1) = 0
    futex(0x7fed20aa7f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 697245000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
    futex(0x41f7b328, FUTEX_WAKE_PRIVATE, 1) = 0
    futex(0x7fed20aa7f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 697814000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
    futex(0x41f7b328, FUTEX_WAKE_PRIVATE, 1) = 0
    futex(0x7fed20aa7f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 698305000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
    futex(0x41f7b328, FUTEX_WAKE_PRIVATE, 1) = 0
    futex(0x7fed20aa7f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 698818000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
    futex(0x41f7b328, FUTEX_WAKE_PRIVATE, 1) = 0
    futex(0x7fed20aa7f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 699337000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
    futex(0x41f7b328, FUTEX_WAKE_PRIVATE, 1) = 0
    futex(0x7fed20aa7f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 699821000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
    futex(0x41f7b328, FUTEX_WAKE_PRIVATE, 1) = 0
    futex(0x7fed20aa7f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341144531, 700337000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

    Graph:
    ------
    zimbra.jpg
    Last edited by bku09; 07-01-2012 at 05:17 AM.

  4. #4
    MTHead is offline Junior Member
    Join Date
    Jun 2008
    Location
    Sudbury, Canada
    Posts
    6
    Rep Power
    7

  5. #5
    bku09 is offline Member
    Join Date
    Jan 2010
    Posts
    13
    Rep Power
    5

    Default

    thanks to user MTHead, here is the solution (tested and confirmed working):

    -----
    /etc/init.d/ntp stop
    date `date +"%m%d%H%M%C%y.%S"`

    You don’t have to restart your java applications (tomcat, wowza, or whatever using java); it should be enough to run the above commands.
    You may want to enable ntp the next day (there were some reports that enabling it makes java misbehave on some installations).
    -----
    Source: java leap second bug – 30 June / 1 July 2012 – fix » WPKG Blog » Blog Archive
    Last edited by bku09; 07-01-2012 at 05:54 AM.

  6. #6
    cvidal is offline Project Contributor
    Join Date
    Dec 2005
    Posts
    150
    Rep Power
    9

    Default

    It seems like a generic Java problem. One of the servers has a Java tool to monitor the UPS and shutdown if it runs out of battery. This process was also sucking a lot of CPU together with Jetty and zmconfigd. I did the Windows way (yes, mea culpa!), and rebooted the server (after 200 days uptime!), and so far the load averages are back to normal. What is this crap ?

  7. #7
    dave_kempe is offline Partner (VAR/HSP)
    Join Date
    Sep 2005
    Location
    Sydney, Australia
    Posts
    291
    Rep Power
    9

    Default

    You don't need to stop NTP.
    My simple solution was this:
    date -s "`date`"
    and CPU usage returned to normal.
    Also affected a few mysql intances on other apps, so watch out.
    http://www.solutionsfirst.com.au/hosting/zimbra/
    Australia's premier Zimbra Hosting Partner
    Resellers wanted!

  8. #8
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,504
    Rep Power
    57

    Default

    I guess someone will be filing a bug report about this problem and it's solution?
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  9. #9
    cvidal is offline Project Contributor
    Join Date
    Dec 2005
    Posts
    150
    Rep Power
    9

    Default

    Thank you very much everybody, everything is back to normal ;-)

  10. #10
    bku09 is offline Member
    Join Date
    Jan 2010
    Posts
    13
    Rep Power
    5

    Default

    the java "leap second bug" causes some serious problems all around the net:

    'Leap Second' Bug Wreaks Havoc Across Web | Wired Enterprise | Wired.com

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. CPU Consume is High process Jetty
    By mtimbi@ups.edu.ec in forum Administrators
    Replies: 2
    Last Post: 06-21-2011, 08:33 AM
  2. High Load
    By Tripple in forum Installation
    Replies: 2
    Last Post: 11-07-2010, 07:02 PM
  3. High idle load
    By RoyK in forum Administrators
    Replies: 6
    Last Post: 12-01-2009, 01:08 PM
  4. High CPU Load
    By Adrnalnrsh in forum Administrators
    Replies: 0
    Last Post: 05-06-2009, 12:18 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •