Results 1 to 5 of 5

Thread: Memory Usage and Slowness + Backup Error

  1. #1
    cyberdeath's Avatar
    cyberdeath is offline Active Member
    Join Date
    Jan 2008
    Location
    127.0.0.1, Virginia, USA
    Posts
    42
    Rep Power
    7

    Unhappy Memory Usage and Slowness + Backup Error

    Hello everyone,

    I'm experiencing a strange problem since upgrading to 5.0.8 (and 5.0.9) and enabling archiving/discovery. Once I load up zimbra and have it running for a while, the system uses up a huge amount of memory and then clutters up the SWAP file (slowing performance). This has only occurred recently. The only thing I did recently (aside from upgrading + archiving/discovery) was fix the logger mysql database so now it actually gives me a report every day (before it would error and die). What can be done about this memory usage? This is a small server with only 200 users on it. Performance wise: it's a 2xQuad Core Xeon with 4GB of memory running on 32bit SuSE ES10.1 with 15K RPM drives running in a RAID 5.

    Here is the error message I get when I backup the system nightly:
    Code:
    system failure: backing up table: mboxgroup67.appointment
    com.zimbra.common.service.ServiceException: system failure: backing up table: mboxgroup67.appointment
    ExceptionId:BackupSetWorkerThread-DB:1219208557644:5d072cfe01bf9626
    Code:service.FAILURE
            at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253)
            at com.zimbra.cs.db.DbBackup.saveTable(DbBackup.java:132)
            at com.zimbra.cs.backup.BackupAccountSession.storeTablesToLocalFile(BackupAccountSession.java:276)
            at com.zimbra.cs.backup.FileBackupTarget$BackupAcctSession.storeTablesStage(FileBackupTarget.java:1294)
            at com.zimbra.cs.backup.FileBackupTarget$BackupSetWorkerThread.run(FileBackupTarget.java:2494)
    Caused by: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTO OUTFILE '/opt/zimbra/backup/tmp/full-20080820.050013.549/accounts/855/4a9/8' at line 1
    Here is "top":
    Code:
    top - 15:28:02 up  6:36,  1 user,  load average: 2.64, 2.89, 3.64
    Tasks: 196 total,   2 running, 194 sleeping,   0 stopped,   0 zombie
    Cpu(s): 15.3%us,  1.9%sy,  0.0%ni, 72.5%id, 10.3%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:   3365168k total,  3236312k used,   128856k free,    14948k buffers
    Swap:  4200988k total,  1738020k used,  2462968k free,  2114112k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     7282 zimbra    16   0 1787m 164m 3320 S  100  5.0 385:08.73 mysqld
     6746 zimbra    16   0  108m  13m 2296 S   16  0.4  40:04.86 mysqld
    16568 zimbra    19   0  6160 4528 1628 S    2  0.1   0:00.07 zmcontrol
      226 root      15   0     0    0    0 S    2  0.0   3:50.37 kswapd0
    16562 zimbra    19   0  4528 2896 1584 S    1  0.1   0:00.02 zmstatuslog
     9213 postfix   17   0  6676 1792 1368 S    0  0.1   0:00.15 showq
     9243 zimbra    17   0  5224 1888 1248 S    0  0.1   0:11.98 zmstat-proc
    29858 root      15   0     0    0    0 S    0  0.0   0:01.40 pdflush
        1 root      16   0   720   52   28 S    0  0.0   0:03.49 init
        2 root      RT   0     0    0    0 S    0  0.0   0:00.02 migration/0
        3 root      34  19     0    0    0 S    0  0.0   0:00.01 ksoftirqd/0
        4 root      RT   0     0    0    0 S    0  0.0   0:00.02 migration/1
        5 root      34  19     0    0    0 S    0  0.0   0:00.31 ksoftirqd/1
        6 root      RT   0     0    0    0 S    0  0.0   0:00.06 migration/2
        7 root      34  19     0    0    0 S    0  0.0   0:00.17 ksoftirqd/2
        8 root      RT   0     0    0    0 S    0  0.0   0:00.10 migration/3
        9 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3
    The only thing running on the box is Zimbra.

    Suggestions?
    cyberdeath

  2. #2
    Jesster's Avatar
    Jesster is offline Trained Alumni
    Join Date
    Nov 2007
    Location
    Santa Barbara, CA.
    Posts
    109
    Rep Power
    7

    Default

    I have similar slowness during backups. My backups are to an iSCSI (gigE) EMC LUN (for now) - it seems for us that mysqld is very busy.

    Has anyone moved mysql to another set of disks outside of /opt/zimbra ? I have a spare RAID10 (4 disks) that might work.

  3. #3
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,367
    Rep Power
    10

    Default

    We spent a while working closely with Zimbra support on a similar memory usage problem with a 6GB RAM box. We still see Zimbra periodically cause swap file usage, but it's fairly minor now (100MB-200MB each week). No one was ever able to identify with absolute certainty any single root cause, but we did review the following items which, collectively, made a difference and got us where we are now.



    1. Ensure Novell's AppArmor is turned off or not installed. SUSE installs this by default.
    2. Turn off the SUSE Firewall (assuming this is safe in your environment).
    3. Reduce Java memory usage percentage by 5% (to 25% in our case).
    4. Reduce MySQL memory usage by 5% (again, to 25% in our case).
    5. In YaST > System > Runlevel Editor, change the default runlevel from 5 to 3.
    6. Delete the "locate" package, if installed.
    7. Reboot the server and see what happens.
    8. Use a properly-sized RAM disk for Amavis's temp files, but you'll probably need to add more RAM to make this worthwhile.


    To find out what the current percent of memory used by Java is, as the zimbra user run:

    zmlocalconfig | grep mailboxd_java_heap


    Now, to change the memory percentage used by Java, run the following command as the zimbra user:

    zmlocalconfig -e mailboxd_java_heap_memory_percent=25


    (but change the 25 above to be 5 less than what you are using now!)

    To change the memory percentage used by MySQL, you need to do three things. First, find the amount of memory currently being used by MySQL:

    zmlocalconfig | grep mysql_memory


    The, run the following command as the zimbra user:

    zmlocalconfig -e mysql_memory_percent=25


    (but again, change the 25 above to be 5 less than what you are using now!)

    Next, edit the /opt/zimbra/conf/my.cnf file to change the following line:

    innodb_buffer_pool_size = xxx


    where "xxx" equals the amount of memory used for the MySQL innodb buffer pool. If you are reducing MySQL memory percent usage from 30% to 25%, take the number that is there already and multiply it by 5/6ths. If what you have now is different than 30%, change the fraction accordingly. (I'd suggest commenting out the existing line and adding your new line, so you can easily revert back to from where you started if needed).

    After the box is rebooted, as the zimbra user run:

    mysql


    and then (at the mysql command prompt), run:

    mysql> show innodb status;

    Look for the following lines toward the end of the report:

    ----------------------
    BUFFER POOL AND MEMORY
    ----------------------
    Total memory allocated 1740081304; in additional pool allocated 1048576
    Buffer pool size 92160
    Free buffers 37789
    Database pages 52539
    Modified db pages 132
    Pending reads 0
    Pending writes: LRU 0, flush list 0, single page 0
    Pages read 51202, created 1337, written 765301
    0.00 reads/s, 0.00 creates/s, 11.05 writes/s
    Buffer pool hit rate 1000 / 1000

    Re run this command a few times while the system is being used heavily; you are looking for two things:

    First, make sure there is a "good" (highly subjective) supply of free buffers.

    Second, watch the buffer pool hit rate. It should rarely drop below 997/1000. If it does, you need more RAM.

    To quit out of mysql, type:

    quit;
    and hit the Enter key.

    Another easy thing to do is to run top, and watch the "%wa" statistic. This represents the percentage of cpu time spent waiting for the system to give the cpu work to do. Typically, this means waiting for the disks. If the %wa is consistently above 25%, in my experience your users will notice a performance hit.

    To make this a little easier on the eyes, as root run in a big window:
    vmstat -n 1

    (Note that you may need to install the vmstat package.)

    On a Zimbra system, the overwhelming majority of disk I/O we have found is Amavis's processing of email in its temp directory. So, if you have consistently high %wa, either you need a faster disk subsystem or to put amavis's temp directory on a RAM disk.

    The latter course requires only sufficient RAM, which is much less disruptive and cheaper usually than replacing a disk subsystem.

    If you can add more RAM to your box and want to know how to configure a RAM disk for Zimbra on SUSE, let me know and I'll post that here as well.

    Hope that helps,
    Mark

  4. #4
    hh2201 is offline Junior Member
    Join Date
    Aug 2008
    Posts
    9
    Rep Power
    6

    Default

    really helpful to me.. it's clearly tell us how to reduce the ram usage..wo~~ many thanks ..

  5. #5
    mayk.backus is offline Intermediate Member
    Join Date
    Apr 2008
    Location
    the Netherlands
    Posts
    22
    Rep Power
    7

    Default

    good information. Thank you for sharing.

    regards,

    Mayk

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •