Zimbra offers Open Source email server software and shared calendar for Linux and the Mac
Go Back   Zimbra :: Forums > Zimbra Collaboration Suite > Administrators

Welcome to the Zimbra :: Forums!
Welcome, if you would like to post a comment please register. We also encourage you to explore all things Zimbra with our team and members of the community.

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 08-21-2008, 12:23 PM
Active Member
 
Posts: 37
Unhappy Memory Usage and Slowness + Backup Error

Hello everyone,

I'm experiencing a strange problem since upgrading to 5.0.8 (and 5.0.9) and enabling archiving/discovery. Once I load up zimbra and have it running for a while, the system uses up a huge amount of memory and then clutters up the SWAP file (slowing performance). This has only occurred recently. The only thing I did recently (aside from upgrading + archiving/discovery) was fix the logger mysql database so now it actually gives me a report every day (before it would error and die). What can be done about this memory usage? This is a small server with only 200 users on it. Performance wise: it's a 2xQuad Core Xeon with 4GB of memory running on 32bit SuSE ES10.1 with 15K RPM drives running in a RAID 5.

Here is the error message I get when I backup the system nightly:
Code:
system failure: backing up table: mboxgroup67.appointment
com.zimbra.common.service.ServiceException: system failure: backing up table: mboxgroup67.appointment
ExceptionId:BackupSetWorkerThread-DB:1219208557644:5d072cfe01bf9626
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253)
        at com.zimbra.cs.db.DbBackup.saveTable(DbBackup.java:132)
        at com.zimbra.cs.backup.BackupAccountSession.storeTablesToLocalFile(BackupAccountSession.java:276)
        at com.zimbra.cs.backup.FileBackupTarget$BackupAcctSession.storeTablesStage(FileBackupTarget.java:1294)
        at com.zimbra.cs.backup.FileBackupTarget$BackupSetWorkerThread.run(FileBackupTarget.java:2494)
Caused by: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTO OUTFILE '/opt/zimbra/backup/tmp/full-20080820.050013.549/accounts/855/4a9/8' at line 1
Here is "top":
Code:
top - 15:28:02 up  6:36,  1 user,  load average: 2.64, 2.89, 3.64
Tasks: 196 total,   2 running, 194 sleeping,   0 stopped,   0 zombie
Cpu(s): 15.3%us,  1.9%sy,  0.0%ni, 72.5%id, 10.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3365168k total,  3236312k used,   128856k free,    14948k buffers
Swap:  4200988k total,  1738020k used,  2462968k free,  2114112k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7282 zimbra    16   0 1787m 164m 3320 S  100  5.0 385:08.73 mysqld
 6746 zimbra    16   0  108m  13m 2296 S   16  0.4  40:04.86 mysqld
16568 zimbra    19   0  6160 4528 1628 S    2  0.1   0:00.07 zmcontrol
  226 root      15   0     0    0    0 S    2  0.0   3:50.37 kswapd0
16562 zimbra    19   0  4528 2896 1584 S    1  0.1   0:00.02 zmstatuslog
 9213 postfix   17   0  6676 1792 1368 S    0  0.1   0:00.15 showq
 9243 zimbra    17   0  5224 1888 1248 S    0  0.1   0:11.98 zmstat-proc
29858 root      15   0     0    0    0 S    0  0.0   0:01.40 pdflush
    1 root      16   0   720   52   28 S    0  0.0   0:03.49 init
    2 root      RT   0     0    0    0 S    0  0.0   0:00.02 migration/0
    3 root      34  19     0    0    0 S    0  0.0   0:00.01 ksoftirqd/0
    4 root      RT   0     0    0    0 S    0  0.0   0:00.02 migration/1
    5 root      34  19     0    0    0 S    0  0.0   0:00.31 ksoftirqd/1
    6 root      RT   0     0    0    0 S    0  0.0   0:00.06 migration/2
    7 root      34  19     0    0    0 S    0  0.0   0:00.17 ksoftirqd/2
    8 root      RT   0     0    0    0 S    0  0.0   0:00.10 migration/3
    9 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3
The only thing running on the box is Zimbra.

Suggestions?
__________________
cyberdeath
Reply With Quote
  #2 (permalink)  
Old 09-02-2008, 09:28 PM
Trained Alumni
 
Posts: 108
Default

I have similar slowness during backups. My backups are to an iSCSI (gigE) EMC LUN (for now) - it seems for us that mysqld is very busy.

Has anyone moved mysql to another set of disks outside of /opt/zimbra ? I have a spare RAID10 (4 disks) that might work.
Reply With Quote
  #3 (permalink)  
Old 09-03-2008, 11:00 AM
Moderator
 
Posts: 1,209
Default

We spent a while working closely with Zimbra support on a similar memory usage problem with a 6GB RAM box. We still see Zimbra periodically cause swap file usage, but it's fairly minor now (100MB-200MB each week). No one was ever able to identify with absolute certainty any single root cause, but we did review the following items which, collectively, made a difference and got us where we are now.


  1. Ensure Novell's AppArmor is turned off or not installed. SUSE installs this by default.
  2. Turn off the SUSE Firewall (assuming this is safe in your environment).
  3. Reduce Java memory usage percentage by 5% (to 25% in our case).
  4. Reduce MySQL memory usage by 5% (again, to 25% in our case).
  5. In YaST > System > Runlevel Editor, change the default runlevel from 5 to 3.
  6. Delete the "locate" package, if installed.
  7. Reboot the server and see what happens.
  8. Use a properly-sized RAM disk for Amavis's temp files, but you'll probably need to add more RAM to make this worthwhile.

To find out what the current percent of memory used by Java is, as the zimbra user run:

zmlocalconfig | grep mailboxd_java_heap


Now, to change the memory percentage used by Java, run the following command as the zimbra user:

zmlocalconfig -e mailboxd_java_heap_memory_percent=25


(but change the 25 above to be 5 less than what you are using now!)

To change the memory percentage used by MySQL, you need to do three things. First, find the amount of memory currently being used by MySQL:

zmlocalconfig | grep mysql_memory


The, run the following command as the zimbra user:

zmlocalconfig -e mysql_memory_percent=25


(but again, change the 25 above to be 5 less than what you are using now!)

Next, edit the /opt/zimbra/conf/my.cnf file to change the following line:

innodb_buffer_pool_size = xxx


where "xxx" equals the amount of memory used for the MySQL innodb buffer pool. If you are reducing MySQL memory percent usage from 30% to 25%, take the number that is there already and multiply it by 5/6ths. If what you have now is different than 30%, change the fraction accordingly. (I'd suggest commenting out the existing line and adding your new line, so you can easily revert back to from where you started if needed).

After the box is rebooted, as the zimbra user run:

mysql


and then (at the mysql command prompt), run:
mysql> show innodb status;
Look for the following lines toward the end of the report:
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 1740081304; in additional pool allocated 1048576
Buffer pool size 92160
Free buffers 37789
Database pages 52539
Modified db pages 132
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages read 51202, created 1337, written 765301
0.00 reads/s, 0.00 creates/s, 11.05 writes/s
Buffer pool hit rate 1000 / 1000
Re run this command a few times while the system is being used heavily; you are looking for two things:

First, make sure there is a "good" (highly subjective) supply of free buffers.

Second, watch the buffer pool hit rate. It should rarely drop below 997/1000. If it does, you need more RAM.

To quit out of mysql, type:
quit;
and hit the Enter key.
Another easy thing to do is to run top, and watch the "%wa" statistic. This represents the percentage of cpu time spent waiting for the system to give the cpu work to do. Typically, this means waiting for the disks. If the %wa is consistently above 25%, in my experience your users will notice a performance hit.

To make this a little easier on the eyes, as root run in a big window:
vmstat -n 1
(Note that you may need to install the vmstat package.)

On a Zimbra system, the overwhelming majority of disk I/O we have found is Amavis's processing of email in its temp directory. So, if you have consistently high %wa, either you need a faster disk subsystem or to put amavis's temp directory on a RAM disk.

The latter course requires only sufficient RAM, which is much less disruptive and cheaper usually than replacing a disk subsystem.

If you can add more RAM to your box and want to know how to configure a RAM disk for Zimbra on SUSE, let me know and I'll post that here as well.

Hope that helps,
Mark
__________________
___________________________________
L. Mark Stone, CIO


"Uptime. All the time."

477 Congress Street | Portland, ME 04101-3431 | (207) 772-5678

proactive maintenance and monitoring | technology consulting
Zimbra groupware | EMR implementations | private cloud hosting
Reply With Quote
  #4 (permalink)  
Old 09-03-2008, 06:56 PM
Junior Member
 
Posts: 9
Default

really helpful to me.. it's clearly tell us how to reduce the ram usage..wo~~ many thanks ..
Reply With Quote
  #5 (permalink)  
Old 09-04-2008, 08:14 AM
Intermediate Member
 
Posts: 22
Default

good information. Thank you for sharing.

regards,

Mayk
Reply With Quote
Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes


Why Join?

Registering let's you ask questions, makes it easier to search, displays any files attached to posts, and notifies you about replies.

blog.zimbra.com




 

SEO by vBSEO ©2011, Crawlability, Inc.