zimbra is not working well... these weeks

[sfraymondli]

Since I'm investigating what is the problem source of my zimbra server;
It just stands and not working before a REBOOT... pop3 / mta / web-service are all not working when it hangs. Single domain, no IMAP.

running 32-bit ubuntu 6.06 LTS; zimbra OSS 6.04; with BIND9 for DNS and LVM for easy adding vmdk to expand disk usage. Under Vmware server 1.09, host OS is windows server 2003 std.

hardware: IBM X346 Xeon 3.6GHz x2, 3G ram, mirrored RAID 10krpm 146Gx4; 2G ram assigned to the vm-guest.

My system is quite underpowered i knew...

Could anyone helps what is the following in /var/log/syslog?
table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting

it has 10 or more such logs before the server not working.

ulimit -n = 524288; serving around 430 users; without antivirus, daily send+receive around 50k messages or 10Gb transactions;

week days usage: (not quite peak..)

Grand Totals
------------
messages

  21807   received
  28303   delivered
      0   forwarded
     22   deferred  (383  deferrals)
    161   bounced
    608   rejected (2%)
      0   reject warnings
      0   held
      0   discarded (0%)

   2261m  bytes received
   6352m  bytes delivered
   1064   senders
    543   sending hosts/domains
    914   recipients
    197   recipient hosts/domains

df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             4.6G  652M  3.8G  15% /
varrun               1014M   52K 1014M   1% /var/run
varlock              1014M  4.0K 1014M   1% /var/lock
udev                 1014M   80K 1014M   1% /dev
devshm               1014M     0 1014M   0% /dev/shm
/dev/sda4             2.0G  779M  1.2G  41% /var
/dev/mapper/ZMSTORE-ZMLV01
                      148G   70G   75G  49% /opt

[y@w]

Originally Posted by sfraymondli

Since I'm investigating what is the problem source of my zimbra server;
It just stands and not working before a REBOOT... pop3 / mta / web-service are all not working when it hangs. Single domain, no IMAP.

I'm not quite sure I understand this. It stops working and the only thing that fixes it is a reboot?

Anything in /opt/zimbra/log/mailbox.log or /var/log/zimbra.log when it stops working?

[sfraymondli]

at the time of around 17:30, POP3, smtp and webclient also admin gui is not responding until around 17:34, and I ssh and reboot; I knew there will still be no response if i only zmcontrol stop and start again:

Dec 29 17:35:25 myhost postfix/cleanup[13828]: 8B88A9EC00E: message-id=<20091229093525.8B88A9EC00E@mail.mydomain.com>
Dec 29 17:35:25 myhost postfix/qmgr[10969]: 8B88A9EC00E: from=<simonpang@mail.mydomain.com>, size=28844, nrcpt=1 (queue active)
Dec 29 17:35:25 myhost amavis[13951]: (13951-06) ESMTP::10024 /opt/zimbra/data/amavisd/tmp/amavis-20091229T173133-13951: <simonpang@mail.mydomain.com> -> <852cs05@mail.mydomain.com> SIZE=28844 Received: from mail.mydomain.com ([127.0.0.1]) by localhost (mail.mydomain.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP for <852cs05@mail.mydomain.com>; Tue, 29 Dec 2009 17:35:25 +0800 (HKT)
Dec 29 17:35:25 myhost amavis[13951]: (13951-06) Checking: y0KMuqZ5-zLZ MYNETS [10.2.8.101] <simonpang@mail.mydomain.com> -> <852cs05@mail.mydomain.com>
Dec 29 17:35:26 myhost postfix/smtpd[13316]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[13561]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[13595]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[13343]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[13586]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[12764]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[12701]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[735]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[13592]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[13315]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[13411]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[32447]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[13575]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[12702]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[13554]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:26 myhost postfix/smtpd[14949]: connect from unknown[10.2.1.225]
Dec 29 17:35:26 myhost saslauthd[11013]: zmauth: authenticating against elected url 'https://mail.mydomain.com:7071/service/admin/soap/' ...
Dec 29 17:35:27 myhost postfix/smtpd[13312]: 368649EC023: client=unknown[10.2.9.23], sasl_method=LOGIN, sasl_username=fungyeung
Dec 29 17:35:27 myhost postfix/cleanup[14037]: 368649EC023: message-id=<014101ca8869$bffdb7e0$0132a8c0@hk852w014>
Dec 29 17:35:27 myhost saslauthd[11013]: zmpost: url='https://mail.mydomain.com:7071/service/admin/soap/' returned buffer->data='<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope"><soap:Header><context xmlns="urn:zimbra"><change token="396507"/></context></soap:Header><soap:Body><AuthResponse xmlns="urn:zimbraAccount"><authToken>0_ecda3e81daefb81ad5532c6ca434e9115140fe59_69643d33363a66323737636633632d336539332d343963652d393336382d3866666634633462386432623b6578703d31333a313236323038363532373835353b76763d313a303b747970653d363a7a696d6272613b</authToken><lifetime>7200000</lifetime><skin>zmail</skin></AuthResponse></soap:Body></soap:Envelope>', hti->error=''
Dec 29 17:35:27 myhost saslauthd[11013]: auth_zimbra: faxadmin2 auth OK...
.
.
.
Dec 29 17:35:35 myhost postfix/smtp[13277]: 368649EC023: to=<tommypoon@mail.mydomain.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=8.7, delays=3/0.04/0.08/5.6, dsn=2.0.0, status=sent (250 2.0.0 Ok, id=14589-01, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 669A99EC00E)
Dec 29 17:35:35 myhost postfix/smtp[13277]: 368649EC023: to=<yantakhung@mail.mydomain.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=8.7, delays=3/0.04/0.08/5.6, dsn=2.0.0, status=sent (250 2.0.0 Ok, id=14589-01, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 669A99EC00E)
Dec 29 17:35:35 myhost postfix/qmgr[10969]: 368649EC023: removed
Dec 29 17:35:38 myhost amavis[14589]: (14589-01) extra modules loaded: /opt/zimbra/zimbramon/lib/i486-linux-gnu-thread-multi/auto/Net/SSLeay/autosplit.ix, /opt/zimbra/zimbramon/lib/i486-linux-gnu-thread-multi/auto/Net/SSLeay/randomize.al, IO/Socket/SSL.pm, Net/LDAP/Extension.pm, Net/SSLeay.pm
Dec 29 17:35:39 myhost CRON[13598]: (pam_unix) session closed for user zimbra
Dec 29 17:35:42 myhost postfix/smtpd[13312]: table hash:/etc/aliases(0,lock|fold_fix) has changed -- restarting
Dec 29 17:35:42 myhost postfix/smtpd[14949]: connect from unknown[10.2.1.227]
Dec 29 17:35:42 myhost saslauthd[11012]: zmauth: authenticating against elected url 'https://mail.mydomain.com:7071/service/admin/soap/' ...
Dec 29 17:35:42 myhost saslauthd[11012]: zmpost: url='https://mail.mydomain.com:7071/service/admin/soap/' returned buffer->data='<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope"><soap:Header><context xmlns="urn:zimbra"><change token="306091"/></context></soap:Header><soap:Body><AuthResponse xmlns="urn:zimbraAccount"><authToken>0_412df948f0300717d35c449d408167cad103b303_69643d33363a38356330313834612d333466332d343334612d383461632d3738393038646338363730643b6578703d31333a313236323136353734323430343b747970653d363a7a696d6272613b</authToken><lifetime>86400000</lifetime><skin>sky</skin></AuthResponse></soap:Body></soap:Envelope>', hti->error=''
Dec 29 17:35:42 myhost saslauthd[11012]: auth_zimbra: faxadmin auth OK
Dec 29 17:35:42 myhost postfix/smtpd[14949]: 81DDA9EC023: client=unknown[10.2.1.227], sasl_method=LOGIN, sasl_username=faxadmin
Dec 29 17:35:42 myhost postfix/smtpd[14949]: 93AD29EC023: client=unknown[10.2.1.227], sasl_method=LOGIN, sasl_username=faxadmin
Dec 29 17:35:42 myhost postfix/cleanup[14154]: 93AD29EC023: message-id=<b4b101ca886a$4c16bd30$e301020a@hk852faxcs>
Dec 29 17:35:42 myhost postfix/qmgr[10969]: 93AD29EC023: from=<faxadmin@mail.mydomain.com>, size=38464, nrcpt=1 (queue active)
Dec 29 17:35:42 myhost amavis[13337]: (13337-10) ESMTP::10024 /opt/zimbra/data/amavisd/tmp/amavis-20091229T172857-13337: <faxadmin@mail.mydomain.com> -> <852cs01@mail.mydomain.com> SIZE=38464 Received: from mail.mydomain.com ([127.0.0.1]) by localhost (mail.mydomain.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP for <852cs01@mail.mydomain.com>; Tue, 29 Dec 2009 17:35:42 +0800 (HKT)
Dec 29 17:35:42 myhost postfix/postfix-script[15018]: stopping the Postfix mail system
Dec 29 17:35:42 myhost amavis[13337]: (13337-10) Checking: B81h4VJ8Drwa MYNETS [10.2.1.227] <faxadmin@mail.mydomain.com> -> <852cs01@mail.mydomain.com>
Dec 29 17:35:42 myhost postfix/smtpd[14949]: disconnect from unknown[10.2.1.227]
Dec 29 17:35:43 myhost postfix/master[10963]: terminating on signal 15
Dec 29 17:35:44 myhost amavis[13337]: (13337-10) mangling by altermime (disclaimer) done, new size: 38404, orig 38464 bytes
Dec 29 17:35:44 myhost amavis[13337]: (13337-10) (!)FWD via SMTP: <faxadmin@mail.mydomain.com> -> <852cs01@mail.mydomain.com>, 450 4.4.1 Can't connect to INET4 socket 127.0.0.1: Connection refused, MTA([127.0.0.1]:10025), id=13337-10
Dec 29 17:35:44 myhost amavis[13337]: (13337-10) Blocked MTA-BLOCKED, MYNETS LOCAL [10.2.1.227] [10.2.1.227] <faxadmin@mail.mydomain.com> -> <852cs01@mail.mydomain.com>, Message-ID: <b4b101ca886a$4c16bd30$e301020a@hk852faxcs>, mail_id: B81h4VJ8Drwa, Hits: -2.677, size: 38464, 2048 ms
Dec 29 17:35:45 myhost saslauthd[11008]: server_exit     : master exited: 11008
Dec 29 17:35:49 myhost zimbramon[14309]: 14309:info: Stopping spell via zmcontrol
Dec 29 17:35:52 myhost zimbramon[14309]: 14309:info: Stopping snmp via zmcontrol
Dec 29 17:35:52 myhost CRON[14045]: (pam_unix) session closed for user zimbra
Dec 29 17:35:53 myhost slapd[3966]: connection_input: conn=56517 deferring operation: binding
Dec 29 17:35:55 myhost zimbramon[14309]: 14309:info: Stopping archiving via zmcontrol
Dec 29 17:35:58 myhost amavis[8354]: Net::Server: 2009/12/29-17:35:58 Server closing!
Dec 29 17:36:01 myhost CRON[15166]: (pam_unix) session opened for user zimbra by (uid=0)
Dec 29 17:36:04 myhost zimbramon[14309]: 14309:info: Stopping antivirus via zmcontrol
Dec 29 17:36:05 myhost CRON[14269]: (pam_unix) session closed for user zimbra
Dec 29 17:36:10 myhost zimbramon[14309]: 14309:info: Stopping antispam via zmcontrol
Dec 29 17:36:12 myhost zimbramon[14309]: 14309:info: Stopping imapproxy via zmcontrol
Dec 29 17:36:12 myhost zimbramon[14309]: 14309:info: Stopping memcached via zmcontrol
Dec 29 17:36:12 myhost zimbramon[14309]: 14309:info: Stopping mailbox via zmcontrol
Dec 29 17:36:17 myhost zmmailboxdmgr[15345]: status requested
Dec 29 17:36:17 myhost zmmailboxdmgr[15345]: status OK

and the server is strange that i type reboot, it is not quite responsive at that moment until i stop the zimbra manually by zmcontrol stop; and the zmcontrol stopped quite fast as normal, using only around 30-60 seconds only!

very appreciate for your help!!!

[xeon]

Whats your disk usage?

iostat -mdx 3

If you are seeing high disk usage during the issues. You have a disk IO bottleneck. If thats the case, I would start by disabling the logger and see if that improves performance.

Hope this helps.