Server stops responding every Monday morning

[cpalsgrove]

For the past two Mondays, our server has stopped responding just before the workday starts. Last Monday it stopped at 8:11 am and today it stopped at 8:20 am. At these times, users were not able to connect to the server in any way to get their mail, and there was absolutely no activity in /opt/zimbra/log/maillog. I was able to ssh to the server and do a 'zmcontrol status' which showed everything to be running.

At that point, I restarted Zimbra where it failed to start antivirus successfully. I ran 'zmclamdctl restart' which seemed to work. I did a 'zmcontrol status' to find that mailboxd was not running properly, and it gave the same error after I restarted it. I then restarted the whole server which brought everything back up as it was supposed to be. (See all commands and output up until server restart below.)

The only thing somewhere around this time that I can see that would affect the system is the spam log rotate at 8:08 am in the crontab:

8 8 * * * [ -f /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log

But then the question: Why only on Mondays and why did this just start happening recently? At this point I am pretty confused and don't know what else to check. Any suggestions?

*****************************
[root@mail ~]# su - zimbra
[zimbra@mail ~]$ zmcontrol status
Host mail.mcdaniel.edu
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running
stats Running
[zimbra@mail ~]$ less /opt/zimbra/log/mailbox.log
[zimbra@mail ~]$ zmcontrol stop; zmcontrol start
Host mail.mcdaniel.edu
Stopping stats...Done
Stopping mta...Done
Stopping spell...Done
Stopping snmp...Done
Stopping archiving...Done
Stopping antivirus...Done
Stopping antispam...Done
Stopping imapproxy...Done
Stopping mailbox...FAILED
Stopping zmmtaconfig...zmmtaconfig is not running.
Stopping mailboxd...done.
Stopping mysqld...failed.


Stopping logger...Done
Stopping ldap...Done
Host mail.mcdaniel.edu
Starting ldap...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting antispam...Done.
Starting antivirus...FAILED
Starting zmmtaconfig...zmmtaconfig is already running.
Starting amavisd...amavisd is already running.
Starting clamd...ClamAV update process started at Mon Aug 24 08:36:32 2009
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.95.1 Recommended version: 0.95.2
DON'T PANIC! Read Clam AntiVirus
Downloading main-51.cdiff [100%]
main.cld updated (version: 51, sigs: 545035, f-level: 42, builder: sven)
WARNING: getfile: daily-9214.cdiff not found on remote server (IP: 65.120.238.2)
WARNING: getpatch: Can't download daily-9214.cdiff from db.us.clamav.net
Trying host db.us.clamav.net (138.123.96.134)...
WARNING: getfile: daily-9214.cdiff not found on remote server (IP: 138.123.96.134)
WARNING: getpatch: Can't download daily-9214.cdiff from db.us.clamav.net
Trying host db.us.clamav.net (194.47.250.218)...
WARNING: getfile: daily-9214.cdiff not found on remote server (IP: 194.47.250.218)
WARNING: getpatch: Can't download daily-9214.cdiff from db.us.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd
Downloading daily.cvd [99%]]
daily.cvd updated (version: 9730, sigs: 67614, f-level: 43, builder: ccordes)
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Current functionality level = 42, recommended = 43
DON'T PANIC! Read Clam AntiVirus
Database updated (612649 signatures) from db.us.clamav.net (IP: 65.120.238.2)
connect(): Connection refused
WARNING: Clamd was NOT notified: Can't connect to clamd on localhost:3310
failed.


Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.
[zimbra@mail ~]$ zmclamdctl
Usage: /opt/zimbra/bin/zmclamdctl start|stop|kill|restart|status
[zimbra@mail ~]$ zmclamdctl restart
Stopping clamd...done.
Starting clamd...done.
[zimbra@mail ~]$ zmcontrol status
Host mail.mcdaniel.edu
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Stopped
zmmtaconfig is running.
mysql.server is not running
mailboxd is running.
mta Running
snmp Running
spell Running
stats Stopped
[zimbra@mail ~]$ zmmailboxdctl restart
Stopping mailboxd...done.
Starting mailboxd...done.
[zimbra@mail ~]$ zmcontrol status
Host mail.mcdaniel.edu
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Stopped
zmmtaconfig is running.
mysql.server is not running
mailboxd is running.
mta Running
snmp Running
spell Running
stats Stopped
[zimbra@mail ~]$ logout
[root@mail ~]# shutdown -r now
*****************************

[andrewfn]

Mine stopped this morning as well. Bizarre!
I shut it down and did a restart and everything is fine again.