Antivirus Crashing nightly

[dionrowney]

We stop the server every night to do the rsync to guarentee the setup is in a steady state for the backup. We assume It was during the restart that the AV usually died.

Here is our script

#!/bin/sh
#
# This scripts monitors the AV status and if it does not restart successfully in morning
# it will start it again.
#

export PATH=$PATH:/opt/zimbra/bin
AVSTATUS=`zmcontrol status |grep antivirus|awk '{ print $2 }'`

date
echo Antivirus is $AVSTATUS

if [[ $AVSTATUS != "Running" ]]; then
        #Start the AV service if not running
        echo Antivirus was not running after backup - Attempting start of antivirus
        zmclamdctl restart
        zmcontrol status
fi

date

Also : I just checked the log for this script and it turns out to be having to start the service ever 5 - 7 days or so. So the AV is still failing. Changing th update and backup times didnt fix it.

Here is the error I see when the system attempts to start:

        Starting ldap...Done.
        Starting logger...Done.
        Starting mailbox...Done.
        Starting antispam...Done.
        Starting antivirus...FAILED
amavisd already running: pid 19085
ClamAV update process started at Fri Jan 23 03:52:47 2009
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.93.3 Recommended version: 0.94.2
DON'T PANIC! Read http://www.clamav.net/support/faq

Downloading main-49.cdiff [100%]

main.cld updated (version: 49, sigs: 437972, f-level: 35, builder: sven)
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Current functionality level = 33, recommended = 35
DON'T PANIC! Read http://www.clamav.net/support/faq
WARNING: getfile: daily-7652.cdiff not found on remote server (IP: 64.246.134.219)
WARNING: getpatch: Can't download daily-7652.cdiff from db.us.clamav.net
WARNING: getfile: daily-7652.cdiff not found on remote server (IP: 138.123.96.134)
WARNING: getpatch: Can't download daily-7652.cdiff from db.us.clamav.net
WARNING: getfile: daily-7652.cdiff not found on remote server (IP: 64.246.134.219)
WARNING: getpatch: Can't download daily-7652.cdiff from db.us.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd

Downloading daily.cvd [100%]

daily.cvd updated (version: 8895, sigs: 61157, f-level: 38, builder: mcichosz)
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Current functionality level = 33, recommended = 38
DON'T PANIC! Read http://www.clamav.net/support/faq
Database updated (499129 signatures) from db.us.clamav.net (IP: 208.67.80.27)
WARNING: Clamd was NOT notified: Can't connect to clamd on 127.0.0.1:3310
connect(): Connection refused
clamd failed to start


        Starting snmp...Done.
        Starting spell...Done.
        Starting mta...Done.
        Starting stats...Done.

[dmmincrjr]

Thank you for posting the script. I have implemented it as a cron job to run a few minutes after my nightly backup stops. I just started experiencing this issue on Tuesday 1/20/09 after having the backup run successfully for over a year. I changed to FIOS for internet access on Monday 1/19 and received a new static ip so I am not sure if this had something to do with this problem developing. The machine is behind a firewall and I am using the split-dns setup so I did not need to make any configuration changes on the mail server. It's just weird the problem showed up after making the switch however. The backup has run correctly 2 out of the 5 nights since the change. I can stop and start zimbra manually and it always starts without the error occurring. It just seems to be from the cron job their is an issue. I am running version 5.0.11 on Centos 4.5. My log files appear to show the same information as contained in previous posts. Hopefully someone will be able to come up with what is causing this error to occur.

[pjfawcett]

I have been having the same problem, i.e. Antivirus not starting after a shutdown for backup.

Looking through the logs there appeared to be some correlation between clamd not starting and freshclam taking a longer time to download updates (either because there were more updates or because of download failures and necessary retries).

I wondered if the "Can't open file /opt/zimbra/data/clamav/db/main.cvd" error might be due to some clash between clamd trying to load the virus definitions while freshclam is still trying to update them.

I am currently running version 5.0.12 on Ubuntu 8.04 64 bit.

I am going to edit "/opt/zimbra/bin/zmclamdctl" so that freshclam is not started until clamd has successfully started.

I'll give it a few days to see if the problem goes away.

[pjfawcett]

Ah well. That didn't fix things. It failed on the next startup, but not on the following (manual) one.

Will need to invetigate further.

[gbrandt]

I'm getting this as well. Ubuntu 8.04 LTS, 5.0.14. Occasionally on a restart after backup I get:

LibClamAV Error: cli_load(): Can't open file /opt/zimbra/data/clamav/db/main.cvd

This happens every 4 or 5 days.

Any solutions yet? Has anybody filed a bug report?

[pjfawcett]

I haven't filed a bug report on this. I think it is a ClamAV error rather than a Zimbra error. The fact that Zimbra isn't running the latest ClamAV code adds further complication.

As a work around I use the following approach:

I have a script '/root/StartZimbra.sh' as follows:

#!/bin/bash
su -c 'zmcontrol start' -l zimbra

In my experience, aided by some examination of the code, it is safe to run this command when Zimbra is running - in such circumstances it will just start those services that are not currently running (in my case the 'antivirus').

I then have a crontab entry for the 'root' user as follows:

#
# Try to ensure Zimbra is running fully (in case something didn't start after reboot
#
00 03 * * * /root/StartZimbra.sh

The 03:00am time is chosen to be about half an hour after Zimbra has been restarted following shutdown for backup.

If the antivirus has failed to start after the backup then this second script has, to date, always succeeded in starting it. As I said above, it is safe to run this even if all the other services are running so I don't bother with any checks to see if it is already running.

The result of this is that, when the AV fails to start (which is still does every now and then) it is only down for about half an hour. During that time messages get queued internally but clear when the AV is started. (Most of them are spam at that time of night).

I can tell if the AV has failed because the "Daily mail report" will show a slew of errors between 2:30am and 3:00am.

As I said, the error still occurs every few days, but with this workaround it is no longer much of a problem and so tracing the root cause has slipped down my priority list.

[dmmincrjr]

I also have not filed as bug report but am still experiencing the problem. I have upgraded to version 5.0.16 hoping each upgrade would correct the problem as I am not sure if it lies with Zimbra or Clam. The restart script provided earlier in this thread has always restarted anti-virus. You just need to add the script back to the cron jobs after upgrades. My anti-virus again failed to start this morning and here is the snippet of text from the freshclam.log

ClamAV update process started at Thu May 14 04:58:35 2009
ERROR: Problem with internal logger (UpdateLogFile = /opt/zimbra/log/freshclam.log).
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.94.1 Recommended version: 0.95.1
DON'T PANIC! Read http://www.clamav.net/support/faq
Trying host db.us.clamav.net (168.143.19.95)...
Downloading main-50.cdiff [100%]
main.cld updated (version: 50, sigs: 500667, f-level: 38, builder: sven)
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Current functionality level = 37, recommended = 38
DON'T PANIC! Read http://www.clamav.net/support/faq
WARNING: getfile: daily-8543.cdiff not found on remote server (IP: 168.143.19.95)
WARNING: getpatch: Can't download daily-8543.cdiff from db.us.clamav.net
Trying host db.us.clamav.net (168.143.19.95)...
WARNING: getfile: daily-8543.cdiff not found on remote server (IP: 168.143.19.95)
WARNING: getpatch: Can't download daily-8543.cdiff from db.us.clamav.net
Trying host db.us.clamav.net (168.143.19.95)...
WARNING: getfile: daily-8543.cdiff not found on remote server (IP: 168.143.19.95)
WARNING: getpatch: Can't download daily-8543.cdiff from db.us.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd
Trying host db.us.clamav.net (168.143.19.95)...
Downloading daily.cvd [100%]
daily.cvd updated (version: 9357, sigs: 49175, f-level: 42, builder: neo)
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Current functionality level = 37, recommended = 42
DON'T PANIC! Read http://www.clamav.net/support/faq
Database updated (549842 signatures) from db.us.clamav.net (IP: 168.143.19.95)
WARNING: Clamd was NOT notified: Can't connect to clamd on localhost:3310

The error problem with internal logger is showing up in the freshclam.log when it fails. I can't figure out what is causing the problem or how to fix and why it might work for a week or better and then fail for a couple days. However like I said the script seems to restart the service so as long as that works this is down on my priority list even though it is annoying.

[paper]

Hello,

I've had similar problems, every few days my clam stopped working

Sat Sep 5 02:16:28 2009 -> --- Stopped at Sat Sep 5 02:16:28 2009
Sat Sep 5 02:16:40 2009 -> +++ Started at Sat Sep 5 02:16:40 2009
Sat Sep 5 02:16:40 2009 -> clamd daemon 0.94.1-broken-compiler (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Sat Sep 5 02:16:40 2009 -> Log file size limited to 20971520 bytes.
Sat Sep 5 02:16:40 2009 -> Reading databases from /opt/zimbra/data/clamav/db
Sat Sep 5 02:16:40 2009 -> Not loading PUA signatures.
LibClamAV Error: Can't load /opt/zimbra/data/clamav/db/main.cvd: MD5 verification error
Sat Sep 5 02:16:41 2009 -> ERROR: MD5 verification error
ERROR: MD5 verification error


This began cycling (with corrupted DB) and mails were not going through. 'zmclamdctl restart' did not help me, since DB was corrupted. I modified a script i got here to:

#!/bin/sh
#
# This scripts monitors the AV status and if it does not restart successfully in morning
# it will start it again.
#

AVSTATUS=`zmcontrol status |grep antivirus|awk '{ print $2 }'`

date >> /root/backupscripts/zimbra_av.log
echo Antivirus is $AVSTATUS >> /root/backupscripts/zimbra_av.log

if [ $AVSTATUS != "Running" ]
then
  echo Antivirus was not running, restoring clamdb and restarting.
  /opt/zimbra/bin/zmclamdctl stop >> /root/backupscripts/zimbra_av.log
  rm -rf /opt/zimbra/data/clamav/db/* >> /root/backupscripts/zimbra_av.log
  /opt/zimbra/clamav/bin/freshclam --config-file=/opt/zimbra/conf/freshclam.conf >> /root/backupscripts/zimbra_av.log
  /opt/zimbra/bin/zmclamdctl start >> /root/backupscripts/zimbra_av.log
fi

AVSTATUS=`zmcontrol status |grep antivirus|awk '{ print $2 }'`
echo Antivirus is $AVSTATUS >> /root/backupscripts/zimbra_av.log

date >> /root/backupscripts/zimbra_av.log

I'll see how it goes in next days, in few tests when I manually crashed clam, it helped. My zimbra version: Release 5.0.16_GA_2921.UBUNTU8_64 UBUNTU8_64 FOSS edition

Best Regards