[SOLVED] mail report says "no messages" or only a few after upgrade to zcs5

[lzmarine]

After upgrade from 5.0.10 to 5.0.11 on Ubuntu 8.0.4, the mail report usually reports no messages, or occaisionally 1 or 2 messages. I have spam and non-spam mail in and out daily. Previous reports were rather lengthy with much detail. Anyone else seen or heard of this? Environment and sample reports below.

Ubuntu 8.04.1
Linux 2.6.24-19-server #1 SMP Wed Aug 20 23:54:28 UTC 2008 i686 GNU/Linux

Sent: Sunday, November 30, 2008 1:10:23 AM GMT -05:00 US/Canada Eastern
Subject: Daily mail report from 2008-11-29 00:00:00 to 2008-11-30 00:00:00

Generating report
From 2008-11-29 00:00:00 to 2008-11-30 00:00:00

No messages found

-- OR --

Sent: Saturday, November 29, 2008 1:10:29 AM GMT -05:00 US/Canada Eastern
Subject: Daily mail report from 2008-11-28 00:00:00 to 2008-11-29 00:00:00

Generating report
From 2008-11-28 00:00:00 to 2008-11-29 00:00:00


2 messages found for 2 total recipients (1 unique)
8158 total bytes
4079.00 average bytes/msg
1.00 average recipients/msg
0.00 average delay/msg (sec)

Errors

Most active senders
1 xxxx@harborhay.com
1 xxxx@bananarepublic.delivery.net

Most active recipients
2 xxxx@magnoliasevents.com

[dwmtractor]

I just upgraded from 5.0.8 to 5.0.11 last night and my mail report is normal, so whatever it is, your problem isn't across the board with .11. Have you checked your /var/log/zimbra.log or /var/log/mail.log to see if there are any errors that might indicate rejected messages? How about doing zmcontrol status from the command line? Are all your services running?

[lzmarine]

Originally Posted by dwmtractor

I just upgraded from 5.0.8 to 5.0.11 last night and my mail report is normal, so whatever it is, your problem isn't across the board with .11. Have you checked your /var/log/zimbra.log or /var/log/mail.log to see if there are any errors that might indicate rejected messages? How about doing zmcontrol status from the command line? Are all your services running?

My only error in zimbra.log is:
Dec 3 10:50:52 mail amavis[20314]: (20314-08) WARN: MIME::Parser error: part did not end with expected boundary

I do see the following in mail.log every 2 minutes:
Dec 3 06:20:11 mail zmmailboxdmgr[18512]: status requested
Dec 3 04:56:09 mail zmmailboxdmgr[8606]: assuming no other instance is running
Dec 3 04:56:09 mail zmmailboxdmgr[8606]: no manager process is running
Dec 3 04:58:08 mail zmmailboxdmgr[8838]: status requested
Dec 3 04:58:08 mail zmmailboxdmgr[8838]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist

The following is in the mail.log every 10 minutes:
Dec 3 06:20:02 mail postfix/postqueue[18314]: fatal: Queue report unavailable - mail system is down

Are the stopped watchers, below, the problem?

mail:~# sudo -u zimbra /opt/zimbra/bin/zmcontrol status
Host mail.domain.com
antispam Running
antivirus Running
ldap Running
logger Stopped
zmlogswatchctl is not running
mailbox Running
mta Running
snmp Stopped
zmswatch is not running.
spell Running
stats Running
mail:~#

[Jesster]

Originally Posted by lzmarine

Are the stopped watchers, below, the problem?

mail:~# sudo -u zimbra /opt/zimbra/bin/zmcontrol status
Host mail.domain.com
antispam Running
antivirus Running
ldap Running
logger Stopped
zmlogswatchctl is not running
mailbox Running
mta Running
snmp Stopped
zmswatch is not running.
spell Running
stats Running
mail:~#

Simply put, "yes".

Logger is responsible for your reports (and stats/trace).

Logger - Zimbra :: Wiki

[lzmarine]

I spent a few hours trying to troubleshoot to no avail. I also installed Date::Time module for perl (from a si milar thread solution) but that didn't seem to have any affect. I then re-ran the same install and this morning I have more "normal" reports as well as the service start messages (from offline backup script.) I don't know the root cause, but re-running 5.0.11 install seemed to work.

[lzmarine]

I thought this was solved by reinstall, but no back to the empty report. The following thread might be te cause: 5.011 Bug - zmlogger fails to stop

[lzmarine]

Running: /opt/zimbra/bin/logmysql zimbra_logger
> select * from mta;

This produces no output for dates after the upgrade from 5.0.10 to 5.0.11 and an offline backup. Re-install thought to fix this as the next days report had data again; but, after the nightly backup (zmcontrol stop; zmcontrol start) the following day's mail report is empty and table has no data from dates running 5.0.11 that were not the upgrade days.

What changed (other than the post about kill 9 vs. kill TERM, which I tried) in the stop and start sequences?

dwmtractor,

Did/do you do an offline backup in your environment/testing?

[dwmtractor]

Originally Posted by lzmarine

Running: /opt/zimbra/bin/logmysql zimbra_logger
> select * from mta;

This produces no output for dates after the upgrade from 5.0.10 to 5.0.11 and an offline backup. Re-install thought to fix this as the next days report had data again; but, after the nightly backup (zmcontrol stop; zmcontrol start) the following day's mail report is empty and table has no data from dates running 5.0.11 that were not the upgrade days.

What changed (other than the post about kill 9 vs. kill TERM, which I tried) in the stop and start sequences?

dwmtractor,

Did/do you do an offline backup in your environment/testing?

Yes, I run a variant of this script from cron nightly. I have not had any issues with the logs failing to show data; the problem with kill referenced above has to do with the logger process failing to stop, not failing to start.

Once you have done the stop & start as you describe, can you verify if zmlogger, and the Linux process syslogd, are running or no? The only time I have seen a total lack of data was when syslogd was stopped. . .which zmcontrol does not affect.

[tonster]

Originally Posted by lzmarine

Running: /opt/zimbra/bin/logmysql zimbra_logger
> select * from mta;

This produces no output for dates after the upgrade from 5.0.10 to 5.0.11 and an offline backup. Re-install thought to fix this as the next days report had data again; but, after the nightly backup (zmcontrol stop; zmcontrol start) the following day's mail report is empty and table has no data from dates running 5.0.11 that were not the upgrade days.

What changed (other than the post about kill 9 vs. kill TERM, which I tried) in the stop and start sequences?

dwmtractor,

Did/do you do an offline backup in your environment/testing?

What happens when you run:

/opt/zimbra/libexec/zmlogprocess

as user zimbra? Do you get any errors or output? What if you try running zmloggerctl start?

[lzmarine]

mail:~$ sudo -u zimbra /opt/zimbra/libexec/zmlogprocess
Pruning service/server entries older then 2008-12-17 06:38:44
Processing mta entries from 13249293
Found 0 messages
Deleting processed postfix logs from raw_logs...Done
Processing clamd entries from 5077072
Found 0 entries
Deleting processed clamd logs from raw_logs...Done
Processing amavis entries from 13249540
Found 0 entries
Deleting processed amavis logs from raw_logs...Done
Processing sendmail entries from 0
Found 0 entries
Deleting processed sendmail logs from raw_logs...Done
Processing zimbramon entries from 13249544
Found 0 entries
Deleting processed zimbramon logs from raw_logs...Done
--- snip ---
aggregating [hour|day] [mta|amavis|disk] from -date- to -date-
--- snip ---
Retaining 31 days worth of raw logs based on zimbraLogRawLifetime config.
Retaining 730 days worth of summary logs based on zimbraLogSummaryLifetime config.
Pruning raw mta logs from 2008-11-16 00:00:00
Pruning raw amavis logs from 2008-11-16 00:00:00
Pruning raw disk logs from 2008-11-16 00:00:00
Pruning summary mta logs from 2006-12-18 00:00:00
Pruning summary amavis logs from 2006-12-18 00:00:00
Pruning summary disk logs from 2006-12-18 00:00:00
Pruning other raw logs from 2008-11-16 00:00:00

Also:
mail:~$ sudo -u zimbra /opt/zimbra/bin/zmloggerctl start
mysqld_safe already running with pid 19818
Starting logswatch...done.
mail:~$ sudo -u zimbra /opt/zimbra/bin/zmcontrol status
Host mail.domain.com
antispam Running
antivirus Running
ldap Running
logger Stopped
zmlogswatchctl is not running
mailbox Running
mta Running
snmp Stopped
zmswatch is not running.
spell Running
stats Running
mail:~$