We are still struggling with this issue but have narrowed it down quite a bit. We noticed that our data loss is always the data between 4:02AM the morning after zimbra was last restarted and the time of the next zimbra service restart. This happens whether we restart the server, restart the zimbra service, or simply do a zmstorectl restart.
We also noticed that all logging stops at this same 4:02AM point in time. No entries in zimbra.log, mailbox.log, logger_myslow.log, etc.
This is also the time the logrotate daily cron job kicks in. We did notice a problem in /etc/logrotate.d/zimbra. The postrotate routine for zimbra.log tries to restart syslogd but CentOS 5.3 (our platform) uses rsyslogd. Once we edited that call, zimbra.log was able to continually log past 4:02AM but it still did not fix the problem.
We removed the zimbra logrotate file from /etc/logrotate.d/ last night but were able to reproduce the data loss this morning so it does not appear to be something within the zimbra file. We are also told by zimbra tech support that data missing from the logs would not provoke a data rollback / loss / corruption upon service restart so we are assuming this log failure is a parallel symptom rather than a cause.
We will begin removing files from logrotate.d in an attempt to find what is the mystic coincidence between this failure and the cron.daily jobs. Any suggestions about what might be happening? Thanks - John |