Hello, all. We are making some progress toward understanding what has happened but still have not fixed the problem. It does not appear that this problem is related to our manually running the cron jobs but is rather a database issue. We have been able to reproduce the issue.
The first time was after we noticed we did not have a clean shutdown of the system. The second, we crashed the system (we are still pre-production and doing some experimentation). After the crash, we saw the same symptoms: It was as if mail had rolled back to a previous point, newly created domains and users were missing, existing users had deleted mail in their inboxes and missing new mails.
When we check the message store, it appears the messages are there but they are not showing up in the database. We also are seeing error messages in the database, e.g., this is when we had our first problems:
Code:
090507 22:27:02 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
090507 22:27:03 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
090507 22:27:05 InnoDB: Started; log sequence number 0 3354455
090507 22:27:05 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
Followed shortly thereafter with:
Code:
090507 22:52:06 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
Version: '5.0.67-log' socket: '/opt/zimbra/db/mysql.sock' port: 7306 Source distribution
090507 22:59:50090507 22:59:50 [ERROR] Cannot find table mboxgroup16/appointment from the internal data dictionary
of InnoDB though the .frm file for the table exists. Maybe you
have deleted and recreated InnoDB data files but have forgotten
to delete the corresponding .frm files of InnoDB tables, or you
have moved .frm files to another database?
See http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
how you can resolve the problem.
090507 22:59:50090507 22:59:50 [ERROR] Cannot find table mboxgroup16/appointment from the internal data dictionary
of InnoDB though the .frm file for the table exists. Maybe you
have deleted and recreated InnoDB data files but have forgotten
to delete the corresponding .frm files of InnoDB tables, or you
have moved .frm files to another database?
See http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
how you can resolve the problem.
We see similar messages tonight. Running zmdbintegrityreport -v gave similar messages and -r did not repair them.
We next followed the database repair procedures described in
Mysql Crash Recovery - Zimbra :: Wiki. Now, zmdbintegrityreport -v comes up clean but we still have the same problem.
How does one get the messages from the message stored back into the database and recovery the other information? I'm suspecting but hoping that I will not hear the only way is restoration from backup and losing anything since the last backup. Thanks - John