4.5.10 logger database errors

[jescombe]

I'd not had stats working for a while since upgrading to 4.5.7, but was hoping that 4.5.10 would get them going again...

I was still getting empty graphs, so checked for logger errors, and am seeing the following lines in my /opt/zimbra/logger/db/data/<host>.err file

[ERROR] /opt/zimbra/logger/mysql/libexec/mysqld: Table './zimbra_logger/raw_logs' is marked as crashed and last (automatic?) repair failed

I've tried repairing the database with the following command, but this hasn't made any difference?

$ /opt/zimbra/logger/mysql/bin/mysqlcheck --socket=/opt/zimbra/logger/db/mysql.sock -p -uroot zimbra_logger --repair
Enter password:
zimbra_logger.amavis OK
zimbra_logger.amavis_aggregate OK
zimbra_logger.config OK
zimbra_logger.disk_aggregate OK
zimbra_logger.disk_status OK
zimbra_logger.mta OK
zimbra_logger.mta_aggregate OK
zimbra_logger.processing_history OK
zimbra_logger.raw_logs Table is already up to date
zimbra_logger.service_status OK

Obviously the output for the raw_logs table looks different, but I've no idea where to go from here.. Any suggestions how to fix this would be welcome, thanks!

Jon.

Edit - I looked back further in the logs and this has been happening since (from memory) the time of the previous upgrade - attaching the last good entries before the errors started...

070927 00:32:18 mysqld started
070927 0:32:19 InnoDB: Started; log sequence number 0 45164
070927 0:32:19 [Note] /opt/zimbra/logger/mysql/libexec/mysqld: ready for connections.
Version: '5.0.45-log' socket: '/opt/zimbra/logger/db/mysql.sock' port: 7307 Source distribution
070927 0:44:01 [Note] /opt/zimbra/logger/mysql/libexec/mysqld: Normal shutdown

070927 0:44:01 [Warning] Warning: Optimize table got errno 120 on zimbra_logger.raw_logs, retrying
070927 0:44:04 InnoDB: Starting shutdown...
070927 0:44:06 InnoDB: Shutdown completed; log sequence number 0 45164
070927 0:44:06 [Note] /opt/zimbra/logger/mysql/libexec/mysqld: Shutdown complete

070927 00:44:06 mysqld ended

070927 00:44:53 mysqld started
070927 0:44:56 InnoDB: Started; log sequence number 0 45164
070927 0:44:56 [Note] /opt/zimbra/logger/mysql/libexec/mysqld: ready for connections.
Version: '5.0.45-log' socket: '/opt/zimbra/logger/db/mysql.sock' port: 7307 Source distribution
070927 0:44:59 [ERROR] /opt/zimbra/logger/mysql/libexec/mysqld: Table './zimbra_logger/raw_logs' is marked as crashed and last (automatic?) repair failed

[jholder]

I know you tried this, but try it verbatim:
Repair Logger Data Corruption - Zimbra :: Wiki

If that doesn't work, try the mysql data corruption article.

If you don't really care about your logger data, you can re-init the logger:

stop logger db  (mysql processes) 
rm -rf /opt/zimbra/logger/db/data 
/opt/zimbra/libexec/zmloggerinit

[jescombe]

Ah yes, rebuilding the index with "REPAIR TABLE raw_logs USE_FRM;" appears to have done the trick. I searched the forums, but should have checked the wiki too, so apologies for that - and thanks for the quick answer!

[swu]

I just upgraded to 4.5.10 over the weekend and notice that all of my stats have stopped.

I tried some of the steps above but it's still no-go

mysql> REPAIR TABLE mta;
+-------------------+--------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+-------------------+--------+----------+----------+
| zimbra_logger.mta | repair | status | OK |
+-------------------+--------+----------+----------+
1 row in set (7.38 sec)

080116 15:50:04 [ERROR] /opt/zimbra/logger/mysql/libexec/mysqld: Table './zimbra_logger/mta' is marked as crashed and last (automatic?) repair failed

I'd like to get the stats/history going again, what do you recommend?

Thx!

[PeterH]

After upgrade last week from 4.5.10 to 5.0.1 the backup filled my partition, disk and thus killed ldap and stats..
Ldap repaired immediately and stats repaired 2day, both thnx to the excellent info and support in the forums!

[math]

When there are logger errors, Im seeing 285 errors per second in the logs (based on the # of lines per second of timestamp in the log). I just deleted a 5 gig log (and deleted it the other week too).

Is there any way to slow this reporting down?

Furthermore, is there any way internally to send the admin or otherwise an email when this error condition is in effect?

[phoenix]

Originally Posted by math

When there are logger errors, Im seeing 285 errors per second in the logs (based on the # of lines per second of timestamp in the log). I just deleted a 5 gig log (and deleted it the other week too).

Is there any way to slow this reporting down?

Furthermore, is there any way internally to send the admin or otherwise an email when this error condition is in effect?

Which version of Zimbra? Have you tried some of the troubleshooting described in this thread or the wiki article, if there's corruption you might want to try these steps to fix it?

[math]

Originally Posted by phoenix

Which version of Zimbra? Have you tried some of the troubleshooting described in this thread or the wiki article, if there's corruption you might want to try these steps to fix it?

Yes that's not the problem. The problem is two fold.

1) mysql sucks. when a machine crashes and mysql is running, it is my experience that mysql suffers 100x the chance of a table being corrupted over postgresql. it seems that even with a crontab flushing tables every minute in mysql, it only helps a little. Mysql is just really horrid for corruption on crash. it's really inferior to postgres in many ways.

2) so there's some corruption, if i dont get around to fixing it (say the box restarts by itself and zimbra comes up automagically), then the logfile logs 250 lines/s or more til i get to it - gigs of logs later. I want the warnings to be

(a) less frequent (perhaps one warning line per second would be ok)
(b) warn administrator or the like via an email msg in zimbra itself.

I've corrected the corruption in the db many times on different machines.
(I have a slightly unstable box in my rack, zimbra's on it, and this happens every time it crashes. While the box shouldnt crash, sometimes crashes happen, but it shouldnt take the whole db with it every time! Yes im running ext3, no its not bad ram (this happens to me on any machine that mysql crashes on...)).