db crash recovery?

[tiger2000]

Dear all,

one 5.0.13 ZCS system encounters a power failure reboot, after that, the ZCS will always do "db crash recovery" when services start up, something like this in mailbox.log:

2010-07-14 16:11:26,237 INFO  [main] [] RedoPlayer - Deferring crash recovery to after startup: txn 1274070006.40888 [IndexItem] ver=1.23, tstamp=1274072796408, mailbox=35278, id=4257385, type=5

2010-07-14 16:11:26,237 INFO  [main] [] RedoPlayer - Deferring crash recovery to after startup: txn 1274070006.40889 [IndexItem] ver=1.23, tstamp=1274072796408, mailbox=35278, id=4257397, type=5

2010-07-14 16:11:26,237 INFO  [main] [] RedoPlayer - Deferring crash recovery to after startup: txn 1274070006.40890 [IndexItem] ver=1.23, tstamp=1274072796408, mailbox=35278, id=4257378, type=5

How can i do a complete DB recovery so that it will not start it again when every reboot??

and the more serious problem is that the system now is suffering poor performance, the obvious messages in mailbox.log are something like:

2010-07-14 17:02:56,909 WARN  [Pop3Server-39] [name=xxxx@yyy.zzz;ip=209.85.216.6;] dbconn - Connection pool is 75% utilized (100 connections out of a maximum of 100 in use).  Turn on debug logging for zimbra.dbconn to see stack traces of connections not returned to the pool.

From the OS "top" utility, it looks to me the ZCS is performaing some heavy disk I/O activities, which caused iowait% pretty high.

any advice on this?

Thanks.

[phoenix]

I'd suggest the first thing you do is make a complete backup of your /opt/zimbra directory then do a (non-destructive) file system check on your HDs on this server. When you've done that and if it's all OK try some of the tips here to determine where the problem is and some fixes for it.

[tiger2000]

thanks for the reply.

i've followed that wiki article to dump/repopulate data back to ZCS, however, it still gives me the same log message when ZCS starts up:

INFO [main] [] RedoPlayer - Deferring crash recovery to after startup: txn xxxxxxx

i also notice that the system is still doing some "PostStartupCrashRecovery" action, something like :

2010-07-15 19:00:32,371 INFO [PostStartupCrashRecovery] [] RedoLogManager - REDOING: txn 1274070006.1134440 [IndexItem] ver=1.23, tstamp=1274251119993, mailbox=25871, id=3980, type=5

any idea what it means? and what should we do for now?

any info will be highly appreciated.

[tiger2000]

i'm wondering if my understanding is correct? could someone please help ?

when ZCS's starting, it's saying the RedoPlayer will defer crash recovery after startup (it also gives a mailbox number: 27213 and id number: 5622).

2010-07-15 22:02:16,183 INFO  [main] [] RedoPlayer - Deferring crash recovery to after startup: txn 1274070006.3911497 [IndexItem] ver=1.23, tstamp=1275449079206, mailbox=27213, id=5622, type=5

so that we can see there's lots of redoing message like this:

2010-07-16 11:16:57,285 INFO  [PostStartupCrashRecovery] [] RedoLogManager - REDOING: txn 1274070006.3991602 [IndexItem] ver=1.23, tstamp=1275483082460, mailbox=22413, id=6220, type=5

after RedoLogManager completes all jobs, it will stop to pop up these kind of message. and ZCS will be back to normal.

Is this correct?

TIA

[tiger2000]

just an update that those messages (RedoLogManager - REDOING) were gone after ~ 2 days recovery. it looks like the system has finished the whole recovery , however, we didn't restart ZCS service after then, thus, we're not sure if it's still same in next restart.

Thanks.