Service unavilable zclient/io error after power outage

[intratek]

Hello.

I am running zimbra version 5.02 on a fedora 7 server since past 2 and half years. All things were going well until a power outage that happened 3 days before. When I started Zimbra, after getting login page and producing username & password credentials I get the above message like service unavilable zclient/io error.

Since past 3 days I am struggling to restore my services. Here are a few steps I took and I further request you to guide me accordingly.

It looks clearly an OS fault as it gives a input output error. So I ran e2fsck with all possible options but it could not be recovered. I got the same message after login.

Henceforth I planned a migration to CentOS 5 and followed exact steps as per Moving ZCS to Another Server » Zimbra :: Blog.
But there I found a problem as service failutre.

Here is the mailbox.log :--

2010-06-27 09:24:30,137 INFO [main] [] soap - Adding service MailService to SoapServlet
2010-06-27 09:24:30,582 INFO [main] [] soap - Adding service IMService to SoapServlet
2010-06-27 09:24:30,632 INFO [main] [] misc - version=5.0.2_GA_1975.F7 release=20080130225918 builddate=20080130-2300 buildhost=build20.re
.zimbra.com
2010-06-27 09:24:30,673 INFO [main] [] misc - LANG environment is set to: C
2010-06-27 09:24:30,821 INFO [main] [] system - Setting mysql connector property: maxActive=10
2010-06-27 09:24:30,822 INFO [main] [] system - Setting mysql connector property: maxActive=10
2010-06-27 09:24:32,277 FATAL [main] [] system - Config initialization failed
com.zimbra.common.service.ServiceException: system failure: getting all config entries
Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:183)
at com.zimbra.cs.db.DbConfig.getAll(DbConfig.java:163 )
at com.zimbra.cs.util.Config.init(Config.java:63)
at com.zimbra.cs.util.Config.initConfig(Config.java:8 9)
at com.zimbra.cs.util.Config.getString(Config.java:10 1)
at com.zimbra.cs.db.Versions.checkDBVersion(Versions. java:77)
at com.zimbra.cs.db.Versions.checkVersions(Versions.j ava:73)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:88)
at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 114)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:241)
at org.mortbay.jetty.servlet.ServletHolder.initServle t(ServletHolder.java:433)
at org.mortbay.jetty.servlet.ServletHolder.doStart(Se rvletHolder.java:256)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.jetty.servlet.ServletHandler.initializ e(ServletHandler.java:612)
at org.mortbay.jetty.servlet.Context.startContext(Con text.java:139)
at org.mortbay.jetty.webapp.WebAppContext.startContex t(WebAppContext.java:1222)
at org.mortbay.jetty.handler.ContextHandler.doStart(C ontextHandler.java:501)
at org.mortbay.jetty.webapp.WebAppContext.doStart(Web AppContext.java:449)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:147)
at org.mortbay.jetty.handler.ContextHandlerCollection .doStart(ContextHandlerCollection.java:161)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:147)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:117)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:117)
at org.mortbay.jetty.Server.doStart(Server.java:217)
at org.mortbay.setuid.SetUIDServer.doStart(SetUIDServ er.java:100)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:979)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:183)
at org.mortbay.start.Main.start(Main.java:497)
at org.mortbay.start.Main.main(Main.java:115)
Caused by: java.sql.SQLException: Incorrect information in file: './zimbra/config.frm'

Here is the output of zmmailboxd.out :--

Caused by: com.zimbra.common.soap.SoapParseException: cannot determine soap protocol in reply: <html><head><meta content="text/html; charse
t=ISO-8859-1" http-equiv="Content-Type"/><title>Error 503 SERVICE_UNAVAILABLE</title></head><body><h2>HTTP ERROR: 503</h2><pre>SERVICE_UNAV
AILABLE</pre><p>RequestURI=/service/admin/soap/</p><p><i><small><a href="http://jetty.mortbay.org">Powered by jetty://</a></small></i></p><
/body></html>
at com.zimbra.common.soap.SoapTransport.extractBodyEl ement(SoapTransport.java:226)
at com.zimbra.common.soap.SoapTransport.parseSoapResp onse(SoapTransport.java:219)
at com.zimbra.common.soap.SoapHttpTransport.invoke(So apHttpTransport.java:251)
at com.zimbra.common.soap.SoapTransport.invoke(SoapTr ansport.java:295)
at com.zimbra.common.soap.SoapTransport.invoke(SoapTr ansport.java:259)
at com.zimbra.cs.account.soap.SoapProvisioning.invoke (SoapProvisioning.java:171)
... 48 more


Also my mailbox server was not running and there were problems in zmmailboxdctl. This leads me to (probably) a MySQL related issue.
The I followed the Mysql crash recovery related article in the wiki. While I was going step by step I got the message like something (incorrect information in the file mboxgroup* appointments.frm) * represents all those databses from mboxgroup1 to mboxgroup99; while creating mysqldump of all the databases.

here is extract of a log file from server.err

100412 10:26:53 mysqld started
100412 10:26:57 InnoDB: Started; log sequence number 1 1432481544
100412 10:26:57 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
Version: '5.0.45-log' socket: '/opt/zimbra/db/mysql.sock' port: 7306 Source distribution
InnoDB: Error: tried to read 16384 bytes at offset 0 3375104.
InnoDB: Was only able to read -1.
100412 10:27:18 InnoDB: Operating system error number 5 in a file operation.
InnoDB: Error number 5 means 'Input/output error'.
InnoDB: Some operating system error numbers are described at
InnoDB: MySQL :: MySQL 5.0 Reference Manual :: 13.2.12.2 Operating System Error Codes
InnoDB: File operation call: 'read'.
InnoDB: Cannot continue operation.

Number of processes running now: 0
100412 10:27:19 mysqld restarted
InnoDB: Warning: we did not need to do crash recovery, but log scan
InnoDB: progressed past the checkpoint lsn 1 1432481544 up to lsn 1 1432481554
100412 10:27:19 InnoDB: Started; log sequence number 1 1432481544
100412 10:27:19 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
Version: '5.0.45-log' socket: '/opt/zimbra/db/mysql.sock' port: 7306 Source distribution
InnoDB: Error: tried to read 16384 bytes at offset 0 3375104.
InnoDB: Was only able to read 8192.
100412 10:27:23 InnoDB: Operating system error number 0 in a file operation.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: MySQL :: MySQL 5.0 Reference Manual :: 13.2.12.2 Operating System Error Codes
InnoDB: File operation call: 'read'.
InnoDB: Cannot continue operation.

100627 09:45:29 mysqld started
InnoDB: Error: auto-extending data file ./ibdata1 is of a different size
InnoDB: 192 pages (rounded down to MB) than specified in the .cnf file:
InnoDB: initial 640 pages, max 0 (relevant if non-zero) pages!
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
100627 9:45:29 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
Version: '5.0.45-log' socket: '/opt/zimbra/db/mysql.sock' port: 7306 Source distribution

This is also couppled with a large number of lines containing

100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/appointment.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/appointment.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/imap_folder.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/imap_folder.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/imap_message.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/imap_message.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/mail_item.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/mail_item.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/open_conversation.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/open_conversation.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/pop3_message.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/pop3_message.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/revision.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/revision.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/tombstone.frm'
100627 9:45:35 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './mboxgroup99/tombstone.frm'
for all mboxgroup databases.

The file size of the file db/data/ibdata1 on the main server as well as on a new server way differs. When I try to copy the ibdata1 file to another location it gives an inout output error

I have not done nothing other than file system check on the main server.

All these experiments I have done on my CentOS server with the same version ie 5.02_GA
I have my Antivirus (Clamav) OFF and antivirus server is NOT running.