Loss of power caused zimbra to get angry

[zachkarpinski]

I am running CentOS 4, Zimbra 4.5.3 -> upgraded to 4.5.5. After the power loss fsck reported quite a few filesystem errors (I had to fix in single user mode).

I could not upgrade to 4.5.5 very easily, I ended up changing my LDAP password in addition to manually starting slapd during the upgrade process so it would load my settings properly. In the end, I was able to complete the upgrade without any errors on the screen.

After the upgrade completed, things worked a bit more but I'm still having problems. Most importantly all of my incoming mail is being deferred.

My box resolves properly (editted). I have an internal DNS server that resolves the zimbra machine to a local IP but the rest of the world sees the IP that is in front of the firewall.
[root@email log]# dig email.hostname.com
;; ANSWER SECTION:
email.hostname.com. 86400 IN A 192.168.1.33

---

Here is the error I am seeing two different errors depending on what my hostname is:
While hostname is "email":
postfix/qmgr[785]: 62C3613B59C: to=, relay=none, delay=0, status=deferred (delivery temporarily suspended: lost connection with email.hostname.com[192.168.1.33] while sending end of data -- message may be sent more than once)

While hostname is "email.hostname.com"
postfix/qmgr[6464]: 4C91C13B5C9: to=, relay=none, delay=0, status=deferred (delivery temporarily suspended: connect to email.hostname.com[192.168.1.33]: Connection refused)

--

After a restart I caught this in my zimbra.log

postfix/postqueue[32708]: fatal: Queue report unavailable - mail system is down

However zmcontrol status shows:
[zimbra@email bin]$ zmcontrol status
Host email.hostname.com
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running

[zachkarpinski]

Any chance of someone with insight on this? No matter what I try I keep getting the deferred errors. Postfix insists it cannot connect to 192.168.1.33 even though I can telnet to 25 and 7025 locally and from remote.

I'm vexed.

[jholder]

Have you checked to make sure your hosts file is correct?

Also, try to do an installation with ./install.sh -s and run a ./zmfixperms to see perhaps if theres a permissions issue.

[zachkarpinski]

Thank you for the reply. I am currently running the install -s command. I will let you know what it says. I have already run the fixperms script as root and that had no effect.

My hosts file looks like:
[zimbra@email ~]$ more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail
127.0.0.1 localhost.localdomain localhost
192.168.1.33 email.domainname.com email

[zachkarpinski]

The install log for install -s reported only the following warnings/errors:

mkdir: cannot create directory `/opt/zimbra/ssl': File exists
mkdir: cannot create directory `/opt/zimbra/.ssh': File exists
---
umount2: Invalid argument
umount: /opt/zimbra/amavisd/tmp: not mounted
chown: changing ownership of `/opt/zimbra/httpd': No such file or directory
chown: changing ownership of `/opt/zimbra/mysql': No such file or directory
chown: changing ownership of `/opt/zimbra/snmp': No such file or directory

Afterwards I re-ran zmfixperms as root and then ran zmcontrol start as user zimbra.

Tomcat appears to be crashing now. I did not run the zmsetup script. Should I?

[jholder]

Wow, I wonder what's goin on here. I assume you've made a backup. If not, make one now, and put it in a safe place.

Next, post your /opt/zimbra/tomcat/logs/catalina.out
That will tell us why Tomcat is foobaring.

[zachkarpinski]

After digging a bit deeper, it seems that mysql may not be starting properly due to InnoDB table corruption. At this point I am not sure if there is a seperate issue with tomcat or if that is a side effect of the mysql issue.

I will follow up once the mysql issues are resolved (if possible)

[jholder]

You may need to attempt to repair. If that doesn't work, then dump all the data to disk, run zmmyinit, then reimport it.

**WARNING- Do not run zmmyinit UNLESS you have backup**

[zachkarpinski]

By default doesn't zimbra use InnoDB?

Either way, all of my databases are InnoDB. As you know, InnoDB does not support repairing.

I will attempt a dump of whatever mboxgroups i can and the rest will just be lost i guess :|

Other than the mboxgroups and the zimbra database, do i need any others?

[zachkarpinski]

I've made a snapshot of my VM from before the zmmyinit. I am not sure if I will be able to restore any of the failed data, but rebuilding mysql has gotten zimbra back on its feet.

I am not sure if the current zimbra on a virtual machine guide stresses this, but it is definitely bad to have this kind of server setup as an Independent Persistent disk. If the power goes out. You're done for unless you have very recent backups.