[SOLVED] Retrieving mail stuck on new server after failed server migration

[flyerguybham]

I was trying to upgrade our Zimbra server last night (not a Zimbra upgrade, a hardware upgrade; I was doing a lateral move from one 4.5.11 to another). It looked like all went well when I fired up the new zimbra server, every component appeared to start based on the zmcontrol start messages. All of the mail that had queued up at our border mail server soon noticed that our mail server was back online, and the new server's mta quickly received about two hours worth of deferred mail. Meanwhile I opened a web browser and the first sign of trouble was that nothing appeared to be listening on ports 80 or 443.

While investigating further I noticed Zimbra was not delivering the mail - the mta was receiving, but lmtp was giving connection refused. zmcontrol status showed that tomcat was not running. (I am still working on why that is the case, but that's actually not what I need help with at the moment.) After an hour of working on a fix I needed to get mail working again, so I shut down the new server and put the old server back in service.

The problem now is that two hours worth of our mail is in the deferred queue on the new server, and I am trying to figure out a plan to rescue it. Does anyone know how I might approach this? Thanks.

[phoenix]

I'd suggest you look at the log files and see what errors are being logged. Assuming this server has the same IP address and hostname I would guess this is possibly to be a DNS problem. What does a 'zmcontrol status' show?

[flyerguybham]

Thanks for the reply. Before I answer your questions, let me address the more urgent issue, which was rescuing the mail from the new server's deferred queue.

I did this:

(on old server)
# zmcontrol stop

(on new server)
# rsync -avr /opt/zimbra/postfix/spool/deferred/ root@oldserver:/opt/zimbra/postfix/spool/deferred/
# rsync -avr /opt/zimbra/postfix/spool/defer/ root@oldserver:/opt/zimbra/postfix/spool/defer/

(on old server)
# zmcontrol start

and then I logged into the old server's web admin and flushed its deferred queue. This appears to have worked so the urgent part of my problem may be resolved.

Now to answer your questions. Servers have same name and IP. DNS config was not changed. zmcontrol status on the new server shows:

Host zimbra.cis.uab.edu
        antispam                Running
        antivirus               Running
        ldap                    Running
        logger                  Running
        mailbox                 Stopped
                tomcat is not running
        mta                     Running
        snmp                    Running
        spell                   Running
        stats                   Running

which is what I expected given the symptoms (mta delivery but no local mail delivery, nothing listening on the web ports, etc...).

Thanks to your pointer to the wiki docs about log files, I found /opt/zimbra/tomcat/logs/catalina.out, which led me to solving my two remaining problems:

Problem 1.

java.io.IOException: Keystore was tampered with, or password was incorrect

Following the directions here: Network Edition: Moving from 32-bit to 64-bit Server - Zimbra :: Wiki
I copied my keystore over from my old server.

Unfortunately, those directions didn't include the steps to update Zimbra with the keystore password. I found those at this bug report: Bug 21174 - create zmkeystorepasswd to simplify changing keystore password

Problem 2.

Zimbra server process is running as root, changing to user=zimbra uid=10000 gid=551
Zimbra server process, after change, is running with uid=10000 euid=10000 gid=551 egid=551
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /opt/zimbra/log/mailbox.log (Permission denied)

Those uid/gid settings were not the same as the old server. I had already set the uid/gid on the new server in /etc/passwd to match the old so that my files would have proper ownership when I rsync'ed them over, but did not realize Zimbra also stores these values in its internal config. I made them so with:

# zmlocalconfig -e zimbra_uid=1000
# zmlocalconfig -e zimbra_gid=1000

Looks like everything is working now and that I owe Zimbra some documentation patches for their 32-bit to 64-bit migration wiki page.

Thanks for the pointer to the log file wiki page, that's what started this solution!

[phoenix]

Glad it's all working.