zmrestoreoffline failure - ServiceException: system failure

[indymail]

I'm trying to move a (single-server) ZCS 5.08 deployment to new hardware and I'm having trouble getting importing the mailbox data to the new server.

I've followed the steps outlined in the "Disaster Recovery Changing Servers" section of the Network Edition 5.0 Administrators Guide (pages 154-157) but when I get to step 7 of "Restoring to the new server" (page 156) the zmrestoreoffline process throws an exception and stops. The exception is: com.zimbra.common.service.ServiceException: system failure: unable to lookup server by name: mail.indyimaging.com message: [LDAP: error code 49 - Invalid Credentials]

I started over to make sure that I didn't just make a mistake somewhere and got exactly the same error.

Here are the exact steps I took:
[on the old server]
1) did a full backup of the old server to a spare drive
[on the new server]
2) installed the same ZCS version, matching the spam, not-spam, global doc account, etc. settings.
3) ran zmcontrol stop
4) mounted the spare drive with the backup data at /opt/zimbra/backup
5) deleted /opt/zimbra/db/data/* and reinitialized it with zmmyinit
6) ran zmrestoreldap -lb [the backup label]
7) started zmconvertctl
8) ran zmrestoreoffline -sys -a all -c -br

zmrestoreoffline returned immediately and produced the following output in mailbox.log:

2008-08-03 17:19:35,299 INFO  [main] [] misc - version=5.0.8_GA_2462.RHEL5_64 release=20080709155301 builddate=20080709-1554 buildhost=build11
2008-08-03 17:19:35,310 INFO  [main] [] misc - LANG environment is set to: en_US.UTF-8
2008-08-03 17:19:35,310 INFO  [main] [] misc - System property java.home=/opt/zimbra/jdk1.5.0_15/jre
2008-08-03 17:19:35,310 INFO  [main] [] misc - System property java.runtime.version=1.5.0_15-b04
2008-08-03 17:19:35,310 INFO  [main] [] misc - System property java.version=1.5.0_15
2008-08-03 17:19:35,310 INFO  [main] [] misc - System property java.vm.info=mixed mode
2008-08-03 17:19:35,310 INFO  [main] [] misc - System property java.vm.name=Java HotSpot(TM) 64-Bit Server VM
2008-08-03 17:19:35,310 INFO  [main] [] misc - System property java.vm.version=1.5.0_15-b04
2008-08-03 17:19:35,310 INFO  [main] [] misc - System property os.arch=amd64
2008-08-03 17:19:35,311 INFO  [main] [] misc - System property os.name=Linux
2008-08-03 17:19:35,311 INFO  [main] [] misc - System property os.version=2.6.18-92.1.6.el5
2008-08-03 17:19:35,311 INFO  [main] [] misc - System property sun.arch.data.model=64
2008-08-03 17:19:35,311 INFO  [main] [] misc - System property sun.cpu.endian=little
2008-08-03 17:19:35,311 INFO  [main] [] misc - System property sun.cpu.isalist=
2008-08-03 17:19:35,311 INFO  [main] [] misc - System property sun.os.patch.level=unknown
2008-08-03 17:19:35,353 INFO  [main] [] system - Setting mysql connector property: maxActive=100
2008-08-03 17:19:35,366 INFO  [main] [] system - Setting mysql connector property: maxActive=100
2008-08-03 17:19:36,462 FATAL [main] [] system - Config initialization failed
com.zimbra.common.service.ServiceException: system failure: unable to lookup server by name: mail.indyimaging.com message: [LDAP: error code 49 - Invalid Credentials]
ExceptionId:main:1217798376459:5a9b158b360d8c45
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253)
        at com.zimbra.cs.account.ldap.LdapProvisioning.getServerByName(LdapProvisioning.java:2231)
        at com.zimbra.cs.account.ldap.LdapProvisioning.getServerByName(LdapProvisioning.java:2209)
        at com.zimbra.cs.account.ldap.LdapProvisioning.getLocalServer(LdapProvisioning.java:2593)
        at com.zimbra.cs.util.Config.init(Config.java:80)
        at com.zimbra.cs.util.Config.initConfig(Config.java:91)
        at com.zimbra.cs.util.Config.getString(Config.java:103)
        at com.zimbra.cs.db.Versions.checkDBVersion(Versions.java:77)
        at com.zimbra.cs.db.Versions.checkVersions(Versions.java:73)
        at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:156)
        at com.zimbra.cs.util.Zimbra.startupCLI(Zimbra.java:133)
        at com.zimbra.cs.backup.RestoreOfflineUtil.setup(RestoreOfflineUtil.java:101)
        at com.zimbra.cs.backup.RestoreOfflineUtil.main(RestoreOfflineUtil.java:46)
Caused by: javax.naming.AuthenticationException: [LDAP: error code 49 - Invalid Credentials]; remaining name 'cn=mail.indyimaging.com,cn=servers,cn=zimbra'
        at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:2985)
        at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2931)
        at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2732)
        at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2646)
        at com.sun.jndi.ldap.LdapCtx.ensureOpen(LdapCtx.java:2546)
        at com.sun.jndi.ldap.LdapCtx.ensureOpen(LdapCtx.java:2520)
        at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1901)
        at com.sun.jndi.ldap.LdapCtx.doSearchOnce(LdapCtx.java:1893)
        at com.sun.jndi.ldap.LdapCtx.c_getAttributes(LdapCtx.java:1286)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:213)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:121)
        at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:133)
        at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:128)
        at com.zimbra.cs.account.ldap.ZimbraLdapContext.getAttributes(ZimbraLdapContext.java:524)
        at com.zimbra.cs.account.ldap.LdapProvisioning.getServerByName(LdapProvisioning.java:2222)
        ... 11 more

Can anyone suggest how to troubleshoot/fix this?

Thanks,
Tim

[gmsmith]

Kinda looks like your ldap password is wrong.

You may want to look at this: Moving ZCS to Another Server » Zimbra :: Blog

Much easier for migrating server hardware.

[indymail]

Originally Posted by gmsmith

You may want to look at this: Moving ZCS to Another Server » Zimbra :: Blog

Much easier for migrating server hardware.

Thanks for the tip. I saw that article but I'm migrating from ZCS 5.0.8 RHEL4 to ZCS 5.0.8 RHEL5_64 so I can't just copy the /opt/zimbra folder. The binaries are different.

Originally Posted by gmsmith

Kinda looks like your ldap password is wrong.

I also thought that the LDAP password might be wrong but changing it didn't fix the problem. What puzzles me is that everything was working correctly right before I ran the zmrestoreldap command (e.g. I could log into the zimbra admin console and use the CLI tools). Why would importing LDAP data break the method ZCS uses to connect to slapd? The way I understand it zmrestoreldap only changes the LDAP directory contents, not the slapd configuration.

I'm at a loss as to how to troubleshoot this. Is there an easy way to turn on slapd debugging? I tried changing the way the slapd service starts but I couldn't get it to dump any useful information.

[gkra]

Bump.

I've been having similar difficulties with following the instructions on the Disaster Recovery page on the wiki, and following the "move to another server" blog entry isn't all that useful when all you have is the contents of /opt/zimbra/backup, which should be all you need.

[tecnalb]

I have been working on this for two weeks and have it down pretty well. I started this as a test before migrating to a new server and found it didn't work well. After much practice it does work with some additional steps. I followed the wiki for Disaster recovery.

A couple of things:

when moving to a new server simple add a hostfile entry on the new server for the new IP. The install will bark that the MX is wrong, but it will still work. This way you can actually test while your existing server is up and not deal with any DNS changes. It works.

Also, the wiki mentioned turing off auto start and backups. If you do that, some of the zimlets (for backup and restore) won't initialize.

Also, you will notice many mentions by people that the logging doesnt work during the zmrestoreoffline. If, during the initial install you DO start and then stop the server, it does fix this problem and the mailbox.log will be created properly. What you will do is install a fresh ZCS. Then start it with zmcontrol start. Then zmcontrol stop. After that continue with the install and remove the database contents with rm -rf /opt/zimbra/db/data/* then reinitialize with /opt/zimbra/libexec/zmmyinit. and continue.

Before executing the procedure, you can get your ldap passwords with the zmlocalconfig -s. Then you can place those in during the initial config. Dont change all the passwords, just the ones listed for ldap. Later in the procedure you are asked for zimbra_ldap_password. You'll need to init it also.

Lastly, after you have restored, you may log in and see your mail. If you try to send it will issue an MTA error. You can continue to utilize the new IP by adding it to the trusted networks of the new server. zmprov getServer <servername> | grep zimbaMtaMyNetworks to get the exsiting and zmprov modifyServer <server> zimbraMtaMyNetworks "ip addreses"

I have not documented any of this. I've just done it like 60 times in the last two weeks and have this procedure memorized now.