Migrating to new server

[cedbobking]

I'm wondering if there's a flaw my plan to migrate to a new server. The idea is to create a multi-server environment from our single-server one, move everything from the old to the new server, then shut down the old server resulting in a single-server environment again.

We currently have a single RHEL4 upgraded to zcs-NETWORK-5.0.11_GA_2696, and we're looking to move to new hardware running RHEL5_64-bit, and want to do so smoothly.

1. Enable LDAP replication on rhel4.mydomain.tld per Configuring LDAP Replication.
2. Build rhel5.mydomain.tld.
3. Install zcs-NETWORK-5.0.11_GA_2695.RHEL5_64 pointing to Master LDAP per Configuring LDAP Replication.
4. Move mailboxes.
5. Move MTA, spam and all other roles except LDAP.
6. Promote replica LDAP to master per.
7. Shut down rhel4.mydomain.tld.

Any thoughts, comments, or suggestions?

[uxbod]

Would it not be easier to schedule and inform your user base of the downtime ? Then just let your backup MX hold the email until your new server is online.

[cedbobking]

Technically yes. Organizationally no. I did a trail migration from RHEL4 32-bit to RHEL5 65-bit by essentially creating a new server named the same as the old server and migrating with a backup and restore. That all worked fine, but took quite a while.

We host many different customers on Zimbra through many different domains. We provide them with much more than just email. The way things work for us is if we do a schedule such an event, invariably a customer will have some other issue that will suck up all of our time right when we need to focus on the migration. However, if we can make this work. I can move all of them without any of their knowledge and do the work twenty minutes at a time.

Lastly, we will probably be forced to expand our configuration to a multi-server setup sometime in the future anyway. Why not learn how to make it work now?

[cedbobking]

Since I know some people are following along, I'll post an update.

The running server is RHEL4 32-bit with default LVM partitioning in ext3 from Anaconda on a RAID5 device. It has been updated it to 5.0.11.

The new server is RHEL5 64-bit with a large XFS partition for /opt/zimbra on RAID1 with a hot spare and another for /opt/zimbra/backup and no LVM.

Made sure DNS resolves both servers correctly. Made sure they can communicate with each other.

The original installation on the RHEL4 box started with ZCS 4.5.something using an evaluation license when we were looking for an email solution. It has been through a hostname and default domain name change and bunches of updates. Since I knew less than nothing when I started, I'm not entirely sure of it's configuration. Thanks to the training group at Zimbra I know a lot more than nothing now, but maybe not as much as I'd like.

As a result of my initial skill level I decided to ensure that I know its service passwords by resetting them per this wiki article. I did have to zmcontrol stop and start for things to work again.

Then I installed ZCS 5.0.11 onto the new RHEL5 box as a simple Mailbox Server using the instructions in the Multi-Server Installation Guide. I did not install the logger because that service is running on the original server and the instructions say there should be only one instance of it. Obviously LDAP had to be pointed to the original server and authenticated with the new password created previously. I told it not to create a new admin account. My global administration account has a different name than the default. Perhaps if admin@mydomain.tld existed, it wouldn't have tried to create one.

That install seemed to go just fine. I can see the new server in the Administration Console. I did have to configure the MTA Trusted Networks, because the field was blank and any change to the server configuration requires at least the local IP to be included. I added 127.0.0.0/8 and the RHEL4 server too.

We have a test domain name that I'll use to create and test some new users stored in the new server. If that goes well, I'll move my email. If that goes well, I'll move our whole domain. If that goes well I'll move all the other domains.

After that I'll work on turning it into a LDAP replica.

[cedbobking]

After the install on the new RHEL5 server I decided to change the name of the original RHEL4 server. It was "mail" which made sense for a one server environment, but not a multi-server configuration. The new server started life as "host2" and I renamed the original RHEL4 to "host1" per this wiki article. The name "mail" is now a virtual host and a CNAME to "host1".

A problem arose when I tried to log into the Administration Console on host2. The log server name was wrong. After it was changed you could log into the console without error. I did find it strange that the restart turned off services I didn't think were running or had installed, and turned on only the services I expected.

[zimbra@host2 ~]$ zmprov gacf | grep zimbraLogHostname
zimbraLogHostname: mail.mydomain.tld
[zimbra@host2 ~]$ zmprov mcf zimbraLogHostname host1.mydomain.tld
[zimbra@host2 ~]$ zmcontrol stop
Host host2.mydomain.tld
	Stopping stats...Done
	Stopping spell...Done
	Stopping snmp...Done
	Stopping archiving...Done
	Stopping antivirus...Done
	Stopping antispam...Done
	Stopping imapproxy...Done
	Stopping mailbox...Done
	Stopping logger...Done
[zimbra@host2 ~]$ zmcontrol start
Host host2.mydomain.tld
	Starting mailbox...Done.
	Starting spell...Done.
	Starting stats...Done.
[zimbra@host2 ~]$

So after a test domain was set up and a user created on host1 it seemed to work great. When I moved it to host2, I couldn't "verify" it. When I logged back into mail.testdomain.tld it redirected my query to host2.mydomain.tld but the page wouldn't open. After a day's worth of poking around I discovered that the web server modes weren't the same on the two servers. The server host1 was in redirect mode to force HTTPS all the time, but apparently host2 was not. This wiki article cleared that up.

[zimbra@host2 ~]$ zmtlsctl redirect
[zimbra@host2 ~]$ zmmailboxdctl restart
[zimbra@host2 ~]$

Now the test user seems to work great. I'll backup then migrate my own mailbox next.

Perhaps the moral of this story is to check common configurations in a multi-server setup.

[cedbobking]

So, redirecting URLs to the new mailstore works just great for the web client, but not so well for Outlook or phones. When I first moved my mail, there was no problem using the web client, but my iPhone lost all connectivity. That's because it was pointed at the wrong server now. One answer is to reconfigure all the clients. Another is to install and configure zimbra-proxy.

The beauty of zimbra-proxy is that you can move mailboxes around however you like and never have to touch a phone or workstation. The problem is that correctly configuring it has caused me confusion.

It seems that there is a bug out there that was causing me greif...

Make sure that `zmhostname` matches the URL of the proxy server.

[zimbra@host1 ~]$ zmhostname
host1.mydomain.tld
[zimbra@host1 ~]$ zmprov md mydomain.tld -zimbraVirtualHostname mail.mydomain.tld
[zimbra@host1 ~]$ libexec/zmsetservername mail.mydomain.tld
[zimbra@host1 ~]$ /libexec/zmproxyinit -e -w -m -H mail.mydomain.tld
[zimbra@host1 ~]$ zmcontrol stop; zmcontrol start

I also learned that if you run zmproxyinit it will reset your TLS settings back to http. So if your proxy target is set to https or redirect, it will fail.

[zimbra@host2 ~] zmtlsctl http
[zimbra@host2 ~] zmcontrol stop; zmcontrol start

I also had some issues with outbound mail. It's important to double check all the suggestions from this wiki article.

I think it's possible to configure things such that host1 acts as the sole MTA in the system. However, since my plan involves migrating those roles anyway, I installed zimbra-mta and configured the web relay to be localhost and the outbound relay to be mail.mydomain.tld. Obviously I had to re-check trusted network settings.

As soon as proxy and relay were working I closed up the firewall on host2 to all outside traffic. The only allowable communication is between host1 and host2. Everything continues to work just fine; the proxy service functions correctly. So, without changing my iPhone settings at all, not even looking at them, I was able to send from my Zimbra to my Yahoo! account and vice versa.

I will start moving all the users from mydomain.tld to host2 and see how that goes. Next will creating a secondary LDAP, then our customers' domains.