Migrating from one server to another and upgrading from 4.5 to 5.0 AND port changes

[jadestorm]

Hi Folk! I just successfully upgraded and moved my server and since it was pretty complex and someone else may try the same thing, I thought I'd post what I did.

First off, I used this as a reference:
http://www.zimbra.com/blog/archives/...er_server.html

So... here's the scenario. I have a primary server, we'll call it example.org (10.0.0.1 internally). It handles DNS, NAT/being the internal primary router, and just about everything. To start with, it is running Zimbra. This box is running CentOS 4.6 and Zimbra 4.5.10. The mail domain is example.org, same as the host. Now, seeing as I had HTTP and LDAP services of my own, I was changed the following Zimbra ports:
- HTTP (80) -> 6390
- HTTPS (443) -> 6391
- LDAP (143) -> 6392

I want to move this to another CentOS 4.6 box, move the ports back to their original locations, still allow port 6390/6391 to work, and upgrade to Zimbra 5. I will also need to change the hostname since the original server will retain the example.org hostname. New host will be zimbra.example.org (10.0.0.100 internally). This is going to be a VM under example.org actually, but this isn't really important.

So, pretty complex procedure. Had a number of failures at first but finally nailed it. I'm going to prefix these things with newhost (zimbra.example.org) and oldhost (example.org) to distinguish them. Here is what all I did:

Initial Migration

1. newhost: just in case I was trying things before, make sure all of the zimbra RPMs are not installed
2. newhost: download zimbra package, and: install -s (software only install)
3. newhost: edit /etc/passwd and /etc/group and make the zimbra and postfix users and groups have the same uid and gid as they have on oldhost
4. oldhost: tar up current zimbra install (cd /opt; tar cvjpf zimbra.tar.bz2 zimbra) this takes a while and there are a number of other ways you can do this
5. oldhost: scp zimbra.tar.bz2 root@zimbra.example.org:~
6. newhost: cd /opt
7. newhost: rm -rf zimbra
8. newhost: tar xvjpf ~/zimbra.tar.bz2
9. newhost: edit /etc/hosts, make example.org resolve to this host for now
10. newhost: su - zimbra
11. newhost: zmcontrol start (now, the logger job failed for me once here, i just did a zmcontrol stop, make sure to clean up all zimbra processes, and zmcontrol start again and it was fine the next time)
12. newhost: /opt/zimbra/libexec/zmsetservername zimbra.example.org
13. newhost: zmcontrol stop
14. newhost: edit the following files, changing port 6390 to 80, 6391 to 443, 6392 to 143, admin@zimbra.example.org to admin@example.org, and any hostname (not domain) related example.org to zimbra.example.org:
- conf/swatchrc
- conf/amavisd.conf
- conf/localconfig.xml
- postfix/conf/main.cf
- cyrus-sasl/etc/saslauthd.conf
- ssl/ssl/zmssl.cnf (needed to get root to change this one)
15. newhost: /opt/zimbra/libexec/zmmtainit
16. newhost: zmcontrol start
17. newhost: zmprov ms zimbra.example.org zimbraMailPort 80 zimbraMailSSLPort 443
18. newhost: zmcontrol stop
19. newhost: get root back, remove the /etc/hosts example.org reference to this host
20. newhost: zmcontrol start
21. newhost: log into admin console for zimbra, set up new self signed ssl cert of *.example.org (and set the bottom hosts thing to zimbra.example.org and example.org) I also had to do a zmcontrol stop and zmcontrol start to catch the new certs.
22. oldhost: change the MX record for example.org to point to zimbra.vorpalcloud.org
23. oldhost: I set up postfix and relayhost = [10.0.0.100], mydestination = (yes, leave that blank), relay_domains = localhost, localhost.localdomain, example.org this set up all local mail to go through my new mail host
24. oldhost: set up iptables rules to redirect external network traffic to port 6390 to 10.0.0.100:80, 6391 to 10.0.0.100:443, 25 to 10.0.0100:25
25. oldhost: set up apache to answer also on 6390 and 6391 and set a rewrite rule to redirect to zimbra.example.org port 80 and port 443 (external world never sees this, but this took care of internal world)

Ok, now we have migrated Zimbra 4.5 to the new host. I tried it out and make sure things were working right. Once I was happy with it, I went to upgrade to 5.0.

26. newhost: set up PERLLIB and PERL5LIB variables for root's .bashrc to match zimbra's .bashrc
27. newhost: . ~/.bashrc (to make sure perl vars are loaded)
28. newhost: download zcs-5.0
29. newhost: extract and ./install.sh

At this point, all was working great! Complex, but I got through it! I know some of this could have been done via command line tools instead of direct file edits, but I preferred the manual edit method.

I hope this can be helpful to someone else!

[jholder]

Jade,
Do you have a Zimbra Shirt? Send me your Shipping info to jholder@zimbra.com. I'll send you one.

[jadestorm]

Wow really? Thanks! I'll private message you that info. =)

One other thing, I also set up iptables forwarding for 143 and 993 to the new server from the outside world. =)

[aoddy]

I think option of tar command of No.8 is "xvjpf"

tar xvjpf ~/zimbra.tar.bz2

[jadestorm]

Ack, yes, you are correct.

[dlbewley]

I'm doing basically the same thing. I'm trying to begin by cloning my system
and giving it a new name.

Production server: prodserver.example.com 192.168.215.186 RHEL 4.6x86_64 Zimbra zcs-NETWORK-4.5.9_GA_1454

Test server: testserver.example.com 192.168.102.112 RHEL 4.6x86_64 zcs-NETWORK-4.5.9_GA_1454

1. Install RH on new server.
2. Install Zimbra 'install.sh -s'
3. Remove /opt/zimbra/*
4. Restore a backup of prodserver:/opt/zimbra to testserver.
5. Create a snapshot for repeated testing:

#!/bin/bash
# quick script to revert zimbra to fresh snapshot
mkdir -p /opt/zimbra-static
umount /opt/zimbra/backup
umount /opt/zimbra
umount /opt/zimbra-static
lvremove -f /dev/VG0/SnapZimbra
mount /dev/VG0/LVZimbra /opt/zimbra-static
lvcreate -L2G -s -n SnapZimbra /dev/VG0/LVZimbra
mount /dev/VG0/SnapZimbra /opt/zimbra
mount /dev/VG0/LVZimbra /opt/zimbra-static
mount /dev/VG0/LVZimbraBak /opt/zimbra/backup

6. Make sure testing on testserver can not affect prodserver

[root@prodserver log]# iptables -I INPUT  -s 192.168.102.112 -j REJECT

7. The `hostname` command on testserver returns testserver.example.com. I didn't do any DNS magic, but I did:

echo "192.168.102.112         testserver.example.com testserver" >> /etc/hosts
echo "192.168.102.112         prodserver.example.com prodserver" >> /etc/hosts

8. As user zimbra:

/opt/zimbra/libexec/zmsetservername testserver.example.com

Watched it scroll by as it manipulated all the accounts. This started up and left a slapd running with a reference to prodserver:

zimbra   28016     1  0 15:41 ?        00:00:00 /opt/zimbra/openldap/libexec/slapd -l LOCAL0 -4 -u zimbra -h ldap://prodserver.example.com:389 -f /opt/zimbra/conf/slapd.conf

So I did a zmcontrol stop.
9. Fix all the config files that still have the prodserver hostname in them:

[root@testserver conf]# for f in `grep -l prodserver /opt/zimbra/conf/*`; do \
echo "fixing $f"; \
perl -pi.bak -e 's/prodserver/testserver/g'; \
done

10. zmcontrol start
11. Try to run zmprov and get error

[zimbra@testserver ~]$ zmprov help
[] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
[] INFO: Retrying request
[] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
[] INFO: Retrying request
[] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
[] INFO: Retrying request
ERROR: zclient.IO_ERROR (invoke Connection refused, server: localhost) (cause: java.net.ConnectException Connection refused)

So I dug a little and found and tomcat is not running.

[zimbra@testserver ~]$ zmcontrol status
Host testserver.example.com
        antispam                Running
        antivirus               Running
        ldap                    Running
        logger                  Stopped
                zmlogswatchctl is not running
        mailbox                 Stopped
                tomcat is not running
        mta                     Running
        snmp                    Stopped
                swatch is not running
        spell                   Running
        stats                   Stopped

In /opt/zimbra/apache-tomcat-5.5.15/logs/catalina.out I find the following

Apr 2, 2008 5:08:26 PM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-80
Apr 2, 2008 5:08:27 PM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-443
Apr 2, 2008 5:08:27 PM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-7071
Apr 2, 2008 5:08:27 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1799 ms
Apr 2, 2008 5:08:27 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Apr 2, 2008 5:08:27 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.15
Apr 2, 2008 5:08:27 PM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
log4j:WARN No appenders could be found for logger (org.apache.catalina.startup.TldConfig).
log4j:WARN Please initialize the log4j system properly.
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base /opt/zimbra/apache-tomcat-5.5.15/server/webapps/manager does not exist or is not a readable directory
        at org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:140)
        at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:3812)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:3983)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:759)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:739)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:524)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:603)
        at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:535)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:470)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1118)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:310)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1020)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:718)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442)
        at org.apache.catalina.core.StandardService.start(StandardService.java:450)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:700)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:551)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:275)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:406)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.zimbra.cs.launcher.TomcatLauncher.start(TomcatLauncher.java:42)
        at com.zimbra.cs.launcher.TomcatLauncher.main(TomcatLauncher.java:64)
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext start
SEVERE: Error in resourceStart()
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext start
SEVERE: Error getConfigured
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/manager] startup failed due to previous errors
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext stop
INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] has not been started
Apr 2, 2008 5:08:29 PM org.apache.catalina.startup.HostConfig deployDescriptor
WARNING: A docBase /opt/zimbra/apache-tomcat-5.5.15/webapps/zimbra inside the host appBase has been specified, and will be ignored
log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base /opt/zimbra/apache-tomcat-5.5.15/server/webapps/host-manager does not exist or is not a readable directory
        at org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:140)
        at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:3812)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:3983)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:759)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:739)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:524)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:603)
        at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:535)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:470)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1118)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:310)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1020)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:718)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442)
        at org.apache.catalina.core.StandardService.start(StandardService.java:450)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:700)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:551)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:275)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:406)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.zimbra.cs.launcher.TomcatLauncher.start(TomcatLauncher.java:42)
        at com.zimbra.cs.launcher.TomcatLauncher.main(TomcatLauncher.java:64)
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext start
SEVERE: Error in resourceStart()
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext start
SEVERE: Error getConfigured
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/host-manager] startup failed due to previous errors
Apr 2, 2008 5:08:29 PM org.apache.catalina.core.StandardContext stop
INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager] has not been started
log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.
Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Zimbra server reserving server socket port=143 bindaddr=null ssl=false
Zimbra server reserving server socket port=993 bindaddr=null ssl=true
Zimbra server process is running as root, changing to user=zimbra uid=500 gid=500
Zimbra server process, after change, is running with uid=500 euid=500 gid=500 egid=500

Well, /opt/zimbra/apache-tomcat-5.5.15/server/webapps is empty on prodserver and testserver. Red herring?

I found this post Tomcat error messages and I did

[root@testserver logs]# /opt/zimbra/libexec/zmfixperms

and it made no perceptable change.

Any ideas how to make tomcat happy?

[dlbewley]

Yay.

I looked in mailbox.log and found

 [root@testserver zimbra]# grep config.frm /opt/zimbra/log/mailbox.log
Caused by: java.sql.SQLException: Incorrect information in file: './zimbra/config.frm'

And I then looked in
/opt/zimbra/db/data/`hostname`.err and found many similar errors like this

080403 13:22:10 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './zimbra/volume.frm'
080403 13:22:24 [ERROR] /opt/zimbra/mysql/libexec/mysqld: Incorrect information in file: './zimbra/config.frm'

Turns out the innodb buffers are set to large for my test machine. The
production machine as 16GB and the test machine has 2GB.

[zimbra@testserver conf]$ grep size /opt/zimbra/conf/my.cnf
sort_buffer_size = 1048576
read_buffer_size = 1048576
# Increase the size of the table cache, since each mailbox has its
innodb_buffer_pool_size   = 5025638400
innodb_log_file_size      = 104857600
innodb_log_buffer_size    = 8388608

[zimbra@prodserver conf]$ free
             total       used       free     shared    buffers     cached
Mem:      16359336   16249640     109696          0     335676    7426924
-/+ buffers/cache:    8487040    7872296
Swap:      5261240       5120    5256120

[zimbra@testserver conf]$ free
             total       used       free     shared    buffers     cached
Mem:       2057292    1472876     584416          0     125600     621432
-/+ buffers/cache:     725844    1331448
Swap:      1052248      51188    1001060

So after turning innodb_buffer_pool_size way down tomcat started and Zimbra
WUI works. Now I'll just have to coax out the appropriate values and continue
with upgrade testing.

I wonder if there is a zimbra tool that comes up with some sane defaults for a given RAM configuration.

[dlbewley]

I wonder if there is a zimbra tool that comes up with some sane defaults for a given RAM configuration.

Why yes. There is such a handy script. It is /opt/zimbra/libexec/zmmycnf. This looks a little more sane for a 2G box.

[zimbra@testserver libexec]$ ./zmmycnf |grep size
sort_buffer_size = 1048576
read_buffer_size = 1048576
# Increase the size of the table cache, since each mailbox has its
innodb_buffer_pool_size   = 632000102
innodb_log_file_size      = 104857600
innodb_log_buffer_size    = 8388608

[wfweaver]

I currently have a multi-server install of Zimbra 5.0.2 NE. There are 2 smtp servers, 2 mailbox servers serving production, and 1 server used internally that has smtp and mailbox installed (as well as logger). The master ldap server is currently on one of the production smtp servers. When initially installed, some of the servers were built under 32 bit RedHat and others under 64 bit. I'd like for them all to be 64 bit. In addition, I want to split out the master ldap to its own server and have replicated ldap's on the mta's. Finally, I have a new server I've built to replace the server used internally so I need to do a 32 bit to 64 bit migration on it as well as moving all the mailboxes, logger db, etc over to it.

I expect this to be a fairly complex migration. What is the best/recommended way of performing this upgrade and migration? How can I best do this to minimize downtime to the users?

[Sean1072]

I followed the steps as noted above, and after copying over the oldhost zimbra folder I start having issues. The admin ui doesn't come up (regular ui is fine) and when I try to run zmprov I get the following:

ERROR: zclient.IO_ERROR (invoke unable to parse response: <html><head><title>Apache Tomcat/5.5.15 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - Servlet AdminServlet is not available</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Servlet AdminServlet is not available</u></p><p><b>description</b> <u>The requested resource (Servlet AdminServlet is not available) is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/5.5.15</h3></body></html>, server: localhost) (cause: com.zimbra.soap.SoapParseException unable to parse response: <html><head><title>Apache Tomcat/5.5.15 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - Servlet AdminServlet is not available</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Servlet AdminServlet is not available</u></p><p><b>description</b> <u>The requested resource (Servlet AdminServlet is not available) is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/5.5.15</h3></body></html>)

I was able to install zimbra 4.5.10 on the newhost successfully. The admin ui worked and zmprov wouldn't barf when I tried to use it. After copying over the old files, I noticed this in mailbox.log after the zmcontrol start:

2008-05-28 12:51:04,007 FATAL [main] [] soap - Unable to start servlet
com.zimbra.common.service.ServiceException: system failure: Could not bind to port=7025 bindaddr=xxx.xxx.xxx.xxx ssl=false useChannels=false

The first thing I noticed is that binaddr is the ip address of the oldhost. It's not a hostname (like everyone else's examples seem to be). I've grep'd through all the files for the IP looking for something that the zmsetservername may have missed and haven't found anything.

I've tried several different times and still encounter the same error.

Has anyone else had this problem? Are there any other Zimbra logs that might point me in the right direction?

Any help would be greatly appreciated.

Thanks!