Upgrading from 4.5.6 to 4.5.9: LDAP error

[smcgrath1111]

I am trying to upgrade our server from 4.5.6 to 4.5.9, but I am getting the following error:

 
Setting defaults from saved config in /opt/zimbra/.saveconfig/config.save
   HOSTNAME=phosphor.leddynamics.com
   LDAPHOST=phosphor.leddynamics.com
   LDAPPORT=389
   SNMPTRAPHOST=phosphor.leddynamics.com
   SMTPSOURCE=admin@phosphor.leddynamics.com
   SMTPDEST=admin@phosphor.leddynamics.com
   SNMPNOTIFY=yes
   SMTPNOTIFY=yes
   LDAPROOTPW=(censored)
   LDAPZIMBRAPW=(censored)
Restoring existing configuration file from /opt/zimbra/.saveconfig/config.save...done
Operations logged to /tmp/zmsetup.log.23153
Checking ldap status
Starting ldap
ldap startup failed with exit code 256
daemon: bind(7) failed errno=98 (Address already in use)
slap_open_listener: failed on ldap://phosphor.leddynamics.com:389
Checking ldap status
Starting ldap
ldap startup failed with exit code 256
daemon: bind(7) failed errno=98 (Address already in use)
slap_open_listener: failed on ldap://phosphor.leddynamics.com:389
Setting defaults...Checking ldap status
Starting ldap
ldap startup failed with exit code 256
daemon: bind(7) failed errno=98 (Address already in use)
slap_open_listener: failed on ldap://phosphor.leddynamics.com:389
Checking ldap status
Starting ldap
ldap startup failed with exit code 256
daemon: bind(7) failed errno=98 (Address already in use)
slap_open_listener: failed on ldap://phosphor.leddynamics.com:389
Checking ldap status
Starting ldap
ldap startup failed with exit code 256
daemon: bind(7) failed errno=98 (Address already in use)
slap_open_listener: failed on ldap://phosphor.leddynamics.com:389
Done

The funny thing is, slapd is starting:

root@phosphor:~/zcs# ps ax
23356 ?        Ssl    0:00 /opt/zimbra/openldap/libexec/slapd -l LOCAL0 -4 -u zimbra -h ldap://phosphor.leddynamics.com:389 -f /opt/zimbra/conf/slapd.conf

root@phosphor:~/zcs# telnet phosphor.leddynamics.com 389
Trying 192.168.2.235...
Connected to phosphor.leddynamics.com.
Escape character is '^]'.

Any ideas? Thanks
-Scott

[mmorse]

Most likely you still had some leftover stuff still running when you started the upgrade.

ps auxx |grep zimbra
& then kill any left over services etc

When you think everything's stopped, check your netstat -a or some lsof checks to see if anything else was running.

You can also, as root try running ./zmfixperms from /opt/zimbra/libexec/ when you reach the step "The system will be modified - continue?"

[smcgrath1111]

Thanks mmorse-

Ok, I stopped everything and killed any leftover processes. Here's the output of ps ax before I started the installer:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 *:10000                 *:*                     LISTEN     
tcp        0      0 phosphor.leddyna:domain *:*                     LISTEN     
tcp        0      0 localhost:domain        *:*                     LISTEN     
tcp        0      0 localhost:953           *:*                     LISTEN     
tcp        1      0 phosphor.leddynami:4100 ledterra.l:microsoft-ds CLOSE_WAIT 
tcp6       0      0 *:ssh                   *:*                     LISTEN     
tcp6       0      0 ip6-localhost:953       *:*                     LISTEN     
tcp6       0      0 ip6-localhost:1320      ip6-localhost:4068      TIME_WAIT  
tcp6       0      0 phosphor.leddynamic:ssh ::ffff:192.168.2.5:3003 ESTABLISHED
tcp6       0      0 ip6-localhost:2949      ip6-localhost:1744      TIME_WAIT  
udp   107284      0 *:syslog                *:*                                
udp        0      0 *:1038                  *:*                                
udp        0      0 *:10000                 *:*                                
udp        0      0 phosphor.leddyna:domain *:*                                
udp        0      0 localhost:domain        *:*                                
udp        0      0 *:bootpc                *:*                                
udp6       0      0 *:1039                  *:*                                
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ]         DGRAM                    4165     @/com/ubuntu/upstart
unix  2      [ ACC ]     STREAM     LISTENING     4166     @/com/ubuntu/upstart/logd
unix  2      [ ]         DGRAM                    4266     @/org/kernel/udev/udevd
unix  3      [ ]         DGRAM                    1151949  /dev/log
unix  2      [ ]         DGRAM                    1152696  
unix  2      [ ]         DGRAM                    1045712  
unix  2      [ ]         DGRAM                    1010510  
unix  2      [ ]         DGRAM                    739623   
unix  2      [ ]         DGRAM                    7482     
unix  3      [ ]         STREAM     CONNECTED     7479     
unix  3      [ ]         STREAM     CONNECTED     7478     
unix  2      [ ]         DGRAM                    7368

When the installer asked me if I wanted to upgrade, I shelled out and ran chown zimbra:zimbra -R /opt/zimbra
and
/opt/zimbra/libexec/zmfixperms

Then continued with the upgrade, but unfortunately I still got the same errors. Again slapd was started successfully but the installer keeps trying to start it for some reason(?)

[smcgrath1111]

bump
..........

[phoenix]

You should also disable ipv6 on that system, I assume that you don't have a firewall enabled and you have SElinux (if you have it) disabled ?

[rajahd]

I'm on 4.5.6 on Ubuntu 6.10 and this same problem has been happening to me on every attempted upgrade from 4.5.7 on. I've now tried to install 4.5.10 and same deal - I get stuck in the ldap failed to startup loop. I've ensured all Zimbra processes are killed before doing the upgrade. I've set the directory ownership to Zimbra. I've run the zmfixperms script. Mind you I started this system on 4.5.1 and went through 4 upgrades without any hitch till 4.5.7.

Does anyone have suggestions on how to resolve this? Is there any info I can post that would help determine the root problem?

[rajahd]

Some follow-up info from beating my head on the wall trying to resolve this ...

#1: The ldap failure loop appears to be tied to the identification of the OS, at least in my case. While I'm on Ubuntu Server 6.10 (confirmed from the /etc/lsb-release file) I also have a debian_version file. I'm far from a linux guru but since my upgrades were working up through version 4.5.6 either the file got created (perhaps an apt-get upgrade result?) or 4.5.7+ should be checking for the Debian version AFTER looking for Ubuntu. Anyway, renaming the debian_version file got me past the ldap loop.

#2: At the first "starting mysql" the process hung. After about 20 minutes of no further messages, I used ps -aux to determine that there were 2 concurrent mysql threads. The first was a constant PID while the 2nd would disappear and reappear as a new PID (I assume from the script getting a failure message and trying to restart it). Note that I verified NO zimbra services were running prior to the install script. Killing the mysql process with the constant PID caused the upgrade to continue. It identified the version of the Zimbra database (4.5.6) and then tried to stop mysql at which point it hung again. Killing the process once more caused the script to continue and ultimately complete with no more issues or error messages.

#3: After the process completed I suffered the same problem described in another thread: no e-mail accounts. Towards the beginning of the upgrade process I remembered seeing a message that said something like "removing web apps", followed by a message that said the "openldap-data" directory does not exist. Checking the backups shows this directory does exist (or did at the time the install was started). Perhaps this has something to do with the disappearing users in ldap? Following the instructions in the blog link from that thread I restored the ldap config.xml and the openldap-data directory from my backup. It took a reboot of the server but it appears this finally fixed it.

So, I'm on 4.5.10 but this has been a nightmare of an upgrade. To my knowledge there is nothing other than zimbra and the other required components (bind9 for example) on this server so it should be as plain and simple as they come. I sure hope future upgrades don't go this badly.

[padraig]

i'm on 4.5.6 and was just about to upgrade to 4.5.10 i think i'll wait now.
thanks for the heads up

[rajahd]

FWIW padraig, I think the problems described herein are isolated to Ubuntu installations. It looks from your user info that you're on Red Hat, so I'm thinking you might find the upgrade smooth. If you do go forward with it, make sure you do that full backup just in case

[jholder]

We're currently investigating the LDAP issues that have been encountered.
If you hit any type of issues, please post your entire logs:
/opt/zimbra/.install_history
/tmp/install.log.xxx
/tmp/zmsetup.log.xxxx