DBRD and Zimbra, Return Code 127

[buee]

Running a HA cluster via DRBD and Zimbra. The two servers are identical, trying to get ZCS7.1.1 running on CentOS 5.6. Followed a number of How-To's to get this guy up and running.

The plot thickens, multiple domains configured on the server, Split-DNS deployed.

The plot thickens...again. HTTPD running on port 80, Zimbra is proxied from port 80 to port 81 via virtual hosting and described in the Wiki article pertaining to Zimbra and Apache.

Therefore, Zimbra and HTTPD are the services needed to start during a failover. Syncs and all that are done. When I manually kill node1 to test the resource takeover on node2, it errors.

The log of the error:

ResourceManager[4212]:  2011/08/03_13:27:23 info: Running /etc/init.d/zimbra  start
ResourceManager[4212]:  2011/08/03_13:27:23 ERROR: Return code 127 from /etc/init.d/zimbra
ResourceManager[4212]:  2011/08/03_13:27:23 CRIT: Giving up resources due to failure of zimbra
ResourceManager[4212]:  2011/08/03_13:27:23 info: Releasing resource group: zimbra-1 IPaddr::192.168.168.10/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/opt::ext3 zimbra httpd
ResourceManager[4212]:  2011/08/03_13:27:23 info: Running /etc/init.d/httpd  stop
ResourceManager[4212]:  2011/08/03_13:27:23 info: Running /etc/init.d/zimbra  stop
ResourceManager[4212]:  2011/08/03_13:27:23 ERROR: Return code 127 from /etc/init.d/zimbra
ResourceManager[4212]:  2011/08/03_13:27:24 info: Retrying failed stop operation [zimbra]
ResourceManager[4212]:  2011/08/03_13:27:24 info: Running /etc/init.d/zimbra  stop

For testing purposes, I removed httpd from my /etc/ha.d/haresources file and tried again, same thing.

Now, when I installed Zimbra on node2, I did a software only install since /opt/zimbra is already taken care of by the node1 install, that's the working theory anyway.

My DRBD.conf file (confirmed to be the same on both servers, different interface for the sync part):

#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd82/drbd.conf
#

common { syncer { rate 100M; al-extents 257; } }

resource r0 {
        protocol C;
        handlers { pri-on-incon-degr "halt -f"; }
        disk { on-io-error detach; }
        net {  cram-hmac-alg "sha1"; shared-secret "pass"; }
        startup { degr-wfc-timeout 15; wfc-timeout 20; }

        on zimbra-1 {
        address 172.16.0.1:7789;
        device /dev/drbd0;
        disk /dev/sda6;
        meta-disk internal;
        }

        on zimbra-2 {
        address 172.16.0.2:7789;
        device /dev/drbd0;
        disk /dev/sda6;
        meta-disk internal;
        }
}

My /etc/ha.d/haresources file (again, confirmed to be the same on both):

zimbra-1 IPaddr::192.168.168.10/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/opt::ext3 zimbra httpd

My hosts file:

127.0.0.1               localhost.localdomain localhost
192.168.168.10          zimbra.domain1.com zimbra
192.168.168.10          mail.domain1.com mail
192.168.168.10          mail.domain2.com mail
192.168.168.10          mail.domain3.com mail
192.168.168.10          mail.domain4.com mail
192.168.168.11          zimbra-1
192.168.168.12          zimbra-2

I cannot figure out what the error code 127 is.

I'm wondering if it has something to do with the fact that node1 zimbra is listening on 81 and maybe node2 tries to listen on 80?

[vavai]

Try to starting Zimbra manually without DRBD & HA, does this command working successfully or not?

To keep it simple, you must verify that all Zimbra services should be running without problem on both node before move it onto DRBD+HA.

To test whether all services running OK, set your sistem target as DRBD primary (it's partner as secondary node), mount DRBD manually and then try to start your Zimbra services manually.

[buee]

Originally Posted by vavai

Try to starting Zimbra manually without DRBD & HA, does this command working successfully or not?

To keep it simple, you must verify that all Zimbra services should be running without problem on both node before move it onto DRBD+HA.

To test whether all services running OK, set your sistem target as DRBD primary (it's partner as secondary node), mount DRBD manually and then try to start your Zimbra services manually.

Vavai, it's a pleasure to see you responded. I was reading your assistance over at this topic and started running through those paces. My biggest fear is that my primary node is done, ready to go live. I really don't want to have to step back and re-do everything I did on that node so I'm trying to keep it away from there.

I wiped the install off of node2, went back to basics as if I was configuring it for the first time. I took your advice and installed Zimbra, from the beginning, as if it was standalone with the hostname of zimbra, hosts file reading the zimbra.fqdn.com zimbra, and virtual IP. Zimbra is happy as a clam here, starts, stops, does cartwheels, it's perfect.

I moved /opt/zimbra over to tmp. Started DRBD, disconnected node1 (again, trying to keep that data safe), made node2 primary (or in this case, standalone primary) and mounted /dev/drbd0 to /opt. Try service zimbra start, I get:

-bash: zmcontrol: command not found

Try su - zimbra, I get this prompt:

-bash-3.2$

So at least I have a different point that I can troubleshoot from. I just hope I'm not deeper in to screwing it up.

[buee]

Well, I managed to lose the working install when node2 came back online. I really with there was an up to date tutorial on how to do this...

[phoenix]

Originally Posted by buee

Well, I managed to lose the working install when node2 came back online. I really with there was an up to date tutorial on how to do this...

There are plenty of links to the set-up of DRBD, this is obviously a Community Supported feature and you're at the whim of someone writing an up-to-date tutorial.

[buee]

Originally Posted by phoenix

There are plenty of links to the set-up of DRBD, this is obviously a Community Supported feature and you're at the whim of someone writing an up-to-date tutorial.

I understand that. Unfortunately, most of them are outdated and/or missing something. Currently working on another one that I found here. It's old, but it seems to be referenced by other posts/topics as recently as 2011. I've already lost my previous config, so why not try it.

[buee]

One last quick question about DRBD with Zimbra...how are upgrades handled? Do I upgrade one, switch to the other, then upgrade that?

[tibby]

I wish if I would have seen your problem earlier.
I was struggling to get DRBD and Zimbra to work together in the post you have mentioned above.

Regarding your first problem:
HEARTBEAT is the one that starts the services. Zimbra in this case
DRBD is working as storage what you can mount between servers and Heartbeat is lunches zimbra off from this partition. If you got nothing under /opt/zimbra on node#2 after a take-over, then DRBD is probably not mounted correctly.

Regarding the Upgrades:
As far as I know you would have to do upgrades only on the active node. The Passive node will use the same storage you have used on the active.

If you still need help with this, let me know.

[buee]

Originally Posted by tibby

I wish if I would have seen your problem earlier.
I was struggling to get DRBD and Zimbra to work together in the post you have mentioned above.

Regarding your first problem:
HEARTBEAT is the one that starts the services. Zimbra in this case
DRBD is working as storage what you can mount between servers and Heartbeat is lunches zimbra off from this partition. If you got nothing under /opt/zimbra on node#2 after a take-over, then DRBD is probably not mounted correctly.

Regarding the Upgrades:
As far as I know you would have to do upgrades only on the active node. The Passive node will use the same storage you have used on the active.

If you still need help with this, let me know.

I finally resolved the error code 127 by starting completely over. I basically repeated my steps and the second time around, it worked.

About updates:
I was under that thought process, too. But my concern comes in when I start to wonder if anything is not stored in /opt and comes in to play elsewhere later on down the line. For example, an executable. I know they say that everything is stored at /opt/zimbra, but there's always that "what if". I wish someone that has done this could chime in.

[tibby]

Originally Posted by buee

I finally resolved the error code 127 by starting completely over. I basically repeated my steps and the second time around, it worked.

About updates:
I was under that thought process, too. But my concern comes in when I start to wonder if anything is not stored in /opt and comes in to play elsewhere later on down the line. For example, an executable. I know they say that everything is stored at /opt/zimbra, but there's always that "what if". I wish someone that has done this could chime in.

What if...
And what if you wouldn't know about Zimbra...
I think everything has a limit... Even Zimbra and DRBD :-)

But to get back to your question...
Links are working on-off from Zimbra partitions, so you can just simply create a link for the executable where zimbra would use it from.
If you look inside /opt/zimbra/postfix after an upgrade it's done the same way, the only difference is, that links are inside of the /opt/zimbra and not on other part of the file system.