Isse with DRBD, OSS Cluster

[blason]

Hi Fellas,

I am configuring my first DRBD cluster on CentOS 5.5 and facing hell lot of issues however I managed to configure the DRBD on both the nodes and somehow I am both the nodes are unable to detect each other. Also i do have few queries and would appreciate if DRBD Gurus out there can help me understand.

My scenario is like this

mail1.clus.net 192.168.1.122
mail2.clus.net 192.168.1.123

Here is the /etc/init.d status on mail1
[root@mail1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
m:res cs st ds p mounted fstype
0:r0 StandAlone Primary/Unknown UpToDate/DUnknown - /opt ext3

###############

Hoever on seciondary is uable to detect Primary.

Can someone shed light on this issue?

Also my queries are

1] Since after every reboot though my DRBD services comes up I do need to execute below command on both the nodes else they unable to detect their role from drbd.conf

drbdsetup /dev/drbd0 primary -o AND then mount /dev/drbd0 /opt

So, I would like to know that where I can put that command so that same will be taken after reboot and will survive

Also, do I need to install zimbra before DRBD or can that be installed after everything is finished?

[vavai]

Hi,

Could you please provide the following settings :

1. /etc/hosts
2. /etc/drbd.conf
3. fdisk -l

Recommendation : Installing Zimbra first, configure it and then move (rsync) /opt contents to drbd mount and then remount drbd disk as /opt.

[blason]

Ok, here you go. Well at first I tried everything at one go by following article poseted on other site and I messed up the entire stuff hence I decided to follow one by one approach.
Here are my files

Two nodes that I need to play with are

mail1.clus.net 192.168.1.122
mail2.clus.net 192.168.1.123

Domain - clus.net
#####################

This is on 1.122 [mail1.clus.net]
127.0.0.1 localhost.localdomain localhost
192.168.1.122 mail1.clus.net
192.168.1.123 mail2.clus.net
::1 localhost6.localdomain6 localhost6

*****************
global {
usage-count yes;
}

common {
syncer {
rate 100M;
al-extents 257;
}
}

resource r0 {

protocol C;

handlers {
pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";
pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";
local-io-error "echo O > /proc/sysrq-trigger ; halt -f";
}
startup {
#become-primary-on both; ### For Primary/Primary ###
become-primary-on mail1.clus.net;
degr-wfc-timeout 60;
wfc-timeout 30;
}

disk {
on-io-error detach;
}

net {
# allow-two-primaries; ### For Primary/Primary ###
cram-hmac-alg sha1;
shared-secret "mysecret";
after-sb-0pri discard-zero-changes;
after-sb-1pri violently-as0p;
after-sb-2pri violently-as0p;
}

on mail1.clus.net {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.1.122:7788;
meta-disk internal;
}

on mail2.clus.net {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.1.123:7788;
meta-disk internal;
}
}
*************
[root@mail1 ~]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 2418 19422553+ 83 Linux
/dev/sda2 2419 2609 1534207+ 82 Linux swap / Solaris

Disk /dev/sdb: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 10240 10485744 83 Linux
********************
[root@mail1 ~]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
0: cs:StandAlone st:Primary/Unknown ds:UpToDate/DUnknown r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:3 lo:0 pe:0 ua:0 ap:0 oos:319116

##############################
On 1.123 [mail2.clus.net]

# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.1.122 mail1.clus.net
192.168.1.123 mail2.clus.net
::1 localhost6.localdomain6 localhost6

********************
global {
usage-count yes;
}

common {
syncer {
rate 100M;
al-extents 257;
}
}

resource r0 {

protocol C;

handlers {
pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";
pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";
local-io-error "echo O > /proc/sysrq-trigger ; halt -f";
}

startup {
# become-primary-on both; ### For Primary/Primary ###
degr-wfc-timeout 60;
wfc-timeout 30;
}

disk {
on-io-error detach;
}

net {
# allow-two-primaries; ### For Primary/Primary ###
cram-hmac-alg sha1;
shared-secret "mysecret";
after-sb-0pri discard-zero-changes;
after-sb-1pri violently-as0p;
after-sb-2pri violently-as0p;
}

on mail1.clus.net {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.1.122:7788;
meta-disk internal;
}

on mail2.clus.net {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.1.123:7788;
meta-disk internal;
}
}
***************
root@mail2 ~]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
0: cs:StandAlone st:Secondary/Unknown ds:UpToDate/DUnknown r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 oos:8192

********

The issue right now I see is both the peers are detecting each other I am not able to mount the drive /dev/drbd0 /opt on other secondary box

[root@mail1 ~]# mount -t ext3 /dev/drbd0 /opt/
[root@mail1 ~]#

*****
[root@mail2 ~]# mount -t ext3 /dev/drbd0 /opt/
mount: block device /dev/drbd0 is write-protected, mounting read-only
mount: Wrong medium type

What could be the issue?

[vavai]

Originally Posted by blason

Ok, here you go. Well at first I tried everything at one go by following article poseted on other site and I messed up the entire stuff hence I decided to follow one by one approach.
Here are my files

*************
[root@mail1 ~]# fdisk -l


Disk /dev/sdb: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 10240 10485744 83 Linux
********************

********

The issue right now I see is both the peers are detecting each other I am not able to mount the drive /dev/drbd0 /opt on other secondary box

[root@mail1 ~]# mount -t ext3 /dev/drbd0 /opt/
[root@mail1 ~]#

*****
[root@mail2 ~]# mount -t ext3 /dev/drbd0 /opt/
mount: block device /dev/drbd0 is write-protected, mounting read-only
mount: Wrong medium type

What could be the issue?

It seems that you have wrong setup on your DRBD disk. What is the contents of /dev/sdb1 and does this partition already mounted on some folder elsewhere?

This is an example of my partition for DRBD :

Disk /dev/sdc: 268.4 GB, 268435456000 bytes
255 heads, 63 sectors/track, 32635 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table

As you'll see, Disk /dev/sdc states that it doesn't contain a valid partition table because the partition will be take over by drbd setup.

From your /etc/drbd.conf, disk /dev/sdb was configured for drbd disk. If this is true, you should not make any partition on it.

My Suggestion :

1. Modify /etc/drbd.conf (on both nodes) and change the following line :

disk /dev/sdb1

to :

disk /dev/sdb

2. Backup a mounted disk of /dev/sdb1 if you have important data on it

3. Unmounting /dev/sdb1 on both nodes

4. Re-create drbd meta data disk on both nodes :

drbdadm create-md r0
service drbd start
service drbd status

5. Run the following command on first node to become a primary node :

drbdsetup /dev/drbd0 primary --overwrite-data-of-peer
service drbd status

Paste the results of command no 4 & 5 and let's discuss the rest setup after completing the above step.

[blason]

Argh!!! my DRBD and herbeat is now up but not sure why /dev/drb0 is bing not being mounted on mail2 automatically?
Again this is CentOS5.5 and I am seeing drbddisk script under /etc/ha.d/resources, now any clue what that can be used with Hearbeat so that automatic fail over can happen.

[vavai]

Originally Posted by blason

Argh!!! my DRBD and herbeat is now up but not sure why /dev/drb0 is bing not being mounted on mail2 automatically?
Again this is CentOS5.5 and I am seeing drbddisk script under /etc/ha.d/resources, now any clue what that can be used with Hearbeat so that automatic fail over can happen.

Hmmmh, this make me confused. It seems that you open up 2 different threads for discuss about DRBD+Heartbeat, isn't it?

Try to add the following line on both /etc/ha.d/ha.cf :

logfile /var/log/ha-log

then restart heartbeat services on all node and look at the above log file, why /dev/drb0 is bing not being mounted on mail2 automatically?

[blason]

So, sorry about it rather I was searcing on forum and found similar thread hence I directly started posting there and before that I had initiated my own thread.

So, for my reference would it be possible to share your haresource and ha.cf file? I can quickly compare my wrong one [may be] against good one.

[vavai]

Originally Posted by blason

So, sorry about it rather I was searcing on forum and found similar thread hence I directly started posting there and before that I had initiated my own thread.

So, for my reference would it be possible to share your haresource and ha.cf file? I can quickly compare my wrong one [may be] against good one.

Below is my current configuration, on a running system (Zimbra with DRBD+Heartbeat) :

# cat /etc/ha.d/ha.cf
keepalive 2
warntime 5
deadtime 15
initdead 90
udpport 694
logfile /var/log/ha-log
auto_failback on
bcast eth0
node alpha bravo

# cat /etc/ha.d/haresources 
alpha IPaddr::192.168.1.254/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/opt::ext3 named zimbra

[blason]

Greattt...finally my ha and drbd started working now since I got a hang of It. I am planning to remove the RPM; backup the files then follow the standard sequence. i.e first Zimbra, then drbd and then heartbeat.

However one thing confuses me and its a prerequisite about fiddling with hostnames and IP addresses. Since zimbra very much rely on correct hostname and domain name; MX records how can that be taken care while integrating all those components. Would you please shed some light on that.

[vavai]

Originally Posted by blason

Greattt...finally my ha and drbd started working now since I got a hang of It. I am planning to remove the RPM; backup the files then follow the standard sequence. i.e first Zimbra, then drbd and then heartbeat.

Glad to hear this

Originally Posted by blason

However one thing confuses me and its a prerequisite about fiddling with hostnames and IP addresses. Since zimbra very much rely on correct hostname and domain name; MX records how can that be taken care while integrating all those components. Would you please shed some light on that.

Following explanation may clarify the concept of Zimbra cluster using DRBD & Heartbeat :

Configuration :
mail1.vavai.net : IP 192.168.1.126
mail2.vavai.net : IP 192.168.1.127
mail.vavai.net : IP 192.168.1.128


Note : mail1 is primary node, mail2 is secondary node.

1. Installing Zimbra on Primary node with hostname mail.vavai.net with IP 192.168.1.128

2. Shutting down Zimbra services and then revert setting on primary node (mail1.vavai.net : IP 192.168.1.126)

3. Installing Zimbra on secondary node with hostname mail.vavai.net with IP 192.168.1.128

4. Shutting down Zimbra services and then revert setting on secondary node (mail2.vavai.net : IP 192.168.1.127)

5. Setting up DRBD as your setting

6. Setting up Heartbeat as your setting

Summary : make sure that you have installing Zimbra with Heartbeat IP (virtual IP given by Heartbeat) and proper hostname (mail, instead of mail1 or mail2).

Hope this helps you for simplify your setting.