iSCSI target for Zimbra, ESXi vs dedicated machine vs VMWare Server local install?

[batfastad]

Hi everyone

I'm finally getting round to trying to evaluate Zimbra (I really don't want to renew our SBS 2003 licenses) and I'm looking at hardware/virtualisation.

I'm mainly confused/concerned about the storage side of things.
We have a hardware RAID card (3Ware 9690SA-8i) with 4 spare drive bays in the backplane which I was planning to use for a zimbra RAID 10 array.
However this machine is currently our shared files NAS (Samba/Netatalk) and our intranet database system (Apache/PHP/MySQL)

I have already tried Xen but found a problem with my Tyan motherboard which would only boot with a garbled boot screen. Tried OpenVZ too but couldn't get the memory allocation correct to prevent Zimbra Java VM errors.

Option A:
- Install Openfiler on the big NAS box and export the storage as iSCSI targets (or maybe configure the targets manually under CentOS if I'm feeling brave)
- Build a new beast box for ESXi and run 3 VMs (zimbra / shared files / intranet DB)
- Connect the 2 machines via gigabit ethernet directly, bypassing the network switch (or maybe on a separate VLAN, not used them before) so the storage traffic is isolated from the rest of the network

Option B:
- Configure iSCSI targets manually under CentOS on the NAS box
- Run our shared files and intranet DB on the NAS box as we do currently
- Build a dedicated Zimbra box that mounts the iSCSI storage
- Connect the 2 machines via gigabit ethernet directly, bypassing the network switch (or maybe on a separate VLAN, not used them before) so the storage traffic is isolated from the rest of the network

Option C:
- Install VMWare Server on the big storage box and run Zimbra in a CentOS VM accessing the storage by SCSI passthrough
So everything's running purely on one machine, but VMWare Server I/O performance is another concern

I like the idea of option B the most... it's simple with no virtualisation bottlenecks (or potential bottlenecks) and all configured ourselves.
But I worry how complicated iSCSI targets and multi-pathing are to set up manually. It seems to be ok from a few guides I've read, but performance tweaking could prove difficult.
Also means I could dedicate some RAM as a ramdisk for the anti spam/virus definitions (I've heard that can be beneficial).
And I'd like to stay away from large VM disk files and keep everything as a raw file system - so I can just mount in case of disaster.

Then 2nd is option A, and VMWare Server would be my last option as I've heard I/O performance can be slow and I'm concerned about stability of VMware Server... random freezes etc.

On to my questions:
1) Should I be worried about iSCSI latency compared to directly connected storage when using Zimbra?
We're a small business of 15 users all with relatively Exchange mailboxes of 2-8GB.

2) For option B... I would be looking to mount the zimbra iSCSI target as /opt
Zimbra will the only thing that will be running on that server so that should mean anything Zimbra-related (mail store, logs, settings etc) will all be stored on the iSCSI storage and not locally.
Is Zimbra completely contained in /opt/zimbra or does it keep anything elsewhere on the system?

3) With ESXi, could I store the whole VM on an iSCSI target?
Or would it be better to just have Zimbra or the mail store stored on iSCSI and the rest of the OS on a local internal HDD?

4) Would iSCSI targets in ESXi be a better option than VMWare Server's (apparently) relatively poor I/O performance with local storage?

It seems a bit pointless creating a RAID 10 array, if iSCSI bandwidth is going to be the bottleneck.
I've also read that iSCSI performance with ESXi can vary... apparently sometimes its better to use the iSCSI initiator in the guest OS rather than in ESXi. So that's another reason I'm tempted by option B.

What does everyone reckon?

Cheers, B

[adx442]

Here's what I'm doing successfully, with a similar situation. I have Zimbra NE on a RAID-10 array ESXi server with a limited (big, but limited) amound of space. Mount iSCSI on the Zimbra box through fstab at boot time, and that's where HSM messages and the backups go.

This way, you have very little latency for the live mail server, and you can set HSM to archive messages older than 30 or 60 days, and never have to worry about running out of space in the virtual machine. I haven't had any problems at all with this setup. I have heard counterindications to using Zimbra's live /opt/zimbra on iSCSI though, here on the forums.

Hope this helps a bit with your decision.

------------------------

Edit: I have personally run Zimbra on VMware Server for testing. It works fine ... for a while. I ended up with corrupt databases a couple of times, odd freezes, and total lockups on a weekly basis (odd things like SSH into the server would cause it to hang completely after 30 minutes). No problems since moving to ESXi.

[y@w]

I personally haven't run Zimbra on an iSCSI target, but there's a thread you may want to read: NAS/iSCSI

I know of a fairly large organization running Exchange over iSCSI without issue, so I'd imagine it would be pretty good as long as you throw decent hardware at it. You'll also want to either make sure you have some hardware assist with your ethernet cards or plenty of CPU. iSCSI hammers on processor time whether that's in a TOE card proc or on the main CPU.

I mostly wanted to throw out there that you should avoid option C at all costs. There's nothing wrong running it on ESX/ESXi (we do on a FC-based SAN), but I've heard nothing but woes with running it on VMware Server (and have experienced a few myself).

[batfastad]

@y@w
Strangely enough I did read that thread a few days ago after a quick google turfed it up. That's where my idea of Openfiler came from. A few years ago I was going to use FreeNAS/Openfiler to power our NAS device but ended up building it ourselves.

That thread is interesting actually. So 250+ NE users access a mailstore over iSCSI... sounds promising.
I'd like to have as much as possible stored on the RAID array rather than on local storage so I guess I'll just have to give it a go.
Also it seems like the bottleneck could well be processor... would a quad core xeon X3360/L3360 cut it?

Thanks all for ruling out the VMware Server option. It felt like a bit of a kludge when typing it up. It's those random freezes that scare me the most.

I'm looking into this multipathing thing as well, to try and give extra bandwidth between the NAS box and the Zimbra box... anyone got any experience with that?

I was thinking I would directly connect the NAS and Zimbra boxes, so all the storage traffic is isolated down a dedicated gigabit link. Then have another gigabit link going to the switch for the "public" traffic.
Is that practical?

Anyone else got any experience of having the full Zimbra partition stored on iSCSI?

Cheers, B

[bdial]

we run 2 mailbox servers serving about 600 users. Both are virtual machines running in esx 3.5. We use the iscsi initiator at the guest level and have 3 separate targets mounted for

/opt/zimbra/store
/opt/zimbra/index
/opt/zimbra/backup

The rest of the /opt/zimbra is on the virtual disk. We're using a HP/Lefthand iscsi SAN.

It's been rock solid.

[y@w]

Reading through the OP again, I noticed that it's only 15 users.. If the iSCSI target can't handle that I'm not sure how useful it would be for anything

[batfastad]

Wow that's great news. So our 15 user set-up should work perfectly!

@bdial: It's encouraging to know you have the full zimbra mailstore over iSCSI, I basically want as much as possible stored on the iSCSI target taking advantage of our 3Ware card.

I probably won't be going with VMware initially... I'll check what performance I get out of a dedicated machine. Also we don't have loads of servers that we need to virtualise, only 2 separate boxes and I can keep our file sharing on the NAS box as it is, and just have an iSCSI target exported for Zimbra

I'll need to buy new motherboard/processor/memory for the new box anyway, whether it be for ESXi or a raw CentOS/Zimbra installation.
I'll try it with iSCSI and post back. I can always just stuff a couple more drives in the box and have it as local storage

Cheers, B

[kevinriahi]

We are in the process of a new NE 6.0.5 deployment and the Mail store servers are very large, Dual Quad Core, 32GB RAM, 880GB RAID 50 (8 x 147GB 15K SAS).
But we are also going to use iSCSI for HSM and Backup.
My questions are regarding /etc/fstab + LVM on CentOS 5.4. 64bit

The backup iSCSI dev is 3.4TB
The HSM iSCSI dev is 1.4TB

1) iSCSI setup, any info on /etc/iscsi/iscsid.conf

2) Format and Mount iSCSI Volumes
# mkfs.ext3 /dev/sdx (options)

3) What is the the best way to add the iSCSI devs in a LVM setup.

4) Where to mount iSCSI drives automatically at boot time for Backup and HSM?
/dev/sdx /xxx/xxx ext3 _netdev 0 0 (additional performance options)


5) Using UUID instead of traditional /dev/sdx


Thanks
Kevin

[dave_kempe]

Kevin
not sure about some of your points - are they actually questions.
But for what is worth, I will add some knowledge I have gained.

Make sure you use bonding, and understand the bonding modes.
for point 3, we use a script that provisions the iscsi volumes using ietadm, creates a tid for the volume, and then adds the LVM slice to the tid in a second command. I don't think you can initiate an LVM lun into iscsi all on one line. we then write the correct parameters into /etc/iet/ietd.conf for surviving reboots. btw, get the latest iscsi target for this, as MaxSessions is good, and the reliability is much improved in later linux versions.

when mounting iscsi targets, we use /dev/disk/by-path/ to address the volumes as that seems to me to be the most reliable way of dealing with them. Then you can have a predictable name for the iscsi target.