High I/O wait times leads to a slow system

[cpalsgrove]

We recently went through a major upgrade in our email system (about 3500 active users) where we changed quite a few aspects of the system, all presumably helpful changes that would make the system work better and/or give us greater flexibility to make changes in the future. Now, we are experiencing significant I/O problems, especially when a large volume of messages get delivered to the server all at once, like our daily campus announcements. Briefly, here are the changes that we made:

ZCS 5.0.9 --> ZCS 5.0.12
RHEL5 32-bit --> RHEL5 64-bit
2TB SATA DAS, RAID 1-0 via fibre channel --> 5TB SATA SAN, RAID 5-0 via iSCSI
OS on a dedicated server --> OS installed on VMWare
Single server running ZCS --> One server running MTA, other server running the rest (both virtual servers)

There are three things that I think may be causing this problem, but I'm not sure which is the real culprit: iSCSI, the RAID 5-0, or VMWare. My first guess is the RAID 5-0, but our hope was that going from 4 disks in a RAID0 mirrored to another 4 disks and moving to 14 active disks in a RAID 5-0 would provide an increase in I/O speed since it was writing to many more disks at once and was managed by a SAN which should be able to run a RAID 5-0 quickly and efficiently. Before we wait for a maintenance time to implement a change that may not even work, I'm wondering if anyone has any suggestions as to what we might do to optimize our setup and make this situation better. If more detailed information on any piece of our setup would help, please let me know and I will post it. Thanks in advance for any suggestions!

[phoenix]

I believe that parity based raid level will impact your performance and a RAID10 would be more beneficial. Have a look at this wiki article and check the settings for the file system and recommendations for RAID level.

[bdial]

have you run like iometer on the vm to see that you're actually getting the performance from the san that you're expecting? we encountered an issue when implimenting vmware esx with iscsi where vms would have horrible io performance unless the vmdk was zero'd out first it had to do with writing to clean sectors.

[uxbod]

And how much memory is allocated to the VM ?

[cpalsgrove]

phoenix -
I was afraid the RAID level was what was doing it. We're looking towards a second box for our SAN with SAS drives and putting it in a RAID 10.

bdial -
We are working on getting IOMeter up and running so we can get some good data on I/O speeds.

uxbod -
We have 16GB of memory allocated for the VM where Zimbra lives.

Thanks to all three of you for your responses. We're hoping to get these issues all worked out by this Sunday for our maintenance time.

[jholder]

Install sar (if not already installed)

and run it. It runs a cron job every 10 minutes, and should give you an idea what's causing it.

We don't recommend RAID5, but many people run it without issue.

[uxbod]

Is your iSCSI installation using a separate switch from everything else ?

[cpalsgrove]

sar seems to be giving us the I/O data that we need. I'll wait a full day to get a real feel for how the disks behave under different loads at different times of day.

The iSCSI installation is using a dedicated switch independent of our main network.

[jholder]

Originally Posted by cpalsgrove

sar seems to be giving us the I/O data that we need. I'll wait a full day to get a real feel for how the disks behave under different loads at different times of day.

The iSCSI installation is using a dedicated switch independent of our main network.

Use sar -B for more detailed IO info

[cpalsgrove]

Two weeks ago we switched our storage over from the previously mentioned RAID-50 configuration to a RAID-10 configuration and we're still experiencing incredibly high I/O wait times which are leading to great frustration from our users. I'm really not sure what else might be causing this issue. Are there any other parts of our configuration that I should check or anything else that I've previously mentioned that might be causing such a lag in the system?