Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: High I/O wait times leads to a slow system

  1. #1
    cpalsgrove is offline Intermediate Member
    Join Date
    Oct 2008
    Location
    Westminster, Maryland, USA
    Posts
    17
    Rep Power
    6

    Default High I/O wait times leads to a slow system

    We recently went through a major upgrade in our email system (about 3500 active users) where we changed quite a few aspects of the system, all presumably helpful changes that would make the system work better and/or give us greater flexibility to make changes in the future. Now, we are experiencing significant I/O problems, especially when a large volume of messages get delivered to the server all at once, like our daily campus announcements. Briefly, here are the changes that we made:

    ZCS 5.0.9 --> ZCS 5.0.12
    RHEL5 32-bit --> RHEL5 64-bit
    2TB SATA DAS, RAID 1-0 via fibre channel --> 5TB SATA SAN, RAID 5-0 via iSCSI
    OS on a dedicated server --> OS installed on VMWare
    Single server running ZCS --> One server running MTA, other server running the rest (both virtual servers)

    There are three things that I think may be causing this problem, but I'm not sure which is the real culprit: iSCSI, the RAID 5-0, or VMWare. My first guess is the RAID 5-0, but our hope was that going from 4 disks in a RAID0 mirrored to another 4 disks and moving to 14 active disks in a RAID 5-0 would provide an increase in I/O speed since it was writing to many more disks at once and was managed by a SAN which should be able to run a RAID 5-0 quickly and efficiently. Before we wait for a maintenance time to implement a change that may not even work, I'm wondering if anyone has any suggestions as to what we might do to optimize our setup and make this situation better. If more detailed information on any piece of our setup would help, please let me know and I will post it. Thanks in advance for any suggestions!

  2. #2
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,486
    Rep Power
    56

    Default

    I believe that parity based raid level will impact your performance and a RAID10 would be more beneficial. Have a look at this wiki article and check the settings for the file system and recommendations for RAID level.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  3. #3
    bdial's Avatar
    bdial is offline Moderator
    Join Date
    Jul 2007
    Location
    Baltimore
    Posts
    1,649
    Rep Power
    11

    Default

    have you run like iometer on the vm to see that you're actually getting the performance from the san that you're expecting? we encountered an issue when implimenting vmware esx with iscsi where vms would have horrible io performance unless the vmdk was zero'd out first it had to do with writing to clean sectors.

  4. #4
    uxbod's Avatar
    uxbod is offline Moderator
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,017
    Rep Power
    24

    Default

    And how much memory is allocated to the VM ?

  5. #5
    cpalsgrove is offline Intermediate Member
    Join Date
    Oct 2008
    Location
    Westminster, Maryland, USA
    Posts
    17
    Rep Power
    6

    Default

    phoenix -
    I was afraid the RAID level was what was doing it. We're looking towards a second box for our SAN with SAS drives and putting it in a RAID 10.

    bdial -
    We are working on getting IOMeter up and running so we can get some good data on I/O speeds.

    uxbod -
    We have 16GB of memory allocated for the VM where Zimbra lives.

    Thanks to all three of you for your responses. We're hoping to get these issues all worked out by this Sunday for our maintenance time.

  6. #6
    jholder's Avatar
    jholder is offline Former Zimbran
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    Install sar (if not already installed)

    and run it. It runs a cron job every 10 minutes, and should give you an idea what's causing it.

    We don't recommend RAID5, but many people run it without issue.

  7. #7
    uxbod's Avatar
    uxbod is offline Moderator
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,017
    Rep Power
    24

    Default

    Is your iSCSI installation using a separate switch from everything else ?

  8. #8
    cpalsgrove is offline Intermediate Member
    Join Date
    Oct 2008
    Location
    Westminster, Maryland, USA
    Posts
    17
    Rep Power
    6

    Default

    sar seems to be giving us the I/O data that we need. I'll wait a full day to get a real feel for how the disks behave under different loads at different times of day.

    The iSCSI installation is using a dedicated switch independent of our main network.

  9. #9
    jholder's Avatar
    jholder is offline Former Zimbran
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    Quote Originally Posted by cpalsgrove View Post
    sar seems to be giving us the I/O data that we need. I'll wait a full day to get a real feel for how the disks behave under different loads at different times of day.

    The iSCSI installation is using a dedicated switch independent of our main network.
    Use sar -B for more detailed IO info

  10. #10
    cpalsgrove is offline Intermediate Member
    Join Date
    Oct 2008
    Location
    Westminster, Maryland, USA
    Posts
    17
    Rep Power
    6

    Default

    Two weeks ago we switched our storage over from the previously mentioned RAID-50 configuration to a RAID-10 configuration and we're still experiencing incredibly high I/O wait times which are leading to great frustration from our users. I'm really not sure what else might be causing this issue. Are there any other parts of our configuration that I should check or anything else that I've previously mentioned that might be causing such a lag in the system?

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. [SOLVED] Zimbra logwatch.
    By nishith in forum Administrators
    Replies: 5
    Last Post: 06-10-2009, 04:42 PM
  2. Replies: 8
    Last Post: 10-19-2008, 09:52 AM
  3. [SOLVED] Mailserver down when send file attach of 50Mb
    By ZMilton in forum Administrators
    Replies: 20
    Last Post: 04-10-2008, 11:44 AM
  4. Replies: 2
    Last Post: 02-12-2008, 11:55 AM
  5. Error loading on Mac OS X 10.4.10 server PPC
    By qprcanada in forum Installation
    Replies: 7
    Last Post: 10-26-2007, 06:25 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •