Results 1 to 5 of 5

Thread: IMAPStore Replication/Failover IDEA - Multiple Virtual message stores

  1. #1
    kbaker is offline Active Member
    Join Date
    Jan 2006
    Posts
    33
    Rep Power
    9

    Default IMAPStore Replication/Failover IDEA - Multiple Virtual message stores

    R A I S - F S

    IMAPStore - Redundant Array of Inexepensive Servers File System


    So I've been in search of a good failover solution for an IMAP server for some time now. Basically to have a somewhat realtime hot backup of my server that I can failover to.

    The problem is alway replicating the file system in some way while keeping the header and envolope information replicated to a remote server.

    So the idea is to create a sort of RAID-1 array across multiple servers for a message store. Sort of a simple GFS...

    I was thinking that we could pretty easily build a system where all the message "writes" to filesystem could actually be written to multiple machines. Their locations would be stored in a simple association table to the message-id. Then any request for a message would just do a quick lookup to find the message store. Obviosly this is a bit simplified.

    - store a message
    • store message file on multiple backend message stores 2 to n
    • store message physical location (messageid, server, path) in meta db
    • add message stores as storage is running out

    - get message
    • request message via messageid
    • lookup messageid in meta db to get physical location
    • grab message and deliver back into imap server exactly as it is now


    The idea come from mass image storage systems and Google File System. These work in a similiar way. Essentially any file is stored in multiple locations... its location information is stored in a database. When the file is requested it finds the file in storage and returns it. It could be on one machine or hundreds.

    If a system goes down the database with meta location information has already been replicated via db replication. When the file is requested from a server that is down it just grabs the next record for the "other" location on another server.

    Why?
    • Can add infinite number of servers to expand storage
    • Can replicate to multiple data centers pretty easily
    • Very cheap implementation, all software no SAN or expensive hardware
    • Total Failover.... can setup monitor that assures that messages are
      replicated to as many machines as you'd like
    • Tested, this is how we developed the mass-image storage at our company. This is essentially how Google File System works minus splitting files into blocks.
    • Should be very pluggable into Zimbra... meaning make a chance to the MessageStore object and be done with it... the rest of the system shouldn't have a problem.


    Anyone interested...?
    Last edited by kbaker; 07-20-2006 at 04:16 PM.

  2. #2
    KevinH's Avatar
    KevinH is offline Expert Member
    Join Date
    Aug 2005
    Location
    San Mateo, CA
    Posts
    4,789
    Rep Power
    18

    Default

    Interesting... Slightly different approach than our planned replication which will involve journal shipping. The advantage of our approach is you get all the meta changes and all the rest of the collaboration data updates/writes. So it's an entire solution and not just for email.
    Looking for new beta users -> Co-Founder of Acompli. Previously worked at Zimbra (and Yahoo! & VMware) since 2005.

  3. #3
    Dirk's Avatar
    Dirk is offline Moderator
    Join Date
    May 2006
    Location
    England.
    Posts
    927
    Rep Power
    10

    Default

    I've looked into this sort of thing too, I considered using Network Block Device ( http://nbd.sourceforge.net/ ) which is essentially having a raid array split between disparate computers.

    So you could have two servers, each with a set of drives set up in raid 5 via hardware, then you could use the NBD driver to mirror those so they were always in sync, all writes would end up on both sets of drives.

    Anyway, I tried it, and couldnt get it working, your mileage may vary. I think I'd prefer to wait and see what the Zimbra crew come up with. Hacking your way under zimbra to make it fault tolerant is one thing, but having zimbra do it itself has to be a better way.

  4. #4
    goetzi is offline Project Contributor
    Join Date
    Nov 2005
    Location
    Austria
    Posts
    223
    Rep Power
    9

    Default

    Quote Originally Posted by KevinH
    Interesting... Slightly different approach than our planned replication which will involve journal shipping. The advantage of our approach is you get all the meta changes and all the rest of the collaboration data updates/writes. So it's an entire solution and not just for email.
    That replication feature sounds very interesting. Will there be the possibility to replicate just a part of the mailboxes/users? Can you say that user group A will be synced to server X and group B to server Y?

  5. #5
    KevinH's Avatar
    KevinH is offline Expert Member
    Join Date
    Aug 2005
    Location
    San Mateo, CA
    Posts
    4,789
    Rep Power
    18

    Default

    Not sure on that.. We've not completed the design.
    Looking for new beta users -> Co-Founder of Acompli. Previously worked at Zimbra (and Yahoo! & VMware) since 2005.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •