R A I S - F S
IMAPStore - Redundant Array of Inexepensive Servers File System
So I've been in search of a good failover solution for an IMAP server for some time now. Basically to have a somewhat realtime hot backup of my server that I can failover to.
The problem is alway replicating the file system in some way while keeping the header and envolope information replicated to a remote server.
So the idea is to create a sort of RAID-1 array across multiple servers for a message store. Sort of a simple GFS...
I was thinking that we could pretty easily build a system where all the message "writes" to filesystem could actually be written to multiple machines. Their locations would be stored in a simple association table to the message-id. Then any request for a message would just do a quick lookup to find the message store. Obviosly this is a bit simplified.
- store a message
- store message file on multiple backend message stores 2 to n
- store message physical location (messageid, server, path) in meta db
- add message stores as storage is running out
- get message
- request message via messageid
- lookup messageid in meta db to get physical location
- grab message and deliver back into imap server exactly as it is now
The idea come from mass image storage systems and Google File System. These work in a similiar way. Essentially any file is stored in multiple locations... its location information is stored in a database. When the file is requested it finds the file in storage and returns it. It could be on one machine or hundreds.
If a system goes down the database with meta location information has already been replicated via db replication. When the file is requested from a server that is down it just grabs the next record for the "other" location on another server.
Why?
- Can add infinite number of servers to expand storage
- Can replicate to multiple data centers pretty easily
- Very cheap implementation, all software no SAN or expensive hardware
- Total Failover.... can setup monitor that assures that messages are
replicated to as many machines as you'd like - Tested, this is how we developed the mass-image storage at our company. This is essentially how Google File System works minus splitting files into blocks.
- Should be very pluggable into Zimbra... meaning make a chance to the MessageStore object and be done with it... the rest of the system shouldn't have a problem.
Anyone interested...?