ZCS Critical Bug Alert

[jholder]

I. Description
There is a bug in ZCS where the mailbox server may delete data that contains information about user mailboxes. This information includes folder names, a list of messages stored in those folders, contacts, user-defined tags, and more. This behavior occurs under certain, unlikely error conditions, but if encountered the users will be unable to login and a full restore of the impacted mailboxes will be required. To prevent any possible data loss on your system we recommend that you follow the steps below to patch all mailbox stores in your system as soon as possible.

II. Impact
ZCS 4.5.0 and greater are affected. ZCS 4.0.x and 3.1.x are not affected.

We have discovered a lack of robustness in our code to handle MySQL error conditions. Under certain combinations of conditions, such as errors in connectivity to the database during mailbox creation, a Zimbra server can drop mailbox group databases. Mailbox group databases store the metadata for a set of user mailboxes. Metadata includes user folder names, a list of messages in those folders, contacts, user-defined tags, and most other information about the mailbox that is not in the actual messages. The loss of the mailbox group database will prevent logins for all users that were in the mailbox group database. On average a mailbox group database has 1% of the users on a server. This error can happen repeatedly, impacting a different mailbox group database with each occurrence.

Details of this bug can be found in http://bugzilla.zimbra.com/show_bug.cgi?id=16879.

This is a critical data loss issue. Any mailboxes affected by this bug will have to be restored from backups.

ZCS 4.5.6 will include the fix for this issue.

III. Solution
This can be done on a live server without taking the server offline. We recommend that all Zimbra administrators perform the following workaround immediately by editing the file:

/opt/zimbra/db/create_database.sql

and *REMOVE* the following line of SQL code (it should be just below the license preamble text) in that file:

   DROP DATABASE IF EXISTS ${DATABASE_NAME};

This should be done on all servers with a mailbox store. This file is re-read each time a mailbox is created, and therefore can be edited without a server restart. We recommend the follow sequence of steps if you choose to edit the file while the server is running

(commands should be executed as the 'zimbra' Unix user):

   cd /opt/zimbra/db

   cp create_database.sql create_database.sql.orig

   cp create_database.sql create_database.sql.edit

(commands should be executed as the 'root' Unix user):

* edit (vi, emacs etc) create_database.sql.edit to remove line *

   mv create_database.sql.edit create_database.sql

The Unix mv command is atomic, and if you follow the above steps, the server will not see an inconsistent copy of this script.

Network Edition Customers: If you have any questions or would like assistance please contact Zimbra Support via the Support Portal.

Open Source Users: If you have any questions, please confine them to the thread in the administrators forum. You can also private message jholder, KevinH, or Phoenix with any private questions or comments.