BACK STORY: We had a SAN failure last week in which both controllers went offline, thus ripping the storage out from underneath our hosts. This badly corrupted the MySQL database for Zimbra as well as caused some disk corruption that had to be repaired with e2fsck.

Once we had the SAN back online, we brought the Zimbra mailbox server up and discovered the the MySQL DB needed to be restored from backup. We began the process of restoring the MySQL database (just the DB since the files still existed in the storage) and rolling up the redologs. We did not discover until after the restore was complete that the two HSM volumes, though mounted in the file system, were not connected in Zimbra as zmvolumes (zmvolume -l). Thusly....it seems as if during the DB restore...because Zimbra did not list the two HSM volumes anymore....it reconnected all the mail blob entries in the DB, for all the HSM blob files stored on the HSM volumes, to the primary STORE volume. This meant that about 95% of our mail messages on this mailbox no longer pointed to blob files in the correct location, and users got the dreaded "no such blob" error when trying to open one of the affected mail messages.

So....I wrote this script to fix the problem. It took about 5 hours to walk through 3300+ accounts with around 900GB of mis-directed blob data. I'm sharing here in case anyone else is looking for a similar solution in the future.

https://github.com/MattMencel/repair...d_zimbra_blobs

Just FYI....disconnecting your storage from your running MySQL database, intentionally or not, is really really bad....

Matt