[SOLVED] mysql/innodb vserver hashify conflict produces data loss

[gracedman]

Hello, all. After weeks of troubleshooting, we have discovered a conflict between the zimbra mysql/innodb process and the vserver hashify process. We are running zimbra on CentOS 5.3 on VServer with reasonable success other than this major issue.

Whenever we run vserver <zimbra server name> hashify, we see odd
behaviors:

* The mysql and zimbra database directories are time stamped with
the time hashify ran as if mysql was restarted but there is no
record of mysql being restarted.
* MOST IMPORTANTLY, the innodb log stops recording but it appears
that mysql is still functioning. Mail is displayed, events
(e.g., calendar, tasks) are recorded and are fully functional

The disaster occurs when mysql is restarted, e.g., when restarting the
zimbra service. Since the innodb log has recorded no data since hashify
ran, mysql thinks the system has crashed and backs out all data since
the last hashify! At least, I believe that's what is happening in my
MySQL ignorance.

As some background on hashify, the hashify process is not an essential process but it is the single VServer feature which tipped us to choose VServer over OpenVZ and is a life saver in the managed service provider market.

There may be hundreds of vservers on a host. This density is a result
of the excellent resource utilization of VServer. Since many if not most of the systems may duplicate file, e.g., the majority of the distribution binaries, hashify is used to eliminate the redundancy both on disk and in RAM.

The vserver files are hashed. Any files with the same hash are replaced
with immutable but unlinkable hard links except for one base copy. As a result, one hundred instances of the file will take little more space than a single instance on both disk and in RAM. Explanations can be found here util-vserver:Vhashify - Linux-VServer and here Frequently Asked Questions - Linux-VServer

Any ideas on why this scanning/linking process would cause this behavior in innodb? Any suggestions on where to start troubleshooting? Thanks - John

[Rich Graves]

As you were told a month ago, you can get a Linux environment suitable for running Zimbra from VMWare, Xen, or raw hardware. If you need to share hardware, you ought to be able to run your other legacy vservers, unmodified, on top of VMWare or Xen... though if your resources are that constrained, you should also consider outsourcing.

How much would you value a month of your time and 30 days' worth of crashes and lost mail?

[gracedman]

Sorry, I still vehemently disagree. This has been very costly to us but the world is not just about me. I have gladly wrestled this to the ground and now we have solved it so that others may benefit from our work. I believe that is one of the core definitions of community and open source contribution - my and my company's way of contributing rather than just leaching. Yes, this one cost us a lot more than we would have liked and we sure wish someone else had paid the tuition before us but our contribution pales in light of what it has cost others to give us the wealth of open source products that power our company.

Along the way, we have enriched our engineering knowledge base so we can move beyond taking the defaults and being glad it works. We now know much more about how Zimbra works. We have resolved many other issues along the way and can now bend the product to our environment so that our environment can better serve our clients rather than having to bend our environment to the product. We did not find it particularly helpful to be told to disable most of our security, violate best practice security models, change our entire infrastructure and all of our other IT products just so that this one product would work. Although it is understandable from the point of view of containing the cost of support, it is a product centric approach rather than a customer centric one. We have now preserved our security model, our infrastructure, our other product choices, can use Zimbra in that environment and will share that knowledge with others.

We will post the resolution shortly once we have worked out the final details.

[gracedman]

Since we are not running multiple zimbra instances on the same vserver host, it made sense to simply exclude the entire /opt/zimbra directory from hashify. We could have excluded /opt/zimbra/db/data (and I suppose /opt/zimbra/logger/db/data) only but this seemed to kill two birds with one stone (eliminate useless hashification and solve our problem).

To do this, we did:

mkdir /etc/vservers/zimbra/apps/vunify
cp /usr/lib64/util-vserver/defaults/vunify-exclude /etc/vservers/zimbra/apps/vunify/exclude
echo /opt/zimbra >> /etc/vservers/zimbra/apps/vunify/exclude
vserver zimbra hashify

This assumes a 64 bit system and a vserver named zimbra.

There were several other instances we encountered to enable zimbra to run with all features enabled on a vserver although this was certainly the most serious. We also encountered some small issues with CentOS 5.3 We will pull all those together into a how-to for both the Zimbra and VServer communities.

We have elected to not find why hashify and mysql/innodb do not get along. This does not normally manifest itself because MySQL defaults its data directory to /var and /var is excluded from hashify by default. Zimbra's use of /opt/zimbra to store the databases unearthed the problem.

Hope this helps someone else! - John