Every few days web interface hangs on "Loading...", fixed by restarting Zimbra

[simonb]

An instance of Zimbra Release 7.1.2_GA_3268.UBUNTU10_64 UBUNTU10_64 FOSS gets into a state every few days where it stops accepting web logins and just hangs with at the "Loading..." message screen.

During this time the mailbox.log full of error messages...

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown table engine 'InnoDB'

Restarting Zimbra always fixes the problem for another few days. Then it breaks again.

I've tried comparing the zimbra-related processes before and after fault condition and there doesn't seem to be any difference in the process list.

Would appreciate any clues on how to go about debugging this. Previous references to this error message I've found suggest that the MySQL database needs to be updated during an upgrade. However this makes no sense here as this would imply a permanent error condition, not a recurring intermittent one. I've already run the upgrade script anyway and it didn't fix anything.

[phoenix]

Hardware specification and whether this is a new install or upgrade? What does top (or your favourite monitor software) show at the time this problem occurs?

[simonb]

Originally Posted by phoenix

Hardware specification and whether this is a new install or upgrade? What does top (or your favourite monitor software) show at the time this problem occurs?

Hi Phoenix,

It's running on VMWare on a Dell PowerEdge 2950 with 8xXenon E5345 CPUs. The VM is allocated 2CPUs and 3GB RAM.

We have about 40 accounts.

This is an upgrade. I should have mentioned this in the previous post but it the problem appears to have started since we migrated from CentOS to Ubuntu LTS. This was done by rsyncing the /opt/zimbra directory, fixing the permissions and re-installing the Ubuntu version of Zimbra. That was with version 7.0.0 at both ends. We since upgarded to 7.1.2 in the hope that it would clear the error.

I will let you know what is in "top" the next time it is in fault condition.


Also, if it's any help, this is the first error message received in the log printed in full. There are many after this, but the first may give a clue what triggered the condition.

2011-08-20 11:20:21,905 WARN  [mailboxd.csv] [] DbStats - An error occurred while getting current database stats
com.zimbra.common.service.ServiceException: system failure: SQL: 'SHOW INNODB STATUS'
ExceptionId:mailboxd.csv:1313835621905:73ea69cded0f6ef7
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:248)
        at com.zimbra.cs.db.DbUtil.executeQuery(DbUtil.java:134)
        at com.zimbra.cs.db.DbUtil.executeQuery(DbUtil.java:176)
        at com.zimbra.cs.db.DbUtil.executeQuery(DbUtil.java:202)
        at com.zimbra.cs.db.DbStats.getStatData(DbStats.java:47)
        at com.zimbra.common.stats.RealtimeStats.getData(RealtimeStats.java:67)
        at com.zimbra.cs.stats.ZimbraPerf$MailboxdStats.getDataLines(ZimbraPerf.java:520)
        at com.zimbra.common.stats.StatsDumper.call(StatsDumper.java:110)
        at com.zimbra.common.stats.StatsDumper$1.run(StatsDumper.java:79)
        at java.lang.Thread.run(Thread.java:662)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown table engine 'InnoDB'

Query being executed when exception was thrown:
SHOW INNODB STATUS


        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
        at com.mysql.jdbc.Util.getInstance(Util.java:384)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3566)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3498)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1959)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2113)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2113)
        at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2275)
        at com.zimbra.cs.db.DebugPreparedStatement.executeQuery(DebugPreparedStatement.java:156)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
        at com.zimbra.cs.db.DbUtil.executeQuery(DbUtil.java:130)
        ... 8 more

Logging into mysql manually as user zimbra and typing "SHOW INNODB STATUS;" seems to work without throwing any error when not in fault condition. Will try again when in fault condition.

[phoenix]

A single CPU allocated to that server should do, is there any possibility to allocate more RAM to this server (although that should be enough for 40 users)? Do you have any firewall or AppArmor enabled on this system? I don't think I've ever seen this sort of problem mentioned in the forum, any possibility you have a HD problem? Is this on a RAID system? What else runs on this server apart from Zimbra, is it possibly a general performance problem of the host?

Just for my own interest, what was the reason you changed from CentOS to Ubuntu?

[EDIT] I assume that this 'error message' goes away after a restart of Zimbra until the next failure in a couple of days?

[simonb]

Hi Pheonix,

I am using UFW to enable the required access ports and haven't disabled AppArmor. Yes, this is running on a local RAID-10.

I have just noticed that I hadn't allocated any swap file. I've just created a new 10G disk and allocated that as swap. Let's see if that improves things.

The reason I switched from CentOS to Ubuntu is that Ubuntu has a much smaller basic footprint (great with thin provisioning) and starts a lot faster. We are basing all our VMs on an Ubuntu template now.

[phoenix]

Originally Posted by simonb

I am using UFW to enable the required access ports and haven't disabled AppArmor.

I can't believe this would make any difference but could you disable AppArmor to see if anything changes.

Originally Posted by simonb

Yes, this is running on a local RAID-10.

That should be OK.

Originally Posted by simonb

The reason I switched from CentOS to Ubuntu is that Ubuntu has a much smaller basic footprint (great with thin provisioning) and starts a lot faster. We are basing all our VMs on an Ubuntu template now.

CentOS 6 has a minimal install that has a really small footprint (plus a few dependencies you'd need for Zimbra) that should do just as well - CentOS is my distribution of choice but whatever works for you is good.

I can't really find out much about this error (I'm not too hot on debugging mysql problems) other than the fact that most of the information I've read says or implies that the INNODB engine hasn't been enabled (or built in) for the people that are reporting this problem. That seems to be confirmed by the error messages you've posted, as to what's causing it......

When it happens again and before you restart Zimbra can you check the following:

ls -l /opt/zimbra/data/tmp

su - zimbra
mysql

mysql> show variables like 'have_innodb';

mysql> show engines \G

mysql> SHOW INNODB STATUS;

mysql> quit

cat /opt/zimbra/conf/my.cnf

[EDIT]One final question for today. When this problem occurs, is it possible you're running out of space on the volume that this directory is located: " /opt/zimbra/data/tmp"?

[phoenix]

Last question for today. Is it possible that when this problem occurs the partition that holds /opt/zimbra/data/tmp is fulll?

[simonb]

OK. I will check those things next time it goes wrong. The /opt/zimbra (which contains /opt/zimbra/data/tmp) partition certainly isn't full. There's a good 27G left on it.

Thanks for all your help and advice!

[hanspoo]

Hi,

I'm using latest
Release 7.1.2_GA_3268.UBUNTU10_64 UBUNTU10_64 FOSS edition

...the latest upgrade.

Absolutely the same problem, i've checked out all the things commented out and it seems my installation is fine.

Please if someone have a solution besides stop and start zimbra with crontab every night ?

..by the way, the system hangs in the night, may be related to swapping or something ? ... in the day the system work smoothly.

[simonb]

For me it was definitely because I had no swap partition. As soon as I connected to the console I could see all the out of memory errors on the screen.

I can report that there have been no crashes in the few weeks since I added the swap partition.