Full backups failing

[najarbeau]

The past two full backups have failed with the following error in our system:

Server: name.domain.com

Label: full-20091212.060200.209
Type: full
Status: completed (with errors)
Started: Sat, 2009/12/12 01:02:00.209 GMT-05:00
Ended: Sun, 2009/12/13 16:30:02.790 GMT-05:00
Redo log sequence range: 949 .. 950
Number of accounts: 2
Number of errors: 1


ERRORS

user@domain.com: system failure: getting table info for zimbra.mailbox
com.zimbra.common.service.ServiceException: system failure: getting table info for zimbra.mailbox
ExceptionId:BackupSetWorkerThread-DB:1260708684722:bbc6991b3db1f309
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:253)
at com.zimbra.cs.db.DbBackup.getTableInfo(DbBackup.ja va:523)
at com.zimbra.cs.backup.BackupAccountSession.storeTab lesToLocalFile(BackupAccountSession.java:266)
at com.zimbra.cs.backup.FileBackupTarget$BackupAcctSe ssion.storeTablesStage(FileBackupTarget.java:1486)
at com.zimbra.cs.backup.FileBackupTarget$BackupSetWor kerThread.run(FileBackupTarget.java:2819)
Caused by: com.mysql.jdbc.exceptions.MySQLNonTransientConnect ionException: Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: Broken pipe

STACKTRACE:

java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutp utStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStre am.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedO utputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputS tream.java:123)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2637)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:15 54)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java :1665)
at com.mysql.jdbc.Connection.execSQL(Connection.java: 3124)
at com.mysql.jdbc.PreparedStatement.executeInternal(P reparedStatement.java:1149)
at com.mysql.jdbc.PreparedStatement.executeQuery(Prep aredStatement.java:1262)
at org.apache.commons.dbcp.DelegatingPreparedStatemen t.executeQuery(DelegatingPreparedStatement.java:20 5)
at com.zimbra.cs.db.DbBackup.getTableInfo(DbBackup.ja va:510)
at com.zimbra.cs.backup.BackupAccountSession.storeTab lesToLocalFile(BackupAccountSession.java:266)
at com.zimbra.cs.backup.FileBackupTarget$BackupAcctSe ssion.storeTablesStage(FileBackupTarget.java:1486)
at com.zimbra.cs.backup.FileBackupTarget$BackupSetWor kerThread.run(FileBackupTarget.java:2819)


** END NESTED EXCEPTION **



Last packet sent to the server was 0 ms ago.

Query being executed when exception was thrown:

SELECT COLUMN_NAME, ORDINAL_POSITION, COLUMN_TYPE, IS_NULLABLE, COLUMN_DEFAULT FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = 'zimbra' AND TABLE_NAME = 'mailbox' ORDER BY ORDINAL_POSITION
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2652)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:15 54)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java :1665)
at com.mysql.jdbc.Connection.execSQL(Connection.java: 3124)
at com.mysql.jdbc.PreparedStatement.executeInternal(P reparedStatement.java:1149)
at com.mysql.jdbc.PreparedStatement.executeQuery(Prep aredStatement.java:1262)
at org.apache.commons.dbcp.DelegatingPreparedStatemen t.executeQuery(DelegatingPreparedStatement.java:20 5)
at com.zimbra.cs.db.DbBackup.getTableInfo(DbBackup.ja va:510)
... 3 more

We have 50 total accounts in the system, so it also appears to me that it is failing on this one user's account and not going any further.

Any help in diagnosing this problem would be greatly appreciated.

[LMStone]

Welcome to the forums!

To help us better diagnose things, please run as the zimbra user "zmcontrol -v" and paste the result as your Zimbra version in your forum Profile.

I'd suggest reindexing the affected mailbox, and if that doesn't work, then manually run:

/opt/zimbra/libexec/zmdbintegrityreport -m

as the zimbra user from a command prompt and paste any errors here.

This command is normally run once a week on Sundays and uses a lot of resources, so best if you can run it in the off hours manually.

Hope that helps get you started,
Mark

[najarbeau]

The command: /opt/zimbra/libexec/zmdbintegrityreport -m returned no errors. I also attempted reindexing the affected mailbox but backups are still hanging.