Full Backup Failing

[the_griz]

I can no longer get a full backup on the system:

Here's the report I get. I would appreciate any help in explaining:

Server: myserver.com

Label: full-20100623.000007.064
Type: full
Status: completed (with errors)
Started: Tue, 2010/06/22 20:00:07.064 EDT
Ended: Wed, 2010/06/23 02:54:32.869 EDT
Redo log sequence range: 2877 .. 2877
Number of accounts: 52
Number of errors: 3


ERRORS

account1@myserver.com: system failure: getting map of message id to blob digest
com.zimbra.common.service.ServiceException: system failure: getting map of message id to blob digest
ExceptionId:FullBackupThread:1277275671880:79106c4 9fc4e6bd1
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:251)
at com.zimbra.cs.db.DbBackup.getBlobDigestMap(DbBacku p.java:223)
at com.zimbra.cs.backup.BackupAccountSession$BlobDige sts.preload(BackupAccountSession.java:500)
at com.zimbra.cs.backup.BackupAccountSession.loadBlob DigestMap(BackupAccountSession.java:463)
at com.zimbra.cs.backup.FileBackupTarget$BackupAcctSe ssion.storeBlobsBeforeMaintenanceMode(FileBackupTa rget.java:1393)
at com.zimbra.cs.backup.BackupAccountSession.startFul lBackup(BackupAccountSession.java:131)
at com.zimbra.cs.backup.BackupSet.startFullBackup(Bac kupSet.java:386)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSe t.startFullBackup(FileBackupTarget.java:954)
at com.zimbra.cs.backup.BackupManager.doBackupFull(Ba ckupManager.java:815)
at com.zimbra.cs.backup.BackupManager$FullBackupThrea d.run(BackupManager.java:222)
Caused by: com.mysql.jdbc.exceptions.MySQLNonTransientConnect ionException: Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1913 )
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:2357)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:2803)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:812)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1295)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.ja va:2262)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:4 39)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate (MysqlIO.java:1970)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java :1387)
at com.mysql.jdbc.ServerPreparedStatement.serverExecu te(ServerPreparedStatement.java:1186)
at com.mysql.jdbc.ServerPreparedStatement.executeInte rnal(ServerPreparedStatement.java:685)
at com.mysql.jdbc.PreparedStatement.executeQuery(Prep aredStatement.java:1262)
at org.apache.commons.dbcp.DelegatingPreparedStatemen t.executeQuery(DelegatingPreparedStatement.java:20 5)
at com.zimbra.cs.db.DbBackup.getBlobDigestMap(DbBacku p.java:210)
at com.zimbra.cs.backup.BackupAccountSession$BlobDige sts.preload(BackupAccountSession.java:500)
at com.zimbra.cs.backup.BackupAccountSession.loadBlob DigestMap(BackupAccountSession.java:463)
at com.zimbra.cs.backup.FileBackupTarget$BackupAcctSe ssion.storeBlobsBeforeMaintenanceMode(FileBackupTa rget.java:1393)
at com.zimbra.cs.backup.BackupAccountSession.startFul lBackup(BackupAccountSession.java:131)
at com.zimbra.cs.backup.BackupSet.startFullBackup(Bac kupSet.java:386)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSe t.startFullBackup(FileBackupTarget.java:954)
at com.zimbra.cs.backup.BackupManager.doBackupFull(Ba ckupManager.java:815)
at com.zimbra.cs.backup.BackupManager$FullBackupThrea d.run(BackupManager.java:222)


** END NESTED EXCEPTION **



Last packet sent to the server was 9163 ms ago.

Query being executed when exception was thrown:

com.mysql.jdbc.ServerPreparedStatement[24074] - SELECT id, mod_content, blob_digest FROM mboxgroup38.mail_item WHERE mailbox_id = 138
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:2515)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:2803)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:812)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1295)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.ja va:2262)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:4 39)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate (MysqlIO.java:1970)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java :1387)
at com.mysql.jdbc.ServerPreparedStatement.serverExecu te(ServerPreparedStatement.java:1186)
at com.mysql.jdbc.ServerPreparedStatement.executeInte rnal(ServerPreparedStatement.java:685)
at com.mysql.jdbc.PreparedStatement.executeQuery(Prep aredStatement.java:1262)
at org.apache.commons.dbcp.DelegatingPreparedStatemen t.executeQuery(DelegatingPreparedStatement.java:20 5)
at com.zimbra.cs.db.DbBackup.getBlobDigestMap(DbBacku p.java:210)
... 8 more

account2@myserver.com:: system failure: getting database connection
com.zimbra.common.service.ServiceException: system failure: getting database connection
ExceptionId:FullBackupThread:1277275676414:79106c4 9fc4e6bd1
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:251)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 234)
at com.zimbra.cs.backup.BackupAccountSession.startFul lBackup(BackupAccountSession.java:125)
at com.zimbra.cs.backup.BackupSet.startFullBackup(Bac kupSet.java:386)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSe t.startFullBackup(FileBackupTarget.java:954)
at com.zimbra.cs.backup.BackupManager.doBackupFull(Ba ckupManager.java:815)
at com.zimbra.cs.backup.BackupManager$FullBackupThrea d.run(BackupManager.java:222)
Caused by: com.mysql.jdbc.exceptions.MySQLNonTransientConnect ionException: Server connection failure during transaction. Due to underlying exception: 'java.net.SocketException: java.net.ConnectException: Connection refused'.

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: java.net.ConnectException: Connection refused

STACKTRACE:

java.net.SocketException: java.net.ConnectException: Connection refused
at com.mysql.jdbc.StandardSocketFactory.connect(Stand ardSocketFactory.java:156)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:276)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2769)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 531)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager .java:525)
at java.sql.DriverManager.getConnection(DriverManager .java:140)
at org.apache.commons.dbcp.DriverManagerConnectionFac tory.createConnection(DriverManagerConnectionFacto ry.java:97)
at org.apache.commons.dbcp.PoolableConnectionFactory. makeObject(PoolableConnectionFactory.java:300)
at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:140)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 225)
at com.zimbra.cs.backup.BackupAccountSession.startFul lBackup(BackupAccountSession.java:125)
at com.zimbra.cs.backup.BackupSet.startFullBackup(Bac kupSet.java:386)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSe t.startFullBackup(FileBackupTarget.java:954)
at com.zimbra.cs.backup.BackupManager.doBackupFull(Ba ckupManager.java:815)
at com.zimbra.cs.backup.BackupManager$FullBackupThrea d.run(BackupManager.java:222)


** END NESTED EXCEPTION **


Attempted reconnect 3 times. Giving up.
at com.mysql.jdbc.SQLError.createSQLException(SQLErro r.java:888)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2838)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 531)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager .java:525)
at java.sql.DriverManager.getConnection(DriverManager .java:140)
at org.apache.commons.dbcp.DriverManagerConnectionFac tory.createConnection(DriverManagerConnectionFacto ry.java:97)
at org.apache.commons.dbcp.PoolableConnectionFactory. makeObject(PoolableConnectionFactory.java:300)
at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:140)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 225)
... 5 more


account3@myserver.com:: system failure: getting database connection
com.zimbra.common.service.ServiceException: system failure: getting database connection
ExceptionId:FullBackupThread:1277275684425:79106c4 9fc4e6bd1
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:251)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 234)
at com.zimbra.cs.backup.BackupAccountSession.startFul lBackup(BackupAccountSession.java:125)
at com.zimbra.cs.backup.BackupSet.startFullBackup(Bac kupSet.java:386)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSe t.startFullBackup(FileBackupTarget.java:954)
at com.zimbra.cs.backup.BackupManager.doBackupFull(Ba ckupManager.java:815)
at com.zimbra.cs.backup.BackupManager$FullBackupThrea d.run(BackupManager.java:222)
Caused by: com.mysql.jdbc.exceptions.MySQLNonTransientConnect ionException: Server connection failure during transaction. Due to underlying exception: 'java.net.SocketException: java.net.ConnectException: Connection refused'.

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: java.net.ConnectException: Connection refused

STACKTRACE:

java.net.SocketException: java.net.ConnectException: Connection refused
at com.mysql.jdbc.StandardSocketFactory.connect(Stand ardSocketFactory.java:156)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:276)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2769)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 531)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager .java:525)
at java.sql.DriverManager.getConnection(DriverManager .java:140)
at org.apache.commons.dbcp.DriverManagerConnectionFac tory.createConnection(DriverManagerConnectionFacto ry.java:97)
at org.apache.commons.dbcp.PoolableConnectionFactory. makeObject(PoolableConnectionFactory.java:300)
at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:140)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 225)
at com.zimbra.cs.backup.BackupAccountSession.startFul lBackup(BackupAccountSession.java:125)
at com.zimbra.cs.backup.BackupSet.startFullBackup(Bac kupSet.java:386)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSe t.startFullBackup(FileBackupTarget.java:954)
at com.zimbra.cs.backup.BackupManager.doBackupFull(Ba ckupManager.java:815)
at com.zimbra.cs.backup.BackupManager$FullBackupThrea d.run(BackupManager.java:222)


** END NESTED EXCEPTION **


Attempted reconnect 3 times. Giving up.
at com.mysql.jdbc.SQLError.createSQLException(SQLErro r.java:888)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2838)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 531)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager .java:525)
at java.sql.DriverManager.getConnection(DriverManager .java:140)
at org.apache.commons.dbcp.DriverManagerConnectionFac tory.createConnection(DriverManagerConnectionFacto ry.java:97)
at org.apache.commons.dbcp.PoolableConnectionFactory. makeObject(PoolableConnectionFactory.java:300)
at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:140)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 225)
... 5 more

[phoenix]

What's the status of the services?

zmcontrol status

[the_griz]

They are all running fine.

[zimbra@mail ~]$ zmcontrol status
Host myserver.com
antispam Running
antivirus Running
imapproxy Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running
stats Running

Thanks!
Tom

[dik23]

Did you ever find out what the problem was here ?

[the_griz]

This did get resolved but it took awhile. I had to submit two paid support cases on this before it became stable again.

The database had to be rebuilt. There is a wiki article on how to do this. A couple of accounts still did not backup up correctly and in fact two were completely lost - I had to recover them from local Outlook PST files.

There were some follow on errors that required support and these were eventually diagnosed and either disappeared or were tweaked away. Unforutnately, outside of rebuilding the mysql database, there was no real magic bullet for my type of issue.

Good Luck!
Tom