[SOLVED] 32 to 64 bit Failure! Please help!!!

[robncar]

This installation is for Student Email at a school district. The goal is to get to v7.1 in the end. We followed the WIKI article but are now unable to run on the 64-bit server.

Old system:

Release 5.0.26_GA_3365.RHEL5_20101215150457 RHEL5

New System:

Release 5.0.26_GA_3365.RHEL5_64_20101215150659 RHEL5_64 FOSS edition

[zimbra@cusd-zcs01 ~]$ zmcontrol status
Host stumail.xxxx.xxxxxxxxx.edu
antispam Running
antivirus Running
ldap Running
logger Stopped
logmysql.server is not running
zmlogswatchctl is not running
mailbox Stopped
mysql.server is not running.
mta Running
snmp Running
spell Running
stats Stopped

[phoenix]

Posting the status of the services does not provide any information which would allow anyone to help with your problem. Which of the forum threads that cover this type of problem have you reviewed? What errors do you see in the log files?

[robncar]

Sorry, I've attempted to read as much as I could, been working on this for a couple days now.

I think it may be auth related to mysql?

Here's a snip from the <mailbox.log>...

2011-04-13 06:18:51,687 WARN [main] [] misc - Could not establish a connection to the database. Retrying in 5 seconds.
com.zimbra.common.service.ServiceException: system failure: getting database connection
ExceptionId:main:1302700731687:809b8dec0b867579
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:251)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 234)
at com.zimbra.cs.util.Zimbra.waitForDatabase(Zimbra.j ava:122)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:163)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:137)
at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 119)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:241)
at org.mortbay.jetty.servlet.ServletHolder.initServle t(ServletHolder.java:433)
at org.mortbay.jetty.servlet.ServletHolder.doStart(Se rvletHolder.java:256)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initializ e(ServletHandler.java:636)
at org.mortbay.jetty.servlet.Context.startContext(Con text.java:139)
at org.mortbay.jetty.webapp.WebAppContext.startContex t(WebAppContext.java:1222)
at org.mortbay.jetty.handler.ContextHandler.doStart(C ontextHandler.java:517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(Web AppContext.java:449)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection .doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:152)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.setuid.SetUIDServer.doStart(SetUIDServ er.java:120)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:985)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at org.mortbay.start.Main.invokeMain(Main.java:183)
at org.mortbay.start.Main.start(Main.java:497)
at org.mortbay.start.Main.main(Main.java:115)
Caused by: com.mysql.jdbc.exceptions.MySQLNonTransientConnect ionException: Server connection failure during transaction. Due to underlying exception: 'java.net.SocketException: java.net.ConnectException: Connection refused'.

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: java.net.ConnectException: Connection refused

STACKTRACE:

java.net.SocketException: java.net.ConnectException: Connection refused
at com.mysql.jdbc.StandardSocketFactory.connect(Stand ardSocketFactory.java:156)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:276)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2769)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 531)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager .java:525)
at java.sql.DriverManager.getConnection(DriverManager .java:140)
at org.apache.commons.dbcp.DriverManagerConnectionFac tory.createConnection(DriverManagerConnectionFacto ry.java:97)
at org.apache.commons.dbcp.PoolableConnectionFactory. makeObject(PoolableConnectionFactory.java:300)
at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:140)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 225)
at com.zimbra.cs.util.Zimbra.waitForDatabase(Zimbra.j ava:122)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:163)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:137)
at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 119)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:241)
at org.mortbay.jetty.servlet.ServletHolder.initServle t(ServletHolder.java:433)
at org.mortbay.jetty.servlet.ServletHolder.doStart(Se rvletHolder.java:256)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initializ e(ServletHandler.java:636)
at org.mortbay.jetty.servlet.Context.startContext(Con text.java:139)
at org.mortbay.jetty.webapp.WebAppContext.startContex t(WebAppContext.java:1222)
at org.mortbay.jetty.handler.ContextHandler.doStart(C ontextHandler.java:517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(Web AppContext.java:449)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection .doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:152)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.setuid.SetUIDServer.doStart(SetUIDServ er.java:120)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:985)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at org.mortbay.start.Main.invokeMain(Main.java:183)
at org.mortbay.start.Main.start(Main.java:497)
at org.mortbay.start.Main.main(Main.java:115)


** END NESTED EXCEPTION **


Attempted reconnect 3 times. Giving up.
at com.mysql.jdbc.SQLError.createSQLException(SQLErro r.java:888)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2838)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 531)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager .java:525)
at java.sql.DriverManager.getConnection(DriverManager .java:140)
at org.apache.commons.dbcp.DriverManagerConnectionFac tory.createConnection(DriverManagerConnectionFacto ry.java:97)
at org.apache.commons.dbcp.PoolableConnectionFactory. makeObject(PoolableConnectionFactory.java:300)
at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:140)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 225)
... 33 more

# # #

Please let me know if there is another log that will assist.

- Rob.

[robncar]

I found a typo in one of the password entries in <localconfig.xml>. Fixed it and verified with <zmlocalconfig -s>. Restart services and now:

[zimbra@cusd-zcs01 conf]$ zmcontrol status

Host stumail.cusd.claremont.edu
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Stopped

mysql.server is not running.

mta Running
snmp Running
spell Running
stats Stopped

Progress but no cigar.

[robncar]

Well now, I'll surely need assistance with this! <mysql_error.log>

# # #

110413 9:09:35
InnoDB: Page checksum 275536925, prior-to-4.0.14-form checksum 2123801932
InnoDB: stored checksum 10240036, prior-to-4.0.14-form stored checksum 67110015
InnoDB: Page lsn 2229830113 196611, low 4 bytes of lsn at page end 1993062020
InnoDB: Page number (if stored to page already) 33554447,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 3900823040
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 7.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: MySQL :: MySQL 5.0 Reference Manual :: 13.2.6.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
110413 09:09:35 mysqld ended

# # #

[robncar]

Well...

Decided to revert to a pre-5.0.26 (5.0.14) snapshot of the 32-bit host. The <zmdbintegrityreport> ran clean for both mysql and logger on the snapshot. Was then able to upgrade 32-bit host to 5.0.26 cleanly. Will now attempt move to 64-bit 5.0.26 host again, now that the db is clean from the start of the process.

[robncar]

The migration to the 64-bit host went very well. Funny how a clean DB can make such a difference.

Just took a post 5.0.26 64-bit snapshot. Starting 7.1 upgrade now...

[jiqiren]

I found the steps on 32-bit_to_64 were a bit incomplete.

I had to run /opt/zimbra/libexec/zmfixperms -extended instead of just /opt/zimbra/libexec/zmfixperms.

I also had to repush/deploy the certs.

[robncar]

Yep, a tad incomplete. I supplemented with:

Talk:Network Edition: Moving from 32-bit to 64-bit Server - Zimbra :: Wiki

I also had to redeploy the cert(s) with:

</opt/zimbra/bin/zmcertmgr deploycrt comm /tmp/example.ca.crt /tmp/IPS-IPSCABUNDLE.crt>

But, in the end, the goal has been realized:

[zimbra@xxxx-zcs01 ~]$ zmcontrol -v
Release 7.1.0_GA_3140.RHEL5_64_20110329150833 RHEL5_64 FOSS edition.

I'll likely keep my job another day...