8.0.1 upgrade issue -- user accounts gone!

[undersys]

Hi all,

I am having issues upgrading to 8.0.1 from 7.2.0 on Centos.

Last night I attempted an upgrade (did a backup first).

Upgrade looked like it worked , until i checked the upgrade log file. I noticed this error :-

50ae132e ldif_read_file: checksum error on "/opt/zimbra/data/ldap/config/cn=config/olcDatabase={2}mdb.ldif"

Sigh.. So i take a look at this bug report https://bugzilla.zimbra.com/show_bug.cgi?id=77623
I run the following steps :-

cd /opt/zimbra/data/ldap
mv mdb mdb.old
mkdir -p mdb/db
/opt/zimbra/openldap/sbin/slapadd -q -b '' -F /opt/zimbra/data/ldap/config -l /opt/zimbra/data/ldap/ldap.80

So I then browse to my mail servers address and it all looks good. zmstatus confirms this.

Can not log in with any user account.
I notice this error in my mailbox log:-

com.zimbra.common.service.ServiceException: permission denied: can not access account xxxxxxxxxxxxxx
ExceptionId:qtp2134196695-103:https://xxx.xxx.xxx.xxx:443/service/soap/NoOpRequest:1353587457605:fc50505c50181447
Code:service.PERM_DENIED
        at com.zimbra.common.service.ServiceException.PERM_DENIED(ServiceException.java:288)
        at com.zimbra.common.service.ServiceException.DEFEND_ACCOUNT_HARVEST(ServiceException.java:305)
        at com.zimbra.soap.ZimbraSoapContext.<init>(ZimbraSoapContext.java:306)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:198)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:157)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:290)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:206)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:208)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1361)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:57)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1332)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:77)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:181)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1332)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:464)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:327)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1332)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:312)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:77)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.eclipse.jetty.server.Server.handle(Server.java:349)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
        at java.lang.Thread.run(Thread.java:722)
2012-11-22 20:31:14,129 WARN  [ScheduledTask-3] [mid=16;] mailbox - no account found in directory for mailbox 16 (was expecting 11aee3dc-3fdf-4346-ae46-03f59a2d58a2)
2012-11-22 20:31:14,130 WARN  [ScheduledTask-3] [mid=16;] mailbox - no account found in directory for mailbox 16 (was expecting 11aee3dc-3fdf-4346-ae46-03f59a2d58a2)
2012-11-22 20:31:14,130 WARN  [ScheduledTask-3] [mid=16;] datasource - Scheduled DataSource import failed.
com.zimbra.cs.account.AccountServiceException: no such account: 11aee3dc-3fdf-4346-ae46-03f59a2d58a2
ExceptionId:ScheduledTask-3:1353587474130:fc50505c50181447
Code:account.NO_SUCH_ACCOUNT
        at com.zimbra.cs.account.AccountServiceException.NO_SUCH_ACCOUNT(AccountServiceException.java:200)
        at com.zimbra.cs.mailbox.Mailbox.getAccount(Mailbox.java:794)
        at com.zimbra.cs.mailbox.MailboxManager.getMailboxById(MailboxManager.java:458)
        at com.zimbra.cs.mailbox.MailboxManager.getMailboxById(MailboxManager.java:385)
        at com.zimbra.cs.datasource.DataSourceTask.call(DataSourceTask.java:67)
        at com.zimbra.cs.datasource.DataSourceTask.call(DataSourceTask.java:29)
        at com.zimbra.common.util.TaskScheduler$TaskRunner.call(TaskScheduler.java:67)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2012-11-22 20:31:16,213 INFO  [qtp2134196695-104:https://xxx.xxx.xxx.xxx:7071/service/admin/soap/AuthRequest] [name=zimbra;ip=xxx.xxx.xxx.xxx;ua=zmprov/8.0.1_GA_5438;] soap - AuthRequest elapsed=3
2012-11-22 20:31:21,744 INFO  [qtp2134196695-104:https://xxx.xxx.xxx.xxx:7071/service/admin/soap/GetAccountRequest] [name=zimbra;ip=xxx.xxx.xxx.xxx;ua=zmprov/8.0.1_GA_5438;] SoapEngine - handler exception

Note I have replaced my public IP and the user names in the above snippet. This error happened for every one of my accounts. OUCH!

Lucky for me I rolled back my backup and its all happy back at 7.2.0.

Can any one shed some light on this issue? Happy to provide more info if needed.

Thanks

[undersys]

I also had a cert error, but I use a commercial cert so this seems ok.

Thu Nov 22 20:10:29 2012 *** Running as root user: /opt/zimbra/openssl/bin/openssl verify -purpose sslserver -CAfile /opt/zimbra/conf/ca/ca.pem /opt/
zimbra/conf/ca/ca.pem | egrep "^error 10"
error 10 at 0 depth lookup:certificate has expired
Thu Nov 22 20:10:29 2012 *** Running as root user: /opt/zimbra/bin/zmcertmgr createca -new
** Creating directory /opt/zimbra/ssl/zimbra/ca
** Creating directory /opt/zimbra/ssl/zimbra/server
** Creating directory /opt/zimbra/mailboxd/webapps/zimbraAdmin/tmp
** Creating /opt/zimbra/ssl/zimbra/ca/zmssl.cnf...done
** Creating CA private key /opt/zimbra/ssl/zimbra/ca/ca.key...done.
Thu Nov 22 20:10:30 2012 done.

[undersys]

I can't seem to find any logic behind this chain of events. Apart from the operator maybe

I think just restoring from a backup and reattempting the upgrade will be my only option.
I think the issue may of been that I did not re run the installer after the SLAPD command failed.

From my limited understanding I should of done the following :-
1) install .sh --platform-override
2) IF SLAPD failes :-
cd /opt/zimbra/data/ldap
mv mdb mdb.old
mkdir -p mdb/db
/opt/zimbra/openldap/sbin/slapadd -q -b '' -F /opt/zimbra/data/ldap/config -l /opt/zimbra/data/ldap/ldap.80

3) re run install.sh

I am having my doubts about step 3

[adamhatak]

Hi,

Did you find any soultion for that? I have similar issue - can't see user accounts in Admin Console.

2013-03-03 07:53:31,387 INFO [ScheduledTask-10] [name=galsync@zimbra.xxxxxxx;mid=355;ds=zimbra;] datasource - Requested import.
2013-03-03 07:53:31,387 INFO [ScheduledTask-10] [name=galsync@zimbra.xxxxxxx;mid=355;ds=zimbra;] datasource - Importing data for data source 'zimbra'
2013-03-03 07:53:31,390 WARN [ScheduledTask-10] [name=galsync@zimbra.xxxxxxx;mid=355;ds=zimbra;] ldap - unknown GAL op
2013-03-03 07:53:31,394 INFO [ScheduledTask-10] [name=galsync@zimbra.xxxxxxx;mid=355;ds=zimbra;] datasource - Import completed for data source 'zimbra'


Regards,
Adam

[undersys]

Hi Adam,

Nope I rolled back.

[adamhatak]

@undersys,
Thanks for reply.

I have managed to resolve this issue by deploying self-signed cert (even if I used commercial one). After that it I saw all accounts in GAL, in Admin Console etc...
In the end self certificate was also deployed to website, so I had to deploy once again the commercial one. After that HTTPS usues comm one and all data is correct and valid.

Regards,

Adam

[undersys]

Thanks Adam.

Can you detail the steps you used, encase someone (like myself ) hits the same issue ?

thanks

[adamhatak]

@undersys,

It will be difficult, while I was doing a lot of things (trying one after another)... Basically I used this:

/opt/zimbra/bin/zmcertmgr deploycrt self

Next zmcontrol stop && zmcontrol start

Wait a while and see if accounts are available in Administration Console. In my case they appeared then.
Because I use commercial cert i had to re-deploy it again, after deploying self cert (probably self cert overrode comm one).

/opt/zimbra/bin/zmcertmgr deploycrt comm /tmp/commercial.crt /tmp/ca_chain.crt
where /tmp/commercial.crt and /tmp/ca_chain.crt are proper certificate files.

After that: zmcontrol stop && zmcontrol start

This helped me. Remember about one thing: backups, backups, backups! Think before you submit every command - you may hurt yourself.


Regards,

Adam