A network service error has occurred

[bjared]

Long story short: I was in the middle of upgrading from ZCS (NE) 4.0.1 to 4.0.2, when the rpm install process of the install.sh script was working on the first RPM, I lost my ssh connection.

I logged back in to see that it only had 2 of the 8 RPMs installed. This is when my fun began.

I had done a "zmbackup -a all" beforehand, so I had data to restore from, but it wasn't going well for the LONGEST time, until I found 'zmrestoreldap'. Now, I'm so close (after 13 hours now) to getting zimbra back and running, but hit another wall.

Here's the error I'm getting

msg - system failure: java.lang.NullPointerException
com.zimbra.cs.service.ServiceException: system failure: java.lang.NullPointerException
	at com.zimbra.cs.service.ServiceException.FAILURE(ServiceException.java:174)
	at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:229)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
	at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:173)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:667)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
	at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.NullPointerException
	at com.zimbra.cs.service.UserServlet.getRestUrl(UserServlet.java:157)
	at com.zimbra.cs.service.mail.ToXML.encodeRestUrl(ToXML.java:1355)
	at com.zimbra.cs.service.mail.ToXML.encodeFolderCommon(ToXML.java:217)
	at com.zimbra.cs.service.mail.ToXML.encodeFolder(ToXML.java:145)
	at com.zimbra.cs.service.mail.ToXML.encodeFolder(ToXML.java:136)
	at com.zimbra.cs.service.mail.GetFolder.encodeFolderNode(GetFolder.java:91)
	at com.zimbra.cs.service.mail.GetFolder.encodeFolderNode(GetFolder.java:94)
	at com.zimbra.cs.session.SoapSession.putRefresh(SoapSession.java:273)
	at com.zimbra.soap.ZimbraSoapContext.generateResponseHeader(ZimbraSoapContext.java:472)
	at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:195)
	at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:84)
	at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:223)
	... 18 more

code - service.FAILURE
method - ZmCsfeCommand.prototype.invoke
detail - soap:Receiver

The steps I took to get here were:

1. uninstalling all zimbra RPMs
2. removed /opt/zimbra
3. installed zcs 4.0.1
4. zmcontrol stop
5. zmrestoreldap -lb full-20061011.170501.208 -t /opt/zimbra-backup/backup
6. zmcontrol start
7. (one by one, I set each account to "maintenance")
8. zmlicense -i /opt/zimbra-backup/conf/ZCSLicense.xml
9. (one by one, I restored each account)
zmrestore -a user@domain.com -lb full-20061011.170501.208 -t /opt/zimbra-backup/backup
10. zmcontrol stop
11. zmrestoreoffline -sys -lb full-20061011.170501.208 -t /opt/zimbra-backup/backup
12. I set one account to "active" and tried to login. I get "A network service error has occurred."

The zmrestore -sys helped with mysql and the logger and tomcat not starting, now everything is running, but users can't login. The admin UI works fine, and I get the same "network service" error when the admin clicks on "view mail".

Any ideas?

[bjared]

OK, This morning I started working on this again. I decided to poke around the admin UI and check settings, and I noticed the domain for the user I was testing was missing. I didn't delete it, but the restore must not have recovered it properly. I added it, and it complained that it already existed, but a refresh of the domain list showed it there.

And now... I can login.

Once I'm happy that it's working correctly, I'm going to try another upgrade to 4.0.2, but this time...I'm running in 'screen'.

--Brian

[jonnyRo]

I am very sorry for the issues you are having. Just a tip for the future, use GNU screen when doing important remote admin work.

It's a bit tricky to learn at first, but it's better than having long scripts crash if your connection gets pulled. If you need any help using it please let me know.

[someone]

To prevent a problem like this, use the screen program. If your connection disconnects you can easily just do a 'screen -ls' to see what screen's are available and just re-attach. Then you won't have all the problems you are having due to disconnection and your wife and kid won't be so mad at you for staying out so late.

[bjared]

OK, I had it working in 4.0.1, did a backup with "zmbackup -a all -t /opt/zimbra/mybackup" and then rsynced the entire /opt/zimbra directory to /opt/zimbra-20061012-good by:

mkdir /opt/zimbra-20061012-good
rsync -av /opt/zimbra/ /opt/zimbra-20061012-good

I performed an upgrade, and I got an error, but screen kills my screen history, so I ran the install again, but with the "-r" option and luckily got the same error so I could paste it here:

This platform is CentOS4
Packages found: zimbra-core-4.0.2_GA_362.RHEL4-20060922120643.i386.rpm
This may or may not work


Install anyway? [N] y

The system will be modified.  Continue? [N] y

Shutting down zimbra mail

Backing up ldap


Removing existing packages

   zimbra-ldap...done
   zimbra-logger...done
   zimbra-mta...done
   zimbra-snmp...done
   zimbra-store...done
   zimbra-spell...done
   zimbra-apache...done
   zimbra-core...done

Removing deployed webapp directories
Installing packages

    zimbra-core......zimbra-core-4.0.2_GA_362.RHEL4-20060922120643.i386.rpm...done
    zimbra-ldap......zimbra-ldap-4.0.2_GA_362.RHEL4-20060922120643.i386.rpm...done
    zimbra-logger......zimbra-logger-4.0.2_GA_362.RHEL4-20060922120643.i386.rpm...done
    zimbra-mta......zimbra-mta-4.0.2_GA_362.RHEL4-20060922120643.i386.rpm...done
    zimbra-snmp......zimbra-snmp-4.0.2_GA_362.RHEL4-20060922120643.i386.rpm...done
    zimbra-store......zimbra-store-4.0.2_GA_362.RHEL4-20060922120643.i386.rpm...done
    zimbra-apache......zimbra-apache-4.0.2_GA_362.RHEL4-20060922120643.i386.rpm...done
    zimbra-spell......zimbra-spell-4.0.2_GA_362.RHEL4-20060922120643.i386.rpm...done

Setting defaults from saved config in /opt/zimbra/.saveconfig/config.save
   HOSTNAME=zimbra1.domain.com
   LDAPHOST=zimbra1.domain.com
   LDAPPORT=389
   SNMPTRAPHOST=zimbra1.domain.com
   SMTPSOURCE=admin@zimbra1.domain.com
   SMTPDEST=admin@zimbra1.domain.com
   SNMPNOTIFY=yes
   SMTPNOTIFY=yes
   LDAPROOTPW=XXXXXXXXX
   LDAPZIMBRAPW=XXXXXXXX
Restoring existing configuration file from /opt/zimbra/.saveconfig/config.save...Restoring backup schedule...done
Operations logged to /tmp/zmsetup.log.32544
Setting defaults...Setting defaults from existing config...Done
Upgrading from 4.0.1_GA_324 to 4.0.2_GA_362
Thu Oct 12 12:56:12 2006: Stopping zimbra services
Thu Oct 12 12:56:51 2006: Starting mysql
Thu Oct 12 12:57:10 2006: SELECT value FROM config WHERE name = 'db.version'
/opt/zimbra/bin/mysql: line 30: /root/zcs-4.0.2/${zimbra_home}/mysql/bin/mysql: No such file or directory
/opt/zimbra/bin/mysql: line 30: exec: /root/zcs-4.0.2/${zimbra_home}/mysql/bin/mysql: cannot execute: No such file or directory
Error while running '/opt/zimbra/bin/mysql --user=zimbra --password=thisIsNotARealPasswordSoThereYouGo --database=zimbra --batch --skip-column-names'. at /opt/zimbra/libexec/scripts/Migrate.pm line 134.

Now, my concern is why it's trying to run mysql from "/root/zcs-4.0.2/${zimbra_home}/mysql/bin/mysql"

/root/zcs-4.0.2 is the unpacked root directory of the 4.0.2. tarball.
It should be trying to run ${zimbra_home}/mysql/bin/mysql, but it also doesn't appear to be replacing the variable with its contents. Now that I've pasted this here, I'm going to see what I can find. I saw criley's mention of running zmsetup.pl by hand with the "-c" option, so I'm doing that...

...and it seems to be doing well! (/em looks for some wood to knock on...)

Hopefully this is almost done. I didn't see the issue where there were missing "CONFIGURED END" entries in /opt/zimbra/.install_history, so that was nice.

Well...even though the second time it bombed it showed 2 of 500 accounts (there should have been over 10) they are all there, and everything looks good. Now, I wait for a couple of users (one uses Outlook, other the web interface) and see if it looks good. What a saga. e-mail down for over 24 hours. (only a small percentage of all users have been migrated to zimbra so far) The others are still in Lotus Notes world. (yuck)

Well, I hope this is helpful to someone. I'm more than happy to share my blood and sweat.

--Brian