Results 1 to 9 of 9

Thread: [SOLVED] Problem with upgrade from 4.5.4 to 5.0.6 - Error 503 SERVICE_UNAVAILABLE

  1. #1
    Myrddin is offline Intermediate Member
    Join Date
    Aug 2007
    Location
    Stamford, CT USA
    Posts
    16
    Rep Power
    8

    Default [SOLVED] Problem with upgrade from 4.5.4 to 5.0.6 - Error 503 SERVICE_UNAVAILABLE

    I am using zcs-NETWORK-5.0.6_GA_2313.RHEL5_64.20080522105815 and just completed an upgrade from zcs-NETWORK-4.5.4-20070418.RHEL4_64. I'm on a RHEL4 AMD64 platform.

    The upgrade appeared to go smoothly. All the packages installed fine and zmsetup.pl completed with no major problem.

    I get the same error when I run zmprov, try to deliver mail to port 7025, and when I try to login to the user interface, or the Ajax scripts that run on the admin interface login get the same error:

    <html><head><meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"/><title>Error 503 SERVICE_UNAVAILABLE</title></head><body><h2>HTTP ERROR: 503</h2><pre>SERVICE_UNAVAILABLE</pre><p>RequestURI=/service/admin/soap/</p><p><i><small><a href="http://jetty.mortbay.org">Powered by jetty://</a></small></i></p></body></html>, server: zimbra.domain.com) (cause: com.zimbra.common.soap.SoapParseException cannot determine soap protocol in reply: <html><head><meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"/><title>Error 503 SERVICE_UNAVAILABLE</title></head><body><h2>HTTP ERROR: 503</h2><pre>SERVICE_UNAVAILABLE</pre><p>RequestURI=/service/admin/soap/</p><p><i><small><a href="http://jetty.mortbay.org">Powered by jetty://</a></small></i></p></body></html>

    Basically any /service request seems to be broken.

    My mail server has been down for 4 days now trying to get this upgrade completely. This is really a complete disaster. Any quick assistance would be enormously appreciated. Thank you!

    Lastly, here's an exerpt from zmmailboxd.out that I thought might be relevant (note, I turned on debug logging for jetty so it's quite verbose):

    CompilerOracle: exclude com/zimbra/cs/session/SessionMap.putAndPrune
    CompilerOracle: exclude com/zimbra/cs/mailbox/MailItem.delete
    0 INFO [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    Zimbra server reserving server socket port=7995 bindaddr=5.5.5.5 ssl=true
    Zimbra server reserving server socket port=7143 bindaddr=5.5.5.5 ssl=false
    Zimbra server reserving server socket port=7993 bindaddr=5.5.5.5 ssl=true
    Zimbra server reserving server socket port=7025 bindaddr=5.5.5.5 ssl=false
    3065 INFO [main] log - Setting umask=027
    3067 INFO [main] log - Setting GID=269
    3067 INFO [main] log - Setting UID=269
    3067 INFO [main] log - jetty-6.1.5
    3544 INFO [main] log - No Transaction manager found - if your webapp requires one, please configure one.
    Zimbra server process is running as uid=269 euid=269 gid=269 egid=269
    { empty }
    { empty }
    { empty }
    Listening on port 0.0.0.0/0.0.0.0:7335
    Listening on port 0.0.0.0/0.0.0.0:5222
    Listening on port 0.0.0.0/0.0.0.0:5269
    Wildfire 3.1.0 [Jun 5, 2008 12:31:23 AM]
    java.lang.IllegalStateException: A server is already running
    at org.jivesoftware.wildfire.XMPPServer.<init>(XMPPSe rver.java:150)
    at com.zimbra.cs.im.ZimbraIM.startup(ZimbraIM.java:77 )
    at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:196)
    at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:129)
    at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 114)
    at javax.servlet.GenericServlet.init(GenericServlet.j ava:241)
    at org.mortbay.jetty.servlet.ServletHolder.initServle t(ServletHolder.java:433)
    at org.mortbay.jetty.servlet.ServletHolder.doStart(Se rvletHolder.java:256)
    at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
    at org.mortbay.jetty.servlet.ServletHandler.initializ e(ServletHandler.java:612)
    at org.mortbay.jetty.servlet.Context.startContext(Con text.java:139)
    at org.mortbay.jetty.webapp.WebAppContext.startContex t(WebAppContext.java:1222)
    at org.mortbay.jetty.handler.ContextHandler.doStart(C ontextHandler.java:501)
    at org.mortbay.jetty.webapp.WebAppContext.doStart(Web AppContext.java:449)
    at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
    at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:147)
    at org.mortbay.jetty.handler.ContextHandlerCollection .doStart(ContextHandlerCollection.java:161)
    at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
    at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:147)
    at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
    at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:117)
    at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
    at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:117)
    at org.mortbay.jetty.Server.doStart(Server.java:217)
    at org.mortbay.setuid.SetUIDServer.doStart(SetUIDServ er.java:100)
    at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
    at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:979)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.mortbay.start.Main.invokeMain(Main.java:183)
    at org.mortbay.start.Main.start(Main.java:497)
    at org.mortbay.start.Main.main(Main.java:115)
    com.zimbra.cs.zclient.ZClientException: invoke cannot determine soap protocol in reply: <html><head><meta content="text/html; charset
    =ISO-8859-1" http-equiv="Content-Type"/><title>Error 503 SERVICE_UNAVAILABLE</title></head><body><h2>HTTP ERROR: 503</h2><pre>SERVICE
    _UNAVAILABLE</pre><p>RequestURI=/service/admin/soap/</p><p><i><small><a href="http://jetty.mortbay.org">Powered by jetty://</a></smal
    l></i></p></body></html>, server: zimbra.domain.com
    ExceptionId:btpool0-16:1212640344917:367b0e7f4c1ed8a1
    Code:zclient.IO_ERROR
    at com.zimbra.cs.zclient.ZClientException.IO_ERROR(ZC lientException.java:47)
    at com.zimbra.cs.account.soap.SoapProvisioning.invoke (SoapProvisioning.java:178)
    at com.zimbra.cs.account.soap.SoapProvisioning.getDom ainInfo(SoapProvisioning.java:817)
    at com.zimbra.cs.taglib.tag.GetDomainInfoTag.getInfo( GetDomainInfoTag.java:90)
    at com.zimbra.cs.taglib.tag.GetDomainInfoTag.checkCac he(GetDomainInfoTag.java:77)
    at com.zimbra.cs.taglib.tag.GetDomainInfoTag.doTag(Ge tDomainInfoTag.java:66)
    at org.apache.jsp.public_.login_jsp._jspService(org.a pache.jsp.public_.login_jsp:421)
    at org.apache.jasper.runtime.HttpJspBase.service(Http JspBase.java:93)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
    at org.apache.jasper.servlet.JspServletWrapper.servic e(JspServletWrapper.java:373)
    at org.apache.jasper.servlet.JspServlet.serviceJspFil e(JspServlet.java:477)
    at org.apache.jasper.servlet.JspServlet.service(JspSe rvlet.java:371)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
    at com.zimbra.webClient.servlet.JspServlet.service(Js pServlet.java:50)
    at org.mortbay.jetty.servlet.ServletHolder.handle(Ser vletHolder.java:487)
    at org.mortbay.jetty.servlet.ServletHandler.handle(Se rvletHandler.java:362)
    at org.mortbay.jetty.security.SecurityHandler.handle( SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(Se ssionHandler.java:181)
    at org.mortbay.jetty.handler.ContextHandler.handle(Co ntextHandler.java:716)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebA ppContext.java:406)
    at org.mortbay.jetty.servlet.Dispatcher.forward(Dispa tcher.java:268)
    at org.mortbay.jetty.servlet.Dispatcher.forward(Dispa tcher.java:126)
    at org.mortbay.jetty.servlet.DefaultServlet.doGet(Def aultServlet.java:465)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
    at org.mortbay.jetty.servlet.ServletHolder.handle(Ser vletHolder.java:487)
    at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1093)
    at com.zimbra.webClient.filters.SetHeaderFilter.doFil ter(SetHeaderFilter.java:283)
    at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1084)
    at org.mortbay.servlet.UserAgentFilter.doFilter(UserA gentFilter.java:81)
    at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter .java:132)
    at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1084)
    at org.mortbay.jetty.servlet.ServletHandler.handle(Se rvletHandler.java:360)
    at org.mortbay.jetty.security.SecurityHandler.handle( SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(Se ssionHandler.java:181)
    at org.mortbay.jetty.handler.ContextHandler.handle(Co ntextHandler.java:716)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebA ppContext.java:406)
    at org.mortbay.jetty.handler.ContextHandlerCollection .handle(ContextHandlerCollection.java:211)
    at org.mortbay.jetty.handler.HandlerCollection.handle (HandlerCollection.java:114)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(Ha ndlerWrapper.java:139)
    at org.mortbay.jetty.handler.rewrite.RewriteHandler.h andle(RewriteHandler.java:350)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(Ha ndlerWrapper.java:139)
    at org.mortbay.jetty.Server.handle(Server.java:313)
    at org.mortbay.jetty.HttpConnection.handleRequest(Htt pConnection.java:506)
    at org.mortbay.jetty.HttpConnection$RequestHandler.he aderComplete(HttpConnection.java:830)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser. java:514)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpPa rser.java:211)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnec tion.java:381)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(Selec tChannelEndPoint.java:396)
    at org.mortbay.thread.BoundedThreadPool$PoolThread.ru n(BoundedThreadPool.java:442)
    Caused by: com.zimbra.common.soap.SoapParseException: cannot determine soap protocol in reply: <html><head><meta content="text/html;
    charset=ISO-8859-1" http-equiv="Content-Type"/><title>Error 503 SERVICE_UNAVAILABLE</title></head><body><h2>HTTP ERROR: 503</h2><pre>
    SERVICE_UNAVAILABLE</pre><p>RequestURI=/service/admin/soap/</p><p><i><small><a href="http://jetty.mortbay.org">Powered by jetty://</a
    ></small></i></p></body></html>
    at com.zimbra.common.soap.SoapTransport.extractBodyEl ement(SoapTransport.java:228)
    at com.zimbra.common.soap.SoapTransport.parseSoapResp onse(SoapTransport.java:221)
    at com.zimbra.common.soap.SoapHttpTransport.invoke(So apHttpTransport.java:251)
    at com.zimbra.common.soap.SoapTransport.invoke(SoapTr ansport.java:297)
    at com.zimbra.common.soap.SoapTransport.invoke(SoapTr ansport.java:261)
    at com.zimbra.cs.account.soap.SoapProvisioning.invoke (SoapProvisioning.java:174)
    ... 48 more
    Last edited by Myrddin; 06-04-2008 at 09:52 PM.

  2. #2
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Quote Originally Posted by Myrddin View Post
    I am using zcs-NETWORK-5.0.6_GA_2313.RHEL5_64.20080522105815 and just completed an upgrade from zcs-NETWORK-4.5.4-20070418.RHEL4_64. I'm on a RHEL4 AMD64 platform.

    My mail server has been down for 4 days now trying to get this upgrade completely. This is really a complete disaster. Any quick assistance would be enormously appreciated. Thank you!
    Open a U1 or U2 support ticket so we can have a look: https://support.zimbra.com/supportlogin.php

  3. #3
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Just wanted to remind you that since you're paying for support you might as well use it -expecially after 4 days (Don't think I'm missing another thread as last post by you was 08-02-2007.)

    Ok so, OS change plus a big jump (4.5.11 stop off point first might have been good).
    How did you change platforms - Blog or NE Backups changing servers method??
    Is this the same physical machine or did you switch?

    What's your leaway - ie can we throw up 4.5.4 and restore a backup?

  4. #4
    Myrddin is offline Intermediate Member
    Join Date
    Aug 2007
    Location
    Stamford, CT USA
    Posts
    16
    Rep Power
    8

    Default

    MMorse, thanks for the prompt reply.

    I'm going to open a support ticket as soon as the person who changed our password tells me the new password, which is probably going to be tomorrow morning.

    I did not follow a good procedure for backup. I do not have another server available to restore to. Here is my backup procedure:

    mysqldump all databases on main mysql server
    mysqldump all databases on logger mysql server
    cp -a /opt/zimbra/store /mnt/backup


    What logs do you need to begin diagnosing this? Clearly, Jetty isn't registering the web services, but why not? Where are the errors from Jetty?

  5. #5
    Myrddin is offline Intermediate Member
    Join Date
    Aug 2007
    Location
    Stamford, CT USA
    Posts
    16
    Rep Power
    8

    Default

    Also, I did not make an OS change. I'm not sure where you are seeing that. I only changed the Zimbra release.

    Running upgrade script went very well with no errors. I will post the log when I open a support ticket.

    The reason I have not opened a ticket previously is because the "Open Ticket" to "Resolve Issue" time has been about 2 weeks for our past issues. I wanted to try to solve this by myself quickly. You also docked us 2 support tickets for a single issue.

  6. #6
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Quote Originally Posted by Myrddin View Post
    Also, I did not make an OS change. I'm not sure where you are seeing that. I only changed the Zimbra release.
    Then I'm not sure why you went from RHEL 4 > RHEL 5:
    Quote Originally Posted by Myrddin View Post
    zcs-NETWORK-5.0.6_GA_2313.RHEL5_64.20080522105815 and just completed an upgrade from zcs-NETWORK-4.5.4-20070418.RHEL4_64. I'm on a RHEL4 AMD64 platform.
    Quote Originally Posted by Myrddin View Post
    The reason I have not opened a ticket previously is because the "Open Ticket" to "Resolve Issue" time has been about 2 weeks for our past issues. I wanted to try to solve this by myself quickly. You also docked us 2 support tickets for a single issue.
    Probably not intentional, if you don't reply and keep the subject intact it might create a new one, just let us know and we can correct the allotment.
    (If it turns out to be a bug or you we're just voicing need for a particular enhancement we can fix that easily. http://www.zimbra.com/support/support_overview.html "Support Incidents determined to be enhancement requests or bug reports do not count against your allotted Support Incidents.")

    Quote Originally Posted by Myrddin View Post
    I'm going to open a support ticket as soon as the person who changed our password tells me the new password, which is probably going to be tomorrow morning.
    Quote Originally Posted by Myrddin View Post
    Running upgrade script went very well with no errors. I will post the log when I open a support ticket.
    Quote Originally Posted by Myrddin View Post
    What logs do you need to begin diagnosing this? Clearly, Jetty isn't registering the web services, but why not? Where are the errors from Jetty?
    LogFile List
    The setup logs you said we're smooth but we'll take them in the case. Start with /var/log/zimbra.log & /opt/zimbra/log/mailbox.log, & /opt/zimbra/log/zmmailboxd.out (what you posted snips from) are indeed the big ones.

    Quote Originally Posted by Myrddin View Post
    I did not follow a good procedure for backup. I do not have another server available to restore to. Here is my backup procedure:

    mysqldump all databases on main mysql server
    mysqldump all databases on logger mysql server
    cp -a /opt/zimbra/store /mnt/backup
    LDAP data?...

    At the absolute bare minimum:
    /opt/zimbra/db
    /opt/zimbra/store
    /opt/zimbra/openldap-data OR /opt/zimbra/openldap.bak

    For the future, step one of the release notes shows how.

    How about the contents of /opt/zimbra/backup?

    What's your leaway - ie can we:
    -Copy your entire current /opt/zimbra to a safe place (rsync -avHK or other method like snapshot, LVM etc.)
    -Then set up fresh 4.5.4 and restore a NE backup?

    Quote Originally Posted by Myrddin View Post
    I'm going to open a support ticket as soon as the person who changed our password tells me the new password, which is probably going to be tomorrow morning.
    FYI you can also open a U1 via phone - just leave us a message/contact info.
    (Or via another working email address, though please spell out your organization info when you do as you'll be 'unknown' for a bit.)
    Last edited by mmorse; 06-04-2008 at 10:43 PM. Reason: end quote

  7. #7
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    I have updated the above to be sure I'm addressing all your questions (you had gotten another reply in while I was composing).

  8. #8
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    This has been resolved via support ticket.

  9. #9
    Myrddin is offline Intermediate Member
    Join Date
    Aug 2007
    Location
    Stamford, CT USA
    Posts
    16
    Rep Power
    8

    Default

    Ok, I called into support as you suggested and worked with Sean who was fantastic. It took only 45 minutes to resolve my issue.

    All the problems I experienced were due to the port/address binding error visible in my log posts. While they did not make any sense (indeed, java was binding to port 7025), apparently some other process was trying to bind and generating that error.

    The problem was solved by setting the following undocumented bindaddress fields in the LDAP tree:

    zimbraImapBindAddress = 5.5.5.5
    zimbraImapSSLBindAddress = 5.5.5.5
    zimbraLmtpBindAddress = 5.5.5.5
    zimbraPop3BindAddress = 5.5.5.5
    zimbraPop3SSLBindAddress = 5.5.5.5

    After setting these and doing zmailboxdctl restart, everything magically worked.

    It would be great to have online documentation of the LDAP schema at some point. Sean mentioned that this was a project once upon a time that turned out to be very difficult.

    Thanks for all the help!

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. [SOLVED] Upgraded to 5.0 OSS - Sendmail Problem
    By Chewie71 in forum Installation
    Replies: 11
    Last Post: 12-28-2007, 07:07 PM
  2. upgrade problem
    By skak in forum Installation
    Replies: 2
    Last Post: 09-26-2007, 12:56 AM
  3. Replies: 6
    Last Post: 07-04-2007, 08:36 AM
  4. Upgrade 4.5.4 to 4.5.5 questions
    By dlochart in forum Zimbra Connector for Outlook
    Replies: 11
    Last Post: 05-14-2007, 02:44 AM
  5. 4.01 to 4.02 upgrade problem (with solution)
    By criley in forum Migration
    Replies: 2
    Last Post: 09-28-2006, 11:36 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •