Results 1 to 8 of 8

Thread: com.zimbra.cs.mailbox.MailboxLock$LockFailedExcept ion: too many waiters - mailbox.log

  1. #1
    amartin is offline Intermediate Member
    Join Date
    Dec 2010
    Posts
    20
    Rep Power
    4

    Default com.zimbra.cs.mailbox.MailboxLock$LockFailedExcept ion: too many waiters - mailbox.log

    Hello,

    I recently upgraded from ZCS 7.1.3 (on Ubuntu 10.04) to ZCS 8.0.5 (on Ubuntu 12.04). The upgrade went smoothly and 8.0.5 is running well now, except that some users are getting the following error fairly frequently:
    2013-10-14 09:32:09,793 WARN [qtp571606922-24635:http://192.168.0.2:80/service/soap/NoOpRequest] [name=myusername@mydomain.com;mid=95;ip=192.168.1.1 0;ua=ZimbraWebClient - FF24 (Win)/8.0.5_GA_5839;] soap - handler exception
    com.zimbra.cs.mailbox.MailboxLock$LockFailedExcept ion: too many waiters: 15
    at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock .java:50)
    at com.zimbra.cs.mailbox.Mailbox.beginTransaction(Mai lbox.java:1489)
    at com.zimbra.cs.mailbox.Mailbox.beginTransaction(Mai lbox.java:1474)
    at com.zimbra.cs.mailbox.Mailbox.getMessagesByConvers ation(Mailbox.java:3848)
    at com.zimbra.cs.mailbox.Mailbox.getMessagesByConvers ation(Mailbox.java:3841)
    at com.zimbra.cs.service.mail.ToXML.encodeConversatio nSummary(ToXML.java:1038)
    at com.zimbra.cs.service.mail.ToXML.encodeItem(ToXML. java:188)
    at com.zimbra.cs.session.SoapSession.putQueuedNotific ations(SoapSession.java:1452)
    at com.zimbra.cs.session.SoapSession.putNotifications (SoapSession.java:1393)
    at com.zimbra.soap.SoapEngine.generateResponseHeader( SoapEngine.java:639)
    at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:426)
    at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:258)
    at com.zimbra.soap.SoapServlet.doWork(SoapServlet.jav a:293)
    at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:209)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:727)
    at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:208)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
    at org.eclipse.jetty.servlet.ServletHolder.handle(Ser vletHolder.java:654)
    at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1320)
    at com.zimbra.cs.servlet.RequestStringFilter.doFilter (RequestStringFilter.java:52)
    at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1291)
    at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(Set HeaderFilter.java:57)
    at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1291)
    at org.eclipse.jetty.servlets.UserAgentFilter.doFilte r(UserAgentFilter.java:82)
    at org.eclipse.jetty.servlets.GzipFilter.doFilter(Gzi pFilter.java:256)
    at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1291)
    at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ET agHeaderFilter.java:45)
    at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1291)
    at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(Zim braQoSFilter.java:114)
    at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1291)
    at org.eclipse.jetty.servlets.DoSFilter.doFilterChain (DoSFilter.java:474)
    at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSF ilter.java:345)
    at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSF ilter.java:316)
    at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1291)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle( ServletHandler.java:443)
    at org.eclipse.jetty.server.handler.ScopedHandler.han dle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle( SecurityHandler.java:556)
    at org.eclipse.jetty.server.session.SessionHandler.do Handle(SessionHandler.java:227)
    at org.eclipse.jetty.server.handler.ContextHandler.do Handle(ContextHandler.java:1044)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(S ervletHandler.java:372)
    at org.eclipse.jetty.server.session.SessionHandler.do Scope(SessionHandler.java:189)
    at org.eclipse.jetty.server.handler.ContextHandler.do Scope(ContextHandler.java:978)
    at org.eclipse.jetty.server.handler.ScopedHandler.han dle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCol lection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection .handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.ha ndle(HandlerWrapper.java:116)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.h andle(RewriteHandler.java:317)
    at org.eclipse.jetty.server.handler.DebugHandler.hand le(DebugHandler.java:81)
    at org.eclipse.jetty.server.handler.HandlerWrapper.ha ndle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java :369)
    at org.eclipse.jetty.server.AbstractHttpConnection.ha ndleRequest(AbstractHttpConnection.java:486)
    at org.eclipse.jetty.server.AbstractHttpConnection.co ntent(AbstractHttpConnection.java:944)
    at org.eclipse.jetty.server.AbstractHttpConnection$Re questHandler.content(AbstractHttpConnection.java:1 005)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpPa rser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(H ttpParser.java:240)
    at org.eclipse.jetty.server.AsyncHttpConnection.handl e(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.han dle(SelectChannelEndPoint.java:668)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.r un(SelectChannelEndPoint.java:52)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.run Job(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.r un(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:724)
    I have clients syncing via ActiveSync and also users have a particularly large number of calendar mounts (each user has a couple hundred calendar mounts: a folder for each organizational group and then a calendar mount in each folder for each user in said group). I have already adjusted the DoSFilter parameters as outlined here:
    zmprov mcf zimbraHttpDosFilterMaxRequestsPerSec 100
    zmprov mcf zimbraHttpDosFilterDelayMillis 20
    zmmailboxdctl restart
    Any ideas on what is causing this locking error, and how to resolve it?

    Thanks,

    Andrew

  2. #2
    amartin is offline Intermediate Member
    Join Date
    Dec 2010
    Posts
    20
    Rep Power
    4

    Default

    I'm now seeing this error very frequently. With a userbase of around 150 users, this error occurring at some points over 3000 times per minute!
    Quote Originally Posted by frequency of error in mailbox.log
    310 12:29:
    1669 12:30:
    241 12:31:
    3189 12:32:
    1445 12:33:
    3021 12:34:
    I was able to reduce these numbers by an order of magnitude by disabling access for activesync clients, but when all waiting threads would get full, any user trying to access anything in the Zimbra web interface would get denied.

    I then adjusted zimbra_mailbox_lock_max_waiting_threads using zmlocalconfig from 15 to 30 and restarted mailboxd:
    zmlocalconfig -e zimbra_mailbox_lock_max_waiting_threads=30
    This temporarily relieved the problem, but then the error reappeared after all 30 were filled. I then decreased the zimbra_mailbox_lock_timeout from 60 (presumably seconds) to 20:
    zmlocalconfig -e zimbra_mailbox_lock_timeout=20

  3. #3
    amartin is offline Intermediate Member
    Join Date
    Dec 2010
    Posts
    20
    Rep Power
    4

    Default

    Can anyone provide some insight into how to tune these values? It seems that a single user can initiate a bunch of SOAP requests (e.g delete a large folder of mail) and their request consumes all of the available threads, thus causing other users to get the LockFailedException. How can I safely throttle the number of connections per user?

  4. #4
    BCamp is offline Active Member
    Join Date
    Dec 2009
    Posts
    25
    Rep Power
    5

    Default

    Did you ever figure this out? I'm on 8.0.5 Ubuntu 10.04, running into this exact issue.

  5. #5
    amartin is offline Intermediate Member
    Join Date
    Dec 2010
    Posts
    20
    Rep Power
    4

    Default

    Quote Originally Posted by BCamp View Post
    Did you ever figure this out? I'm on 8.0.5 Ubuntu 10.04, running into this exact issue.
    I do not have a complete solution (no fix from the Zimbra developers yet), but I have been able to mitigate the problem pretty successfully by adjusting these tunables:
    * zimbraMailEmptyFolderBatchSize - adjust with zmprov, default is 1000, I have it set to 100
    * zimbraHttpNumThreads - adjust with zmprov - default is 250, I have it set to 600 now
    * zimbra_mailbox_lock_max_waiting_threads - adjust with zmlocalconfig - default is 15, I have it set to 100
    * zimbra_mailbox_lock_timeout - adjust with zmlocalconfig - default is 60, I have it set to 90

    These techniques increase the number of available threads and the max number of locks, so that more threads are available and more threads can lock. The key is to balance the total number of available threads (zimbraHttpNumThreads) with the total number of threads allowed to lock (zimbra_mailbox_lock_max_waiting_threads). You don't want too high of a percentage to be able to lock, otherwise there won't be enough free threads to handle incoming requests. Also remember to restart zmmailboxd after updating any of the above values.

    The developers have posted some information in this bug report as well (similar to what I posted above):
    https://bugzilla.zimbra.com/show_bug.cgi?id=84490

  6. #6
    BCamp is offline Active Member
    Join Date
    Dec 2009
    Posts
    25
    Rep Power
    5

    Default

    Thank you, AMartin! I've been watching my mailbox.log for about 20 minutes now, haven't seen the lock failure come back once yet. I think this was exactly what I needed. I appreciate the help.

  7. #7
    amartin is offline Intermediate Member
    Join Date
    Dec 2010
    Posts
    20
    Rep Power
    4

    Default

    Great, I'm glad to here that it helps! Please let me know if you find any improvements on the modifications I recommended

  8. #8
    BCamp is offline Active Member
    Join Date
    Dec 2009
    Posts
    25
    Rep Power
    5

    Default

    Quote Originally Posted by amartin View Post
    Great, I'm glad to here that it helps! Please let me know if you find any improvements on the modifications I recommended
    The only other change I implemented was to increase the number of lmtp threads. Once I got the lock failure messages to go away, I saw a less frequent error in mailbox.log indicating my lmtp pool was over-utilized. Increasing the number from 20 (default) to 40 seems to have fixed that issue for me.

    Code:
    zmprov mcf zimbraLmtpNumThreads 40
    Not entirely related to the original issue, but it's a close cousin. Hopefully it'll help someone else that comes across this thread.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 0
    Last Post: 07-10-2013, 01:17 PM
  2. Replies: 1
    Last Post: 10-23-2012, 06:09 AM
  3. Replies: 3
    Last Post: 02-16-2012, 09:17 PM
  4. Replies: 2
    Last Post: 05-03-2011, 07:33 AM
  5. Replies: 0
    Last Post: 05-02-2011, 10:49 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •