Results 1 to 5 of 5

Thread: [SOLVED] Mail deferred due to too many files open

  1. #1
    Vladimir is offline Advanced Member
    Join Date
    Aug 2007
    Posts
    220
    Rep Power
    7

    Default [SOLVED] Mail deferred due to too many files open

    Hi all, this morning I've seen this problem for the 3ed time and it is beginning to get to me. The symptoms are that all mail starts being deferred. Stopping the server and restarting it followed by a re-queue of all the deferred messages brings everything back to normal.

    In my mail.log I see lots of message of two types (included are one of each with usernames changed):

    Apr 21 16:39:14 zimtest postfix/lmtp[26309]: E5668598D21: to=<user1@zt2.rutgers.edu>, relay=zimtest.rutgers.edu[128.6.76.201]:7025, delay=0.13, delays=0.11/0.01/0/0, dsn=4.4.2, status=deferred (lost connection with zimtest.rutgers.edu[128.6.76.201] while receiving the initial server greeting)
    Apr 21 16:46:51 zimtest postfix/qmgr[22912]: 0A3A3598D37: to=<user2@zt2.rutgers.edu>, relay=none, delay=287, delays=0.18/287/0/0, dsn=4.3.0, status=deferred (mail transport unavailable)
    At the same time mailbox.log reports:

    2008-04-21 16:38:45,509 WARN [ImapSSLServer-78] [name=user@zt2.rutgers.edu;mid=62;ip=172.16.154.78;] ima
    p - ignoring error during UID FETCH: com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 75097
    ExceptionId:ImapSSLServer-78:1208810325509:64fb616200a6d7ce
    Code:service.FAILURE
    at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:253)
    at com.zimbra.cs.mailbox.MessageCache.getMimeMessage( MessageCache.java:252)
    at com.zimbra.cs.mailbox.Message.getMimeMessage(Messa ge.java:284)
    at com.zimbra.cs.imap.ImapMessage.getMimeMessage(Imap Message.java:176)
    at com.zimbra.cs.imap.ImapHandler.fetch(ImapHandler.j ava:2937)
    at com.zimbra.cs.imap.ImapHandler.doFETCH(ImapHandler .java:2788)
    at com.zimbra.cs.imap.ImapHandler.executeRequest(Imap Handler.java:402)
    at com.zimbra.cs.imap.TcpImapHandler.processCommand(T cpImapHandler.java:151)
    at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:160)
    at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:128)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.io.FileNotFoundException: /opt/zimbra/store/0/62/msg/18/75097-105952.msg (Too many open files
    )
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.jav a:106)
    at com.zimbra.cs.store.FileBlobStore.getContent(FileB lobStore.java:371)
    at com.zimbra.cs.store.FileBlobStore.getContent(FileB lobStore.java:365)
    at com.zimbra.cs.mailbox.MessageCache.fetchFromStore( MessageCache.java:270)
    at com.zimbra.cs.mailbox.MessageCache.getMimeMessage( MessageCache.java:214)
    ... 10 more
    In zmmailbox.out I see:

    com.zimbra.cs.zclient.ZClientException: invoke Too many open files, server: localhost
    ExceptionId:btpool0-238:1208810326293:c7da87bac5c4551d
    Code:zclient.IO_ERROR
    at com.zimbra.cs.zclient.ZClientException.IO_ERROR(ZC lientException.java:47)
    at com.zimbra.cs.account.soap.SoapProvisioning.invoke (SoapProvisioning.java:177)
    at com.zimbra.cs.account.soap.SoapProvisioning.getDom ainInfo(SoapProvisioning.java:775)
    at com.zimbra.cs.taglib.tag.GetDomainInfoTag.getInfo( GetDomainInfoTag.java:90)
    at com.zimbra.cs.taglib.tag.GetDomainInfoTag.checkCac he(GetDomainInfoTag.java:77)
    at com.zimbra.cs.taglib.tag.GetDomainInfoTag.doTag(Ge tDomainInfoTag.java:66)
    at org.apache.jsp.public_.login_jsp._jspService(login _jsp.java:438)
    at org.apache.jasper.runtime.HttpJspBase.service(Http JspBase.java:93)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
    at org.apache.jasper.servlet.JspServletWrapper.servic e(JspServletWrapper.java:373)
    at org.apache.jasper.servlet.JspServlet.serviceJspFil e(JspServlet.java:477)
    at org.apache.jasper.servlet.JspServlet.service(JspSe rvlet.java:371)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
    at com.zimbra.webClient.servlet.JspServlet.service(Js pServlet.java:50)
    at org.mortbay.jetty.servlet.ServletHolder.handle(Ser vletHolder.java:487)
    at org.mortbay.jetty.servlet.ServletHandler.handle(Se rvletHandler.java:362)
    at org.mortbay.jetty.security.SecurityHandler.handle( SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(Se ssionHandler.java:181)
    at org.mortbay.jetty.handler.ContextHandler.handle(Co ntextHandler.java:716)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebA ppContext.java:406)
    at org.mortbay.jetty.servlet.Dispatcher.forward(Dispa tcher.java:268)
    at org.mortbay.jetty.servlet.Dispatcher.forward(Dispa tcher.java:126)
    at org.mortbay.jetty.servlet.DefaultServlet.doGet(Def aultServlet.java:465)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
    at org.mortbay.jetty.servlet.ServletHolder.handle(Ser vletHolder.java:487)
    at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1093)
    at com.zimbra.webClient.filters.SetHeaderFilter.doFil ter(SetHeaderFilter.java:283)
    at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1084)
    at org.mortbay.servlet.UserAgentFilter.doFilter(UserA gentFilter.java:81)
    at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter .java:132)
    at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1084)
    at org.mortbay.jetty.servlet.ServletHandler.handle(Se rvletHandler.java:360)
    at org.mortbay.jetty.security.SecurityHandler.handle( SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(Se ssionHandler.java:181)
    at org.mortbay.jetty.handler.ContextHandler.handle(Co ntextHandler.java:716)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebA ppContext.java:406)
    at org.mortbay.jetty.handler.ContextHandlerCollection .handle(ContextHandlerCollection.java:211)
    at org.mortbay.jetty.handler.HandlerCollection.handle (HandlerCollection.java:114)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(Ha ndlerWrapper.java:139)
    at org.mortbay.jetty.handler.RewriteHandler.handle(Re writeHandler.java:176)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(Ha ndlerWrapper.java:139)
    at org.mortbay.jetty.Server.handle(Server.java:313)
    at org.mortbay.jetty.HttpConnection.handleRequest(Htt pConnection.java:506)
    at org.mortbay.jetty.HttpConnection$RequestHandler.he aderComplete(HttpConnection.java:830)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser. java:514)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpPa rser.java:211)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnec tion.java:381)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(Selec tChannelEndPoint.java:396)
    at org.mortbay.thread.BoundedThreadPool$PoolThread.ru n(BoundedThreadPool.java:442)
    Caused by: java.net.SocketException: Too many open files
    at java.net.Socket.createImpl(Socket.java:388)
    at java.net.Socket.getImpl(Socket.java:451)
    at java.net.Socket.bind(Socket.java:565)
    at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.bin d(BaseSSLSocketImpl.java:95)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.bind(SS LSocketImpl.java:45)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.<init>( SSLSocketImpl.java:392)
    at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl. createSocket(SSLSocketFactoryImpl.java:123)
    at com.zimbra.common.util.EasySSLProtocolSocketFactor y.createSocket(EasySSLProtocolSocketFactory.java
    :124)
    at com.zimbra.common.util.EasySSLProtocolSocketFactor y.createSocket(EasySSLProtocolSocketFactory.java
    :165)
    at org.apache.commons.httpclient.HttpConnection.open( HttpConnection.java:706)
    at org.apache.commons.httpclient.HttpMethodDirector.e xecuteWithRetry(HttpMethodDirector.java:386)
    at org.apache.commons.httpclient.HttpMethodDirector.e xecuteMethod(HttpMethodDirector.java:170)
    at org.apache.commons.httpclient.HttpClient.executeMe thod(HttpClient.java:396)
    at org.apache.commons.httpclient.HttpClient.executeMe thod(HttpClient.java:324)
    at com.zimbra.common.soap.SoapHttpTransport.invoke(So apHttpTransport.java:234)
    at com.zimbra.common.soap.SoapTransport.invoke(SoapTr ansport.java:297)
    at com.zimbra.common.soap.SoapTransport.invoke(SoapTr ansport.java:261)
    at com.zimbra.cs.account.soap.SoapProvisioning.invoke (SoapProvisioning.java:173)
    ... 48 more

    Clearly the Zimbra is running out of something. Anyone have any ideas what is going on?

  2. #2
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    What's your platform? http://www.zimbra.com/forums/announc...html#post62754

    Post the output of-
    As root:
    ulimits -n

    Then:
    su - zimbra
    ulimit -n

    If you're using another command such as sudo - u zimbra -s or something to become the zimbra user and start it please let us know...

  3. #3
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Haven't seen a response/updated profile yet but is it Ubuntu/Debian?
    -I see you're running 5.0.4 per your current profile.

    Root usually returns 1024 while the zimbra user should return 524288 (after 5.0.2 Bug 23211 - Increase max open file descriptors for 'zimbra' user)

    If it doesn't:
    nano /etc/pam.d/su
    Uncomment/remove the # so a line reads:
    session required pam_limits.so

    nano /etc/pam.d/common-session
    Make it read:
    session required pam_limits.so

    Check the /etc/security/limits.conf as well

    zmcontrol stop
    zmcontrol start
    pam.d should update on the fly so you shouldn't have to restart the box, as long as your now getting the proper ulimit from the commands listed in my prior post.

    This has been fixed for 5.0.5: Bug 24897 - update /etc/pam.d/su on ubuntu and debian so ulimits get set properly
    5.0.5 is Released!

    From Gratisoft.us - Current Stable Version of Sudo Info
    Major changes from version 1.6.9p12 to 1.6.9p13:
    * Sudo will now set the nproc resource limit to unlimited on Linux systems to work around Linux's setuid() resource limit semantics. On PAM systems the resource limits will be reset by pam_limits.so before the command is executed.
    So it looks like upgrading your sudo could fix it (if possible on your platform version/if your package update program goes that new -p15 is also out in a few days- or if you want to give a shot at manual methods).

    OR

    If your running sudo as say root - you can up the limits for root and then call sudo.

    As root:
    ulimit -n 2048


    OR

    stick with:
    sudo -i
    enter the password, then:
    su - zimbra

  4. #4
    Vladimir is offline Advanced Member
    Join Date
    Aug 2007
    Posts
    220
    Rep Power
    7

    Default

    Yup that was exactly the problem. The limits were being ignored. changing the pam configs seems to have fixed.

    Thank you guys.

  5. #5
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Good. I see you have several boxes; might want to make sure it's applied to those as well, or upgrade to 5.0.5 when you get the chance. (Suggested the manual on each as not sure if you're waiting for a break to do so - I know how critical of a time it can be at a university around finals to not do any upgrades/changes etc.)

    5.0.5 is Released!

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 7
    Last Post: 02-03-2011, 07:01 AM
  2. Problem with Postfix and MTA
    By ZMilton in forum Administrators
    Replies: 16
    Last Post: 04-16-2008, 06:47 AM
  3. [SOLVED] Mailserver down when send file attach of 50Mb
    By ZMilton in forum Administrators
    Replies: 20
    Last Post: 04-10-2008, 11:44 AM
  4. fresh install down may be due to tomcat
    By gon in forum Installation
    Replies: 10
    Last Post: 07-25-2007, 08:09 AM
  5. receiveing mail
    By maybethistime in forum Administrators
    Replies: 15
    Last Post: 12-09-2005, 04:55 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •