Zimbra offers Open Source email server software and shared calendar for Linux and the Mac
Go Back   Zimbra :: Forums > Zimbra Collaboration Suite > Administrators

Welcome to the Zimbra :: Forums!
Welcome, if you would like to post a comment please register. We also encourage you to explore all things Zimbra with our team and members of the community.

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 04-22-2008, 05:44 AM
Advanced Member
 
Posts: 194
Default [SOLVED] Mail deferred due to too many files open

Hi all, this morning I've seen this problem for the 3ed time and it is beginning to get to me. The symptoms are that all mail starts being deferred. Stopping the server and restarting it followed by a re-queue of all the deferred messages brings everything back to normal.

In my mail.log I see lots of message of two types (included are one of each with usernames changed):

Quote:
Apr 21 16:39:14 zimtest postfix/lmtp[26309]: E5668598D21: to=<user1@zt2.rutgers.edu>, relay=zimtest.rutgers.edu[128.6.76.201]:7025, delay=0.13, delays=0.11/0.01/0/0, dsn=4.4.2, status=deferred (lost connection with zimtest.rutgers.edu[128.6.76.201] while receiving the initial server greeting)
Apr 21 16:46:51 zimtest postfix/qmgr[22912]: 0A3A3598D37: to=<user2@zt2.rutgers.edu>, relay=none, delay=287, delays=0.18/287/0/0, dsn=4.3.0, status=deferred (mail transport unavailable)
At the same time mailbox.log reports:

Quote:
2008-04-21 16:38:45,509 WARN [ImapSSLServer-78] [name=user@zt2.rutgers.edu;mid=62;ip=172.16.154.78;] ima
p - ignoring error during UID FETCH: com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 75097
ExceptionId:ImapSSLServer-78:1208810325509:64fb616200a6d7ce
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:253)
at com.zimbra.cs.mailbox.MessageCache.getMimeMessage( MessageCache.java:252)
at com.zimbra.cs.mailbox.Message.getMimeMessage(Messa ge.java:284)
at com.zimbra.cs.imap.ImapMessage.getMimeMessage(Imap Message.java:176)
at com.zimbra.cs.imap.ImapHandler.fetch(ImapHandler.j ava:2937)
at com.zimbra.cs.imap.ImapHandler.doFETCH(ImapHandler .java:2788)
at com.zimbra.cs.imap.ImapHandler.executeRequest(Imap Handler.java:402)
at com.zimbra.cs.imap.TcpImapHandler.processCommand(T cpImapHandler.java:151)
at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:160)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:128)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /opt/zimbra/store/0/62/msg/18/75097-105952.msg (Too many open files
)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.jav a:106)
at com.zimbra.cs.store.FileBlobStore.getContent(FileB lobStore.java:371)
at com.zimbra.cs.store.FileBlobStore.getContent(FileB lobStore.java:365)
at com.zimbra.cs.mailbox.MessageCache.fetchFromStore( MessageCache.java:270)
at com.zimbra.cs.mailbox.MessageCache.getMimeMessage( MessageCache.java:214)
... 10 more
In zmmailbox.out I see:

Quote:
com.zimbra.cs.zclient.ZClientException: invoke Too many open files, server: localhost
ExceptionId:btpool0-238:1208810326293:c7da87bac5c4551d
Code:zclient.IO_ERROR
at com.zimbra.cs.zclient.ZClientException.IO_ERROR(ZC lientException.java:47)
at com.zimbra.cs.account.soap.SoapProvisioning.invoke (SoapProvisioning.java:177)
at com.zimbra.cs.account.soap.SoapProvisioning.getDom ainInfo(SoapProvisioning.java:775)
at com.zimbra.cs.taglib.tag.GetDomainInfoTag.getInfo( GetDomainInfoTag.java:90)
at com.zimbra.cs.taglib.tag.GetDomainInfoTag.checkCac he(GetDomainInfoTag.java:77)
at com.zimbra.cs.taglib.tag.GetDomainInfoTag.doTag(Ge tDomainInfoTag.java:66)
at org.apache.jsp.public_.login_jsp._jspService(login _jsp.java:438)
at org.apache.jasper.runtime.HttpJspBase.service(Http JspBase.java:93)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
at org.apache.jasper.servlet.JspServletWrapper.servic e(JspServletWrapper.java:373)
at org.apache.jasper.servlet.JspServlet.serviceJspFil e(JspServlet.java:477)
at org.apache.jasper.servlet.JspServlet.service(JspSe rvlet.java:371)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
at com.zimbra.webClient.servlet.JspServlet.service(Js pServlet.java:50)
at org.mortbay.jetty.servlet.ServletHolder.handle(Ser vletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler.handle(Se rvletHandler.java:362)
at org.mortbay.jetty.security.SecurityHandler.handle( SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(Se ssionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(Co ntextHandler.java:716)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebA ppContext.java:406)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispa tcher.java:268)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispa tcher.java:126)
at org.mortbay.jetty.servlet.DefaultServlet.doGet(Def aultServlet.java:465)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(Ser vletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1093)
at com.zimbra.webClient.filters.SetHeaderFilter.doFil ter(SetHeaderFilter.java:283)
at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1084)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserA gentFilter.java:81)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter .java:132)
at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1084)
at org.mortbay.jetty.servlet.ServletHandler.handle(Se rvletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle( SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(Se ssionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(Co ntextHandler.java:716)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebA ppContext.java:406)
at org.mortbay.jetty.handler.ContextHandlerCollection .handle(ContextHandlerCollection.java:211)
at org.mortbay.jetty.handler.HandlerCollection.handle (HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(Ha ndlerWrapper.java:139)
at org.mortbay.jetty.handler.RewriteHandler.handle(Re writeHandler.java:176)
at org.mortbay.jetty.handler.HandlerWrapper.handle(Ha ndlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:313)
at org.mortbay.jetty.HttpConnection.handleRequest(Htt pConnection.java:506)
at org.mortbay.jetty.HttpConnection$RequestHandler.he aderComplete(HttpConnection.java:830)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser. java:514)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpPa rser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnec tion.java:381)
at org.mortbay.io.nio.SelectChannelEndPoint.run(Selec tChannelEndPoint.java:396)
at org.mortbay.thread.BoundedThreadPool$PoolThread.ru n(BoundedThreadPool.java:442)
Caused by: java.net.SocketException: Too many open files
at java.net.Socket.createImpl(Socket.java:388)
at java.net.Socket.getImpl(Socket.java:451)
at java.net.Socket.bind(Socket.java:565)
at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.bin d(BaseSSLSocketImpl.java:95)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.bind(SS LSocketImpl.java:45)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.<init>( SSLSocketImpl.java:392)
at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl. createSocket(SSLSocketFactoryImpl.java:123)
at com.zimbra.common.util.EasySSLProtocolSocketFactor y.createSocket(EasySSLProtocolSocketFactory.java
:124)
at com.zimbra.common.util.EasySSLProtocolSocketFactor y.createSocket(EasySSLProtocolSocketFactory.java
:165)
at org.apache.commons.httpclient.HttpConnection.open( HttpConnection.java:706)
at org.apache.commons.httpclient.HttpMethodDirector.e xecuteWithRetry(HttpMethodDirector.java:386)
at org.apache.commons.httpclient.HttpMethodDirector.e xecuteMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMe thod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMe thod(HttpClient.java:324)
at com.zimbra.common.soap.SoapHttpTransport.invoke(So apHttpTransport.java:234)
at com.zimbra.common.soap.SoapTransport.invoke(SoapTr ansport.java:297)
at com.zimbra.common.soap.SoapTransport.invoke(SoapTr ansport.java:261)
at com.zimbra.cs.account.soap.SoapProvisioning.invoke (SoapProvisioning.java:173)
... 48 more

Clearly the Zimbra is running out of something. Anyone have any ideas what is going on?
Reply With Quote
  #2 (permalink)  
Old 04-22-2008, 06:00 AM
Moderator
 
Posts: 6,237
Default

What's your platform? http://www.zimbra.com/forums/announc...html#post62754

Post the output of-
As root:
ulimits -n

Then:
su - zimbra
ulimit -n

If you're using another command such as sudo - u zimbra -s or something to become the zimbra user and start it please let us know...
Reply With Quote
  #3 (permalink)  
Old 04-22-2008, 06:13 AM
Moderator
 
Posts: 6,237
Default

Haven't seen a response/updated profile yet but is it Ubuntu/Debian?
-I see you're running 5.0.4 per your current profile.

Root usually returns 1024 while the zimbra user should return 524288 (after 5.0.2 Bug 23211 - Increase max open file descriptors for 'zimbra' user)

If it doesn't:
nano /etc/pam.d/su
Uncomment/remove the # so a line reads:
session required pam_limits.so

nano /etc/pam.d/common-session
Make it read:
session required pam_limits.so

Check the /etc/security/limits.conf as well

zmcontrol stop
zmcontrol start
pam.d should update on the fly so you shouldn't have to restart the box, as long as your now getting the proper ulimit from the commands listed in my prior post.

This has been fixed for 5.0.5: Bug 24897 - update /etc/pam.d/su on ubuntu and debian so ulimits get set properly
5.0.5 is Released!

From Gratisoft.us - Current Stable Version of Sudo Info
Quote:
Major changes from version 1.6.9p12 to 1.6.9p13:
* Sudo will now set the nproc resource limit to unlimited on Linux systems to work around Linux's setuid() resource limit semantics. On PAM systems the resource limits will be reset by pam_limits.so before the command is executed.
So it looks like upgrading your sudo could fix it (if possible on your platform version/if your package update program goes that new -p15 is also out in a few days- or if you want to give a shot at manual methods).

OR

If your running sudo as say root - you can up the limits for root and then call sudo.

As root:
ulimit -n 2048


OR

stick with:
sudo -i
enter the password, then:
su - zimbra
Reply With Quote
  #4 (permalink)  
Old 04-22-2008, 06:21 AM
Advanced Member
 
Posts: 194
Default

Yup that was exactly the problem. The limits were being ignored. changing the pam configs seems to have fixed.

Thank you guys.
Reply With Quote
  #5 (permalink)  
Old 04-22-2008, 07:20 AM
Moderator
 
Posts: 6,237
Default

Good. I see you have several boxes; might want to make sure it's applied to those as well, or upgrade to 5.0.5 when you get the chance. (Suggested the manual on each as not sure if you're waiting for a break to do so - I know how critical of a time it can be at a university around finals to not do any upgrades/changes etc.)

5.0.5 is Released!
Reply With Quote
Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes


Similar Threads

Why Join?

Registering let's you ask questions, makes it easier to search, displays any files attached to posts, and notifies you about replies.

blog.zimbra.com




 

SEO by vBSEO ©2011, Crawlability, Inc.