Too many open files?

[davidg]

Hias everybody!

Recently, i've started getting this Too many open files error on my Zimbra install. After a couple of this errors the mailboxd dies.

Zimbra release:
Release 6.0.4_GA_2038.SLES10_64_20091214232236 SLES10_64 FOSS edition.

mailbox.log:
Caused by: java.io.FileNotFoundException: /opt/zimbra/index/0/59/index/0/_2vv.fdx (Too many open files)
2010-03-25 10:34:42,274 ERROR [IndexWriterFlush-1] [] indexadd - Caught Exception java.io.FileNotFoundException: /opt/zimbra/index/0/59/index/0/segments_2or (Too many open files) in LuceneIndex.closeIndexWriter
java.io.FileNotFoundException: /opt/zimbra/index/0/59/index/0/segments_2or (Too many open files)
java.io.FileNotFoundException: /opt/zimbra/store/incoming/1269507585838-112.msg (Too many open files)
java.io.FileNotFoundException: /opt/zimbra/store/incoming/1269507585838-112.msg (Too many open files)

Well, sure, i was suspecting ulimits to be fault of this but it seems they are not. Here's the settings:

/etc/security/limits.conf
zimbra soft nofile 900000
zimbra hard nofile 1024000

If i do ulimit -n on the zimbra used i get 900000

I've never seen /proc/sys/fs/file-nr over 5000, which leaves me puzzled .

Where should i look next? It's pretty damn annoying to keep restarting mailboxd and i'm really out of ideas......




Regards,
D.

[davidg]

Hi!

I have now found out, that the mailboxd always crashes here:

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: Too many open files

STACKTRACE:

java.net.SocketException: Too many open files
at java.net.Socket.createImpl(Socket.java:394)
at java.net.Socket.<init>(Socket.java:368)
at java.net.Socket.<init>(Socket.java:215)
at com.mysql.jdbc.StandardSocketFactory.connect(Stand ardSocketFactory.java:256)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:271)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2771)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 555)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:285)
at java.sql.DriverManager.getConnection(DriverManager .java:582)
at java.sql.DriverManager.getConnection(DriverManager .java:207)
at com.zimbra.cs.db.DbPool.getMaintenanceConnection(D bPool.java:388)
at com.zimbra.cs.db.MySQL.flushToDisk(MySQL.java:202)
at com.zimbra.cs.redolog.RedoLogManager.rollover(Redo LogManager.java:604)
at com.zimbra.cs.redolog.RedoLogManager.forceRollover (RedoLogManager.java:642)
at com.zimbra.cs.redolog.RedoLogManager.forceRollover (RedoLogManager.java:638)
at com.zimbra.cs.redolog.RedoLogManager.start(RedoLog Manager.java:268)
at com.zimbra.cs.redolog.DefaultRedoLogProvider.start up(DefaultRedoLogProvider.java:41)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:209)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:122)
at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 125)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:241)
at org.mortbay.jetty.servlet.ServletHolder.initServle t(ServletHolder.java:440)
at org.mortbay.jetty.servlet.ServletHolder.doStart(Se rvletHolder.java:263)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initializ e(ServletHandler.java:685)
at org.mortbay.jetty.servlet.Context.startContext(Con text.java:140)
at org.mortbay.jetty.webapp.WebAppContext.startContex t(WebAppContext.java:1250)
at org.mortbay.jetty.handler.ContextHandler.doStart(C ontextHandler.java:517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(Web AppContext.java:467)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection .doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:152)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.jetty.handler.DebugHandler.doStart(Deb ugHandler.java:127)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.setuid.SetUIDServer.doStart(SetUIDServ er.java:158)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:985)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)

** END NESTED EXCEPTION **

2010-03-25 13:09:54,743 FATAL [main] [] system - Unable to load timezones from /opt/zimbra/conf/timezones.ics
java.io.FileNotFoundException: /opt/zimbra/conf/timezones.ics (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.jav a:106)
at com.zimbra.cs.mailbox.calendar.WellKnownTimeZones. loadFromFile(WellKnownTimeZones.java:72)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:167)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:122)
at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 125)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:241)
at org.mortbay.jetty.servlet.ServletHolder.initServle t(ServletHolder.java:440)
at org.mortbay.jetty.servlet.ServletHolder.doStart(Se rvletHolder.java:263)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initializ e(ServletHandler.java:685)
at org.mortbay.jetty.servlet.Context.startContext(Con text.java:140)
at org.mortbay.jetty.webapp.WebAppContext.startContex t(WebAppContext.java:1250)
at org.mortbay.jetty.handler.ContextHandler.doStart(C ontextHandler.java:517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(Web AppContext.java:467)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection .doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:152)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.jetty.handler.DebugHandler.doStart(Deb ugHandler.java:127)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.setuid.SetUIDServer.doStart(SetUIDServ er.java:158)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:50)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:985)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)

I do see other "Too many open files" errors but it seems, that mailboxd always crashes ONLY with this error.

Any clues anyone?


Regards,
D.

[davidg]

For now, i have changed this values:

$ zmprov ms this.server.name zimbraPop3NumThreads 300
$ zmprov ms this.server.name zimbraImapNumThreads 500
$ zmprov ms <localservername> zimbraLmtpNumThreads 40

and

$ zmlocalconfig -e postfix_lmtp_destination_concurrency_limit=20

Taken from here Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki

I will monitor what is going on and report the results.



Regards,
D.

[davidg]

Posted stuff above didn't work at all...



Regards,
D.