Zimbra offers Open Source email server software and shared calendar for Linux and the Mac
 
Go Back   Zimbra - Forums > Zimbra Collaboration Suite > Administrators
Reload this Page Zimbra shutdowns every n hours.

Welcome to the Zimbra - Forums!
Welcome, if you would like to post a comment please register. We also encourage you to explore all things Zimbra with our team and members of the community.

Reply
Page 1 of 2 1 2
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 05-31-2007, 12:25 PM
Junior Member
  
Posts: 8
Default Zimbra shutdowns every n hours.
Recently Zimbra (Network Edition 4.5.4) started to shutdown every 8-24 hours.
Strange errors can be found in catalina.out

It is either "Document base ... does not exist"
or "Too many files opened"

zmcontrol status indicate that all services run fine. But web interface gets no accessible.
Later everything except administrator interface dies.

Can anyone advise on this. Searching internet did not help.

catalina.out attached
Attached Files
File Type: zip catalina.zip (8.3 KB, 27 views)
Last edited by Andrewb : 05-31-2007 at 12:31 PM.
Reply With Quote
  #2 (permalink)  
Old 05-31-2007, 03:56 PM
Junior Member
  
Posts: 8
Default Note
One interesting note.
Occasionally kvoop process occupies a lot of file handles. I saw 54000 of them. Here is part of log that repeats a lot of times.
Code:
kvoop     21925  zimbra  cwd       DIR        8,3      4096    3008769 /opt
kvoop     21925  zimbra  rtd       DIR        8,3      4096          2 /
kvoop     21925  zimbra  txt       REG        8,3     44064    3095271 /opt/zimbra/verity/FilterSDK/bin/kvoop
kvoop     21925  zimbra  mem       REG        8,3    112168     719492 /lib/ld-2.3.4.so
kvoop     21925  zimbra  mem       REG        8,3   1529204     784898 /lib/tls/libc-2.3.4.so
kvoop     21925  zimbra  mem       REG        8,3     16732     719495 /lib/libdl-2.3.4.so
kvoop     21925  zimbra  mem       REG        8,3    213772     784900 /lib/tls/libm-2.3.4.so
kvoop     21925  zimbra  mem       REG        8,3     40108     719516 /lib/libgcc_s-3.4.6-20060404.so.1
kvoop     21925  zimbra  mem       REG        8,3    107800     784899 /lib/tls/libpthread-2.3.4.so
kvoop     21925  zimbra  mem       REG        8,3     35328    3095276 /opt/zimbra/verity/FilterSDK/bin/kvxtract.so
kvoop     21925  zimbra  mem       REG        8,3    271320    3095279 /opt/zimbra/verity/FilterSDK/bin/kwad.so
kvoop     21925  zimbra  mem       REG        8,3    343788    3095272 /opt/zimbra/verity/FilterSDK/bin/kvutil.so
kvoop     21925  zimbra  mem       REG        8,3    252456    3095267 /opt/zimbra/verity/FilterSDK/bin/kvfilter.so
kvoop     21925  zimbra  mem       REG        8,3    733488    3410609 /usr/lib/libstdc++.so.5.0.7
kvoop     21925  zimbra    0r      CHR        1,3                 2385 /dev/null
kvoop     21925  zimbra    1w      REG        8,3  10652323    3025293 /opt/zimbra/log/convertd.out
kvoop     21925  zimbra    2w      REG        8,3  10652323    3025293 /opt/zimbra/log/convertd.out
kvoop     21925  zimbra    3u      CHR        5,0                 2384 /dev/tty
kvoop     21925  zimbra    4r      CHR        1,8                  471 /dev/random
kvoop     21925  zimbra    5u      CHR      136,1                    3 /dev/pts/1 (deleted)
kvoop     21925  zimbra    6r      CHR        1,9                  473 /dev/urandom
kvoop     21925  zimbra    7w     FIFO        0,7                61185 pipe
kvoop     21925  zimbra    8w      REG        8,3   4363712    3025139 /opt/zimbra/log/convertd.log
kvoop     21925  zimbra    9u     IPv6      76287                  TCP *:7047 (LISTEN)
kvoop     21925  zimbra   10u     sock        0,4                76285 can't identify protocol
kvoop     21925  zimbra   11r     FIFO        0,7               320990 pipe
kvoop     21925  zimbra   12u     IPv6     320734                  TCP localhost.localdomain:7047->localhost.localdomain:43516 (CLOSE_WAIT)
kvoop     21925  zimbra   13u     IPv6     320981                  TCP localhost.localdomain:7047->localhost.localdomain:43538 (CLOSE_WAIT)
kvoop     21925  zimbra   14w     FIFO        0,7               320922 pipe
kvoop     21925  zimbra   15r     FIFO        0,7               320923 pipe
kvoop     21925  zimbra   16w     FIFO        0,7               320924 pipe
kvoop     21925  zimbra   17r     FIFO        0,7               320925 pipe
kvoop     21925  zimbra   18w     FIFO        0,7               320929 pipe
kvoop     21925  zimbra   19r     FIFO        0,7               320930 pipe
kvoop     21925  zimbra   20w     FIFO        0,7               320931 pipe
kvoop     21925  zimbra   21r     FIFO        0,7               320932 pipe
kvoop     21925  zimbra   22w     FIFO        0,7               320933 pipe
kvoop     21925  zimbra   23r     FIFO        0,7               320934 pipe
kvoop     21925  zimbra   24w     FIFO        0,7               320938 pipe
kvoop     21925  zimbra   25r     FIFO        0,7               320939 pipe
kvoop     21925  zimbra   26w     FIFO        0,7               320943 pipe
kvoop     21925  zimbra   27r     FIFO        0,7               320944 pipe
kvoop     21925  zimbra   28w     FIFO        0,7               320948 pipe
kvoop     21925  zimbra   29r     FIFO        0,7               320949 pipe
kvoop     21925  zimbra   30w     FIFO        0,7               320950 pipe
kvoop     21925  zimbra   31r     FIFO        0,7               320951 pipe
kvoop     21925  zimbra   32w     FIFO        0,7               320955 pipe
kvoop     21925  zimbra   33r     FIFO        0,7               320956 pipe
kvoop     21925  zimbra   34w     FIFO        0,7               320957 pipe
kvoop     21925  zimbra   35r     FIFO        0,7               320958 pipe
kvoop     21925  zimbra   36w     FIFO        0,7               320962 pipe
kvoop     21925  zimbra   37r     FIFO        0,7               320963 pipe
kvoop     21925  zimbra   38w     FIFO        0,7               320964 pipe
kvoop     21925  zimbra   39r     FIFO        0,7               320965 pipe
kvoop     21925  zimbra   40w     FIFO        0,7               320969 pipe
kvoop     21925  zimbra   41r     FIFO        0,7               320970 pipe
kvoop     21925  zimbra   42w     FIFO        0,7               320971 pipe
kvoop     21925  zimbra   43r     FIFO        0,7               320972 pipe
kvoop     21925  zimbra   44w     FIFO        0,7               320976 pipe
kvoop     21925  zimbra   45r     FIFO        0,7               320977 pipe
kvoop     21925  zimbra   46w     FIFO        0,7               320978 pipe
kvoop     21925  zimbra   47r     FIFO        0,7               320979 pipe
kvoop     21925  zimbra   48w     FIFO        0,7               320983 pipe
kvoop     21925  zimbra   49r     FIFO        0,7               320984 pipe
kvoop     21925  zimbra   50w     FIFO        0,7               320985 pipe
kvoop     21925  zimbra   51r     FIFO        0,7               320986 pipe
kvoop     21925  zimbra   54w     FIFO        0,7               320991 pipe
kvoop     21925  zimbra  147u     IPv6     320901                  TCP localhost.localdomain:7047->localhost.localdomain:43526 (CLOSE_WAIT)
kvoop     21925  zimbra  201w     FIFO        0,7               320903 pipe
kvoop     21925  zimbra  205w     FIFO        0,7               320908 pipe
kvoop     21925  zimbra  214r     FIFO        0,7               320909 pipe
kvoop     21925  zimbra  215w     FIFO        0,7               320910 pipe
kvoop     21925  zimbra  218r     FIFO        0,7               320904 pipe
kvoop     21925  zimbra  219r     FIFO        0,7               320911 pipe
kvoop     21925  zimbra  220w     FIFO        0,7               320915 pipe
kvoop     21925  zimbra  221r     FIFO        0,7               320916 pipe
kvoop     21925  zimbra  222w     FIFO        0,7               320917 pipe
kvoop     21925  zimbra  223r     FIFO        0,7               320918 pipe
kvoop     21925  zimbra  279w     FIFO        0,7               320747 pipe
kvoop     21925  zimbra  280w     FIFO        0,7               320745 pipe
kvoop     21925  zimbra  281r     FIFO        0,7               320746 pipe
kvoop     21925  zimbra  282r     FIFO        0,7               320748 pipe
Reply With Quote
  #3 (permalink)  
Old 05-31-2007, 09:23 PM
Zimbra-Yahoo Consultant
  
Posts: 5,608
Default
What is your OS, RAM size, etc?

jh
Reply With Quote
  #4 (permalink)  
Old 06-01-2007, 02:23 AM
Junior Member
  
Posts: 8
Default Os
CentOS ServerCD 4.4
2GB RAM
Reply With Quote
  #5 (permalink)  
Old 06-01-2007, 02:41 AM
Zimbra Consultant & Moderator
  
Posts: 11,320
Default
There's nothing extraordinary in the catalina.out that you've posted. Where are you seeing the "Too many files opened" message? What's in your /var/log/zimbra.log or /opt/zimbra/log/mailbox.log when the system dies? When you say 'dies' do you mean they just stop responding or they actually terminate?

There was a problem with kvoop, described in this bug report but that's supposed to have been fixed. Could you be hitting the same (or similar) problem with attachments?
__________________
Regards


Bill
Reply With Quote
  #6 (permalink)  
Old 06-01-2007, 02:53 AM
Junior Member
  
Posts: 8
Default Logs
No errors in messages or zimbra.log files.
zmcontrol status indicate that services are fine. However web interface stops responding as well as pop/imap/smtp do not work.

I noticed that a number of file handles grow during reindexing/indexing. kvoop can take up to 50000 of file handles and releases them later. Is this normal?

It may be an issue that something wrong with the raid adapter or it is too slow working with large number of small requests.

Here are a couple of more log copy/pastes:
Code:
 May 30, 2007 11:45:08 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint [SSL: ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=7071]] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
        at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:110)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
May 30, 2007 11:45:08 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Reinitializing ServerSocket
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=80] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Reinitializing ServerSocket
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint null ignored exception: java.net.BindException: Permission denied:80
java.net.BindException: Permission denied:80
        at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:297)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:468)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Restarting endpoint
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Permission denied:80
java.net.BindException: Permission denied:80
        at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:297)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:480)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
May 30, 2007 11:55:05 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception (java.lang.ThreadDeath) executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@15a4daf, terminating thread
May 30, 2007 11:59:01 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint [SSL: ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=7071]] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
May 30, 2007 11:45:08 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint [SSL: ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=7071]] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
        at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:110)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
May 30, 2007 11:45:08 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Reinitializing ServerSocket
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=80] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Reinitializing ServerSocket
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint null ignored exception: java.net.BindException: Permission denied:80
java.net.BindException: Permission denied:80
        at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:297)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:468)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Restarting endpoint
May 30, 2007 11:55:05 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Permission denied:80
java.net.BindException: Permission denied:80
        at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:297)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:480)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
May 30, 2007 11:55:05 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception (java.lang.ThreadDeath) executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@15a4daf, terminating thread
May 30, 2007 11:59:01 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint [SSL: ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=7071]] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
Reply With Quote
  #7 (permalink)  
Old 06-01-2007, 04:20 AM
Zimbra Consultant & Moderator
  
Posts: 11,320
Default
What RAID level do you have this on?
__________________
Regards


Bill
Reply With Quote
  #8 (permalink)  
Old 06-01-2007, 05:06 AM
Junior Member
  
Posts: 8
Default Raid
It is RAID 5, but the server hosts only 60 users
Reply With Quote
  #9 (permalink)  
Old 06-25-2007, 05:35 PM
Active Member
  
Posts: 28
Default
Quote:
Originally Posted by Andrewb View Post
Recently Zimbra (Network Edition 4.5.4) started to shutdown every 8-24 hours.
Strange errors can be found in catalina.out

It is either "Document base ... does not exist"
or "Too many files opened"

zmcontrol status indicate that all services run fine. But web interface gets no accessible.
Later everything except administrator interface dies.

Can anyone advise on this. Searching internet did not help.

catalina.out attached
Interesting... I am having the same problem with one machine...

I'll try to get more info and report. In this case, machine runs out of network ports and fill up with CLOSE_WAIT on pop3 port.
Reply With Quote
  #10 (permalink)  
Old 06-25-2007, 06:03 PM
Active Member
  
Posts: 28
Default
Quote:
Originally Posted by ekaftan View Post
Interesting... I am having the same problem with one machine...

I'll try to get more info and report. In this case, machine runs out of network ports and fill up with CLOSE_WAIT on pop3 port.

A hangup just happened before my eyes.

First, a los of there starting to appear:
2007-06-25 18:52:59,496 WARN [LmtpServer-32] [name=losangeles@recaudadora.cl;]
sqltrace - Connection pool is 75% utilized. 10 connections out of a maximum of
10 in use. Turn on debug logging for zimbra.sqltrace to see stack traces of con
nections not returned to the pool.

First with 8, then 9, then 10 connections.


Then two or three of these:


2007-06-25 18:51:02,288 INFO [Pop3Server-13] [] ProtocolHandler - Exception occ
urred while handling connection
java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutp utStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStre am.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedO utputStream.java:65
)
at java.io.BufferedOutputStream.write(BufferedOutputS tream.java:78)
at com.zimbra.cs.pop3.Pop3Handler.sendMessage(Pop3Han dler.java:429)
at com.zimbra.cs.pop3.Pop3Handler.doRETR(Pop3Handler. java:625)
at com.zimbra.cs.pop3.Pop3Handler.processCommandInter nal(Pop3Handler.jav
a:257)
at com.zimbra.cs.pop3.Pop3Handler.processCommand(Pop3 Handler.java:141)
at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHan
dler.java:231)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:198)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown So
urce)
at java.lang.Thread.run(Thread.java:595)

After that, only this messages:

2007-06-25 18:53:03,654 INFO [Pop3Server-1669] [] pop - connected
2007-06-25 18:53:08,016 INFO [Pop3Server-1670] [] pop - connected
2007-06-25 18:53:10,027 INFO [Pop3Server-1671] [] pop - connected


Then, huge amounts of pop3 ports in CLOSE_WAIT state and no more service until
a tomcat reboot...

Any ideas?
Reply With Quote
Reply
Page 1 of 2 1 2

« Previous Thread | Today's Posts or Week's Posts | Next Thread »

Thread Tools
Display Modes
Linear Mode Linear Mode


Similar Threads

Product Information

Why Join?

Registering let's you ask questions, makes it easier to search, displays any files attached to posts, and notifies you about replies.

Zimbrablog.com


Home - Blog - Contact Us - Archive - Top


 

Search Engine Optimization by vBSEO 3.1.0