Results 1 to 7 of 7

Thread: Server no longer working, 21/20 thread limit reached?

  1. #1
    Eclipse is offline Senior Member
    Join Date
    Jun 2011
    Posts
    73
    Rep Power
    3

    Default [SOLVED]Server no longer working, 21/20 thread limit reached?

    Hey guys, I'm having some trouble. Over the weekend some complete idiot and/or super genius got in to the server and screwed a lot of stuff up. I believe that I've fixed the bulk of it but there are still a few things I'm having trouble with. As of right now, nothing works no admin console, web e-mail, client or otherwise.

    CentOS 5.6
    Zimbra 7.1.2

    As stated earlier I cannot even get into the admin console to changes things, it just sits at connected, however I was able to telnet in on the server itself and send an email to an external domain, so it feels like the main receiving process / program is no longer working or broken.

    Firewall has been turned off for testing at the moment so its not that.

    One more thing to note, we recently added around 12 new accounts with a new domain name, one account has a 6GB email account with about 100,000 or so emails in the inbox, if this is whats crashing or preventing the server, is there a way to make Zimbra skip an email domain to prevent it from being loaded or something of the sort to see if that's the problem?

    [zimbra@mail root]$ zmcontrol status
    Host mail.birchlake.ath.cx
    antispam Running
    antivirus Running
    ldap Running
    logger Running
    mailbox Running
    memcached Running
    mta Running
    snmp Running
    spell Running
    stats Running
    zmconfigd Running
    [zimbra@mail root]$ zmprov gacf
    ERROR: zclient.IO_ERROR (invoke Read timed out, server: localhost) (cause: java.net.SocketTimeoutException Read timed out)
    I think the problem lies either with postfix or with the thread limit. The postfix error I get comes from the zimbra.log and runs when I'm running the stop commands.

    Dec 19 01:22:03 mail zmmailboxdmgr[12583]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist
    Dec 19 01:22:03 mail zmmailboxdmgr[12583]: assuming no other instance is running
    Dec 19 01:22:03 mail zmmailboxdmgr[12583]: file /opt/zimbra/log/zmmailboxd.pid does not exist
    Dec 19 01:22:03 mail zmmailboxdmgr[12583]: assuming no other instance is running
    Dec 19 01:22:03 mail zmmailboxdmgr[12583]: no manager process is running
    Dec 19 01:22:19 mail zimbramon[12742]: 12742:info: Stopping services initiated by zmcontrol
    Dec 19 01:22:19 mail zimbramon[12742]: 12742:info: Stopping stats via zmcontrol
    Dec 19 01:22:20 mail zimbramon[12742]: 12742:info: Stopping mta via zmcontrol
    Dec 19 01:22:20 mail postfix/postfix-script[12841]: fatal: the Postfix mail system is not running
    Dec 19 01:22:20 mail zimbramon[12742]: 12742:info: Stopping spell via zmcontrol
    Dec 19 01:22:20 mail zimbramon[12742]: 12742:info: Stopping snmp via zmcontrol
    Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping cbpolicyd via zmcontrol
    Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping archiving via zmcontrol
    Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping antivirus via zmcontrol
    Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping antispam via zmcontrol
    Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping imapproxy via zmcontrol
    Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping memcached via zmcontrol
    Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping mailbox via zmcontrol

    The thread error comes from the mailbox.log

    2011-12-19 00:08:15,300 WARN [LmtpServer] [] TcpServer/7025 - Thread pool is 105% utilized. 21 out of 20 threads in use.
    2011-12-19 00:08:15,300 ERROR [LmtpServer] [] TcpServer/7025 - cannot handle connection; thread pool exhausted
    java.util.concurrent.RejectedExecutionException
    at java.util.concurrent.ThreadPoolExecutor$AbortPolic y.rejectedExecution(ThreadPoolExecutor.java:1768)
    at java.util.concurrent.ThreadPoolExecutor.reject(Thr eadPoolExecutor.java:767)
    at java.util.concurrent.ThreadPoolExecutor.execute(Th readPoolExecutor.java:658)
    at com.zimbra.cs.tcpserver.TcpServer.run(TcpServer.ja va:202)
    at java.lang.Thread.run(Thread.java:662)

    From the mailbox.log file

    Dec 18 22:24:54 mail postfix/lmtp[8895]: BC7F4BD3D07: to=<webmaster@birchlake.ath.cx>, relay=mail.birchlake.ath.cx[192.168.1.25]:7025, delay=944, delays=944/0.01/0/0, dsn=4.0.0, status=deferred (host mail.birchlake.ath.cx[192.168.1.25] refused to talk to me: 421 mail.birchlake.ath.cx Zimbra LMTP server closing connection; service busy)

    Dec 19 03:01:42 mail postfix/error[2460]: 3F829BD3206: to=<admin@birchlake.ath.cx>, relay=none, delay=0.04, delays=0.03/0/0/0.01, dsn=4.4.2, status=deferred (delivery temporarily suspended: conversation with mail.birchlake.ath.cx[192.168.1.25] timed out while sending end of data -- message may be sent more than once)




    I am unable to get into the zimbra admin console to change the thread limit nor am I able to change it via console command.

    here's some config stuff just in case it might help


    [zimbra@mail root]$ cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1 localhost.localdomain localhost
    192.168.1.25 mail.birchlake.ath.cx mail
    ::1 localhost6.localdomain6 localhost6
    [zimbra@mail root]$ cat /etc/resolv.conf
    search mail.birchlake.ath.cx birchlake.ath.cx
    nameserver localhost
    nameserver 192.168.1.25
    nameserver 8.8.8.8
    nameserver 8.8.4.4
    [zimbra@mail root]$ dig mail.birchlake.ath.cx mx
    ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5_7.1 <<>> mail.birchlake.ath.cx mx
    ;; global options: printcmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40288
    ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

    ;; QUESTION SECTION:
    ;mail.birchlake.ath.cx. IN MX

    ;; AUTHORITY SECTION:
    birchlake.ath.cx. 38400 IN SOA mail. admin.birchlake.ath.cx. 1324277785 10800 3600 604800 38400

    ;; Query time: 0 msec
    ;; SERVER: 192.168.1.25#53(192.168.1.25)
    ;; WHEN: Mon Dec 19 02:57:51 2011
    ;; MSG SIZE rcvd: 85
    [zimbra@mail root]$ dig mail.birchlake.ath.cx any
    ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5_7.1 <<>> mail.birchlake.ath.cx any
    ;; global options: printcmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37242
    ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

    ;; QUESTION SECTION:
    ;mail.birchlake.ath.cx. IN ANY

    ;; ANSWER SECTION:
    mail.birchlake.ath.cx. 38400 IN A 192.168.1.25

    ;; AUTHORITY SECTION:
    birchlake.ath.cx. 38400 IN NS mail.

    ;; Query time: 0 msec
    ;; SERVER: 192.168.1.25#53(192.168.1.25)
    ;; WHEN: Mon Dec 19 02:58:11 2011
    ;; MSG SIZE rcvd: 73
    [zimbra@mail root]$ host 'birchlake.ath.cx'
    birchlake.ath.cx mail is handled by 10 mail.birchlake.ath.cx.
    [zimbra@mail root]$ hostname
    mail.birchlake.ath.cx
    [zimbra@mail init.d]$ postfix check
    postfix/postfix-script: warning: not owned by root: /opt/zimbra/postfix-2.7.4.2z/conf/master.cf.in

    If I do postfix status it comes back with nothing, but postfix start and stop seem like they do something....

    Any help or advice would be greatly appreciated, I'm at the end of my rope and about to take a hammer to the computer.
    thanks!
    Last edited by Eclipse; 12-19-2011 at 04:12 PM.

  2. #2
    Eclipse is offline Senior Member
    Join Date
    Jun 2011
    Posts
    73
    Rep Power
    3

    Default

    Quote Originally Posted by vavai View Post
    Hi,

    1. If this is a single server, why you installing memcached package?
    2. It seems that your java consumes a lot of memory. How about :

    Code:
    su - zimbra
    zmcontrol stop
    exit
    ps aux | grep zimbra
    Kill all services related to Zimbra (especially Java instance) and then restart the services

    Code:
    su - zimbra
    zmcontrol start
    If the above doesn't help either, please feel free to paste what the result of all these command.

    Thank you very much for the quick reply, I believe the memcached came with once it was installed, here is the output you requested.

    [zimbra@mail root]$ zmcontrol stop

    Host mail.birchlake.ath.cx
    Stopping stats...Done.
    Stopping mta...Done.
    Stopping spell...Done.
    Stopping snmp...Done.
    Stopping cbpolicyd...Done.
    Stopping archiving...Done.
    Stopping antivirus...Done.
    Stopping antispam...Done.
    Stopping imapproxy...Done.
    Stopping memcached...Done.
    Stopping mailbox...Done.
    Stopping logger...Done.
    Stopping zmconfigd...Done.
    Stopping ldap...Done.
    [zimbra@mail root]$ exit
    [root@mail ~]# ps aux | grep zimbra
    root 13798 0.0 0.0 5016 1284 pts/0 S 03:33 0:00 su zimbra
    zimbra 13799 0.0 0.0 4636 1432 pts/0 S 03:33 0:00 bash
    root 16077 0.0 0.0 4016 696 pts/2 S+ 05:07 0:00 grep zimbra
    root 18417 0.0 0.0 5016 1280 pts/0 S 03:40 0:00 su zimbra
    zimbra 18418 0.0 0.0 4636 1428 pts/0 S 03:40 0:00 bash
    root 22011 0.0 0.0 5016 1276 pts/0 S 03:51 0:00 su zimbra
    zimbra 22012 0.0 0.0 4636 1428 pts/0 S 03:51 0:00 bash
    root 24716 0.0 0.0 5016 1284 pts/2 S 03:56 0:00 su zimbra
    zimbra 24717 0.0 0.0 4636 1432 pts/2 S 03:56 0:00 bash
    root 26444 0.0 0.0 5016 1284 pts/0 S 02:46 0:00 su zimbra
    zimbra 26445 0.0 0.0 4636 1436 pts/0 S 02:46 0:00 bash
    [root@mail ~]# killall zimbra
    zimbra: no process killed
    [root@mail ~]# su - zimbra

    [zimbra@mail ~]$ zmcontrol start
    Host mail.birchlake.ath.cx
    Starting ldap...Done.
    Starting zmconfigd...Done.
    Starting logger...Done.
    Starting mailbox...Done.
    Starting memcached...Done.
    Starting antispam...Done.
    Starting antivirus...Done.
    Starting snmp...Done.
    Starting spell...Done.
    Starting mta...Done.
    Starting stats...Done.
    Still gives the same problem, do you by chance know what those zimbra processes are once its shutdown from the grep?

  3. #3
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,201
    Rep Power
    56

    Default

    The dig commands you're using are incorrect, you have this (you're using a subdomain):

    Code:
    dig mail.birchlake.ath.cx mx
    You should be using this (your domain name):

    Code:
    dig birchlake.ath.cx mx
    You also have memcached enabled and running and and it shouldn't be in use. You should stop it and disable it, search the forums for details on how to do that.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  4. #4
    Eclipse is offline Senior Member
    Join Date
    Jun 2011
    Posts
    73
    Rep Power
    3

    Default

    Quote Originally Posted by phoenix View Post
    The dig commands you're using are incorrect, you have this (you're using a subdomain):

    Code:
    dig mail.birchlake.ath.cx mx
    You should be using this (your domain name):

    Code:
    dig birchlake.ath.cx mx
    You also have memcached enabled and running and and it shouldn't be in use. You should stop it and disable it, search the forums for details on how to do that.
    Thank you for the quick reply, here is the updated dig command

    [root@mail ~]# dig birchlake.ath.cx mx
    ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5_7.1 <<>> birchlake.ath.cx mx
    ;; global options: printcmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56365
    ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

    ;; QUESTION SECTION:
    ;birchlake.ath.cx. IN MX

    ;; ANSWER SECTION:
    birchlake.ath.cx. 38400 IN MX 10 mail.birchlake.ath.cx.

    ;; AUTHORITY SECTION:
    birchlake.ath.cx. 38400 IN NS mail.

    ;; ADDITIONAL SECTION:
    mail.birchlake.ath.cx. 38400 IN A 192.168.1.25

    ;; Query time: 0 msec
    ;; SERVER: 192.168.1.25#53(192.168.1.25)
    ;; WHEN: Mon Dec 19 05:23:41 2011
    ;; MSG SIZE rcvd: 89
    Heres what I tried for disabling the memcached

    [zimbra@mail ~]$ zmmemcachedctl stop
    Stopping memcached...done
    [zimbra@mail ~]$ zmprov ms `zmhostname` -zimbraServiceEnabled memcached
    ERROR: zclient.IO_ERROR (invoke Read timed out, server: localhost) (cause: java.net.SocketTimeoutException Read timed out)
    [zimbra@mail ~]$ zmprov ms mail.birchlake.ath.cx -zimbraServiceEnabled memcached
    ERROR: zclient.IO_ERROR (invoke Read timed out, server: localhost) (cause: java.net.SocketTimeoutException Read timed out)
    Also tried just birchlake.ath.cx and it did the same as the 2nd one, I don't think I can disable it.

    It also started back up with a zmcontrol stop and a start...
    Last edited by Eclipse; 12-19-2011 at 04:36 AM.

  5. #5
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,201
    Rep Power
    56

    Default

    Have you checked that SElinux is disabled and I mean disabled not permissive mode.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  6. #6
    Eclipse is offline Senior Member
    Join Date
    Jun 2011
    Posts
    73
    Rep Power
    3

    Default

    Quote Originally Posted by phoenix View Post
    Have you checked that SElinux is disabled and I mean disabled not permissive mode.
    Firewall: Disabled.
    SELinux Setting: Disabled.

    Both already like that.

    Do you know if there is a way have Zimbra skip loading an entire domain? I think I might have narrowed down the problem to a calendar becoming to big.... One of the users had let their computer go over the weekend populating a calendar on their computer with over 50,000 entries, and I think the server gave up trying, is there a way to skip it from loading the calendar? It may have gotten corrupted...

    Or do you know of any specific diagnostic problems to narrow this down even more?

    Could I take out a folder in the store folder? Would that screw up things?

  7. #7
    Eclipse is offline Senior Member
    Join Date
    Jun 2011
    Posts
    73
    Rep Power
    3

    Default

    Fixed the problem, was a permission issue, found it in this thread - [SOLVED] lmtp crash since update to 5.0.9

    Run all of these as root

    /etc/init.d/zimbra stop
    chown -Rf zimbra:zimbra /opt/zimbra
    chmod -Rf 755 /opt/zimbra
    /opt/zimbra/libexec/zmfixperms
    su - zimbra
    zmcontrol start

    Mark as solve or close if you'd like, web console, email clients, and webmail is now working as it should.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. HELIX 7.1.1 ported to FreeBSD
    By solko in forum Installation
    Replies: 22
    Last Post: 05-21-2013, 04:37 AM
  2. Backspace causing threads to disappear
    By rhutch in forum General Questions
    Replies: 1
    Last Post: 10-22-2012, 07:03 PM
  3. Zimbra don't start
    By cadinor in forum Migration
    Replies: 1
    Last Post: 08-04-2011, 05:27 PM
  4. zmconfigd exiting: Hung threads detected
    By skyflyer in forum Administrators
    Replies: 3
    Last Post: 06-28-2011, 04:27 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •