Server no longer working, 21/20 thread limit reached?

[Eclipse]

Hey guys, I'm having some trouble. Over the weekend some complete idiot and/or super genius got in to the server and screwed a lot of stuff up. I believe that I've fixed the bulk of it but there are still a few things I'm having trouble with. As of right now, nothing works no admin console, web e-mail, client or otherwise.

CentOS 5.6
Zimbra 7.1.2

As stated earlier I cannot even get into the admin console to changes things, it just sits at connected, however I was able to telnet in on the server itself and send an email to an external domain, so it feels like the main receiving process / program is no longer working or broken.

Firewall has been turned off for testing at the moment so its not that.

One more thing to note, we recently added around 12 new accounts with a new domain name, one account has a 6GB email account with about 100,000 or so emails in the inbox, if this is whats crashing or preventing the server, is there a way to make Zimbra skip an email domain to prevent it from being loaded or something of the sort to see if that's the problem?

[zimbra@mail root]$ zmcontrol status

Host mail.birchlake.ath.cx
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
memcached Running
mta Running
snmp Running
spell Running
stats Running
zmconfigd Running

[zimbra@mail root]$ zmprov gacf

ERROR: zclient.IO_ERROR (invoke Read timed out, server: localhost) (cause: java.net.SocketTimeoutException Read timed out)

I think the problem lies either with postfix or with the thread limit. The postfix error I get comes from the zimbra.log and runs when I'm running the stop commands.

Dec 19 01:22:03 mail zmmailboxdmgr[12583]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist
Dec 19 01:22:03 mail zmmailboxdmgr[12583]: assuming no other instance is running
Dec 19 01:22:03 mail zmmailboxdmgr[12583]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Dec 19 01:22:03 mail zmmailboxdmgr[12583]: assuming no other instance is running
Dec 19 01:22:03 mail zmmailboxdmgr[12583]: no manager process is running
Dec 19 01:22:19 mail zimbramon[12742]: 12742:info: Stopping services initiated by zmcontrol
Dec 19 01:22:19 mail zimbramon[12742]: 12742:info: Stopping stats via zmcontrol
Dec 19 01:22:20 mail zimbramon[12742]: 12742:info: Stopping mta via zmcontrol
Dec 19 01:22:20 mail postfix/postfix-script[12841]: fatal: the Postfix mail system is not running
Dec 19 01:22:20 mail zimbramon[12742]: 12742:info: Stopping spell via zmcontrol
Dec 19 01:22:20 mail zimbramon[12742]: 12742:info: Stopping snmp via zmcontrol
Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping cbpolicyd via zmcontrol
Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping archiving via zmcontrol
Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping antivirus via zmcontrol
Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping antispam via zmcontrol
Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping imapproxy via zmcontrol
Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping memcached via zmcontrol
Dec 19 01:22:21 mail zimbramon[12742]: 12742:info: Stopping mailbox via zmcontrol

The thread error comes from the mailbox.log

2011-12-19 00:08:15,300 WARN [LmtpServer] [] TcpServer/7025 - Thread pool is 105% utilized. 21 out of 20 threads in use.
2011-12-19 00:08:15,300 ERROR [LmtpServer] [] TcpServer/7025 - cannot handle connection; thread pool exhausted
java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolic y.rejectedExecution(ThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(Thr eadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(Th readPoolExecutor.java:658)
at com.zimbra.cs.tcpserver.TcpServer.run(TcpServer.ja va:202)
at java.lang.Thread.run(Thread.java:662)

From the mailbox.log file

Dec 18 22:24:54 mail postfix/lmtp[8895]: BC7F4BD3D07: to=<webmaster@birchlake.ath.cx>, relay=mail.birchlake.ath.cx[192.168.1.25]:7025, delay=944, delays=944/0.01/0/0, dsn=4.0.0, status=deferred (host mail.birchlake.ath.cx[192.168.1.25] refused to talk to me: 421 mail.birchlake.ath.cx Zimbra LMTP server closing connection; service busy)

Dec 19 03:01:42 mail postfix/error[2460]: 3F829BD3206: to=<admin@birchlake.ath.cx>, relay=none, delay=0.04, delays=0.03/0/0/0.01, dsn=4.4.2, status=deferred (delivery temporarily suspended: conversation with mail.birchlake.ath.cx[192.168.1.25] timed out while sending end of data -- message may be sent more than once)




I am unable to get into the zimbra admin console to change the thread limit nor am I able to change it via console command.

here's some config stuff just in case it might help


[zimbra@mail root]$ cat /etc/hosts

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.1.25 mail.birchlake.ath.cx mail
::1 localhost6.localdomain6 localhost6

[zimbra@mail root]$ cat /etc/resolv.conf

search mail.birchlake.ath.cx birchlake.ath.cx
nameserver localhost
nameserver 192.168.1.25
nameserver 8.8.8.8
nameserver 8.8.4.4

[zimbra@mail root]$ dig mail.birchlake.ath.cx mx

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5_7.1 <<>> mail.birchlake.ath.cx mx
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40288
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;mail.birchlake.ath.cx. IN MX

;; AUTHORITY SECTION:
birchlake.ath.cx. 38400 IN SOA mail. admin.birchlake.ath.cx. 1324277785 10800 3600 604800 38400

;; Query time: 0 msec
;; SERVER: 192.168.1.25#53(192.168.1.25)
;; WHEN: Mon Dec 19 02:57:51 2011
;; MSG SIZE rcvd: 85

[zimbra@mail root]$ dig mail.birchlake.ath.cx any

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5_7.1 <<>> mail.birchlake.ath.cx any
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37242
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;mail.birchlake.ath.cx. IN ANY

;; ANSWER SECTION:
mail.birchlake.ath.cx. 38400 IN A 192.168.1.25

;; AUTHORITY SECTION:
birchlake.ath.cx. 38400 IN NS mail.

;; Query time: 0 msec
;; SERVER: 192.168.1.25#53(192.168.1.25)
;; WHEN: Mon Dec 19 02:58:11 2011
;; MSG SIZE rcvd: 73

[zimbra@mail root]$ host 'birchlake.ath.cx'

birchlake.ath.cx mail is handled by 10 mail.birchlake.ath.cx.

[zimbra@mail root]$ hostname

mail.birchlake.ath.cx

[zimbra@mail init.d]$ postfix check

postfix/postfix-script: warning: not owned by root: /opt/zimbra/postfix-2.7.4.2z/conf/master.cf.in

If I do postfix status it comes back with nothing, but postfix start and stop seem like they do something....

Any help or advice would be greatly appreciated, I'm at the end of my rope and about to take a hammer to the computer.
thanks!

[vavai]

Hi,

1. If this is a single server, why you installing memcached package?
2. It seems that your java consumes a lot of memory. How about :

su - zimbra
zmcontrol stop
exit
ps aux | grep zimbra

Kill all services related to Zimbra (especially Java instance) and then restart the services

su - zimbra
zmcontrol start

If the above doesn't help either, please feel free to paste what the result of all these command.

[Eclipse]

Originally Posted by vavai

Hi,

1. If this is a single server, why you installing memcached package?
2. It seems that your java consumes a lot of memory. How about :

su - zimbra
zmcontrol stop
exit
ps aux | grep zimbra

Kill all services related to Zimbra (especially Java instance) and then restart the services

su - zimbra
zmcontrol start

If the above doesn't help either, please feel free to paste what the result of all these command.

Thank you very much for the quick reply, I believe the memcached came with once it was installed, here is the output you requested.

[zimbra@mail root]$ zmcontrol stop

Host mail.birchlake.ath.cx
Stopping stats...Done.
Stopping mta...Done.
Stopping spell...Done.
Stopping snmp...Done.
Stopping cbpolicyd...Done.
Stopping archiving...Done.
Stopping antivirus...Done.
Stopping antispam...Done.
Stopping imapproxy...Done.
Stopping memcached...Done.
Stopping mailbox...Done.
Stopping logger...Done.
Stopping zmconfigd...Done.
Stopping ldap...Done.

[zimbra@mail root]$ exit
[root@mail ~]# ps aux | grep zimbra

root 13798 0.0 0.0 5016 1284 pts/0 S 03:33 0:00 su zimbra
zimbra 13799 0.0 0.0 4636 1432 pts/0 S 03:33 0:00 bash
root 16077 0.0 0.0 4016 696 pts/2 S+ 05:07 0:00 grep zimbra
root 18417 0.0 0.0 5016 1280 pts/0 S 03:40 0:00 su zimbra
zimbra 18418 0.0 0.0 4636 1428 pts/0 S 03:40 0:00 bash
root 22011 0.0 0.0 5016 1276 pts/0 S 03:51 0:00 su zimbra
zimbra 22012 0.0 0.0 4636 1428 pts/0 S 03:51 0:00 bash
root 24716 0.0 0.0 5016 1284 pts/2 S 03:56 0:00 su zimbra
zimbra 24717 0.0 0.0 4636 1432 pts/2 S 03:56 0:00 bash
root 26444 0.0 0.0 5016 1284 pts/0 S 02:46 0:00 su zimbra
zimbra 26445 0.0 0.0 4636 1436 pts/0 S 02:46 0:00 bash

[root@mail ~]# killall zimbra

zimbra: no process killed

[root@mail ~]# su - zimbra

[zimbra@mail ~]$ zmcontrol start

Host mail.birchlake.ath.cx
Starting ldap...Done.
Starting zmconfigd...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting memcached...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.

Still gives the same problem, do you by chance know what those zimbra processes are once its shutdown from the grep?

[phoenix]

The dig commands you're using are incorrect, you have this (you're using a subdomain):

dig mail.birchlake.ath.cx mx

You should be using this (your domain name):

dig birchlake.ath.cx mx

You also have memcached enabled and running and and it shouldn't be in use. You should stop it and disable it, search the forums for details on how to do that.

[Eclipse]

Originally Posted by phoenix

The dig commands you're using are incorrect, you have this (you're using a subdomain):

dig mail.birchlake.ath.cx mx

You should be using this (your domain name):

dig birchlake.ath.cx mx

You also have memcached enabled and running and and it shouldn't be in use. You should stop it and disable it, search the forums for details on how to do that.

Thank you for the quick reply, here is the updated dig command

[root@mail ~]# dig birchlake.ath.cx mx

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5_7.1 <<>> birchlake.ath.cx mx
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56365
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;birchlake.ath.cx. IN MX

;; ANSWER SECTION:
birchlake.ath.cx. 38400 IN MX 10 mail.birchlake.ath.cx.

;; AUTHORITY SECTION:
birchlake.ath.cx. 38400 IN NS mail.

;; ADDITIONAL SECTION:
mail.birchlake.ath.cx. 38400 IN A 192.168.1.25

;; Query time: 0 msec
;; SERVER: 192.168.1.25#53(192.168.1.25)
;; WHEN: Mon Dec 19 05:23:41 2011
;; MSG SIZE rcvd: 89

Heres what I tried for disabling the memcached

[zimbra@mail ~]$ zmmemcachedctl stop

Stopping memcached...done

[zimbra@mail ~]$ zmprov ms `zmhostname` -zimbraServiceEnabled memcached

ERROR: zclient.IO_ERROR (invoke Read timed out, server: localhost) (cause: java.net.SocketTimeoutException Read timed out)

[zimbra@mail ~]$ zmprov ms mail.birchlake.ath.cx -zimbraServiceEnabled memcached

ERROR: zclient.IO_ERROR (invoke Read timed out, server: localhost) (cause: java.net.SocketTimeoutException Read timed out)

Also tried just birchlake.ath.cx and it did the same as the 2nd one, I don't think I can disable it.

It also started back up with a zmcontrol stop and a start...

[phoenix]

Have you checked that SElinux is disabled and I mean disabled not permissive mode.

[Eclipse]

Originally Posted by phoenix

Have you checked that SElinux is disabled and I mean disabled not permissive mode.

Firewall: Disabled.
SELinux Setting: Disabled.

Both already like that.

Do you know if there is a way have Zimbra skip loading an entire domain? I think I might have narrowed down the problem to a calendar becoming to big.... One of the users had let their computer go over the weekend populating a calendar on their computer with over 50,000 entries, and I think the server gave up trying, is there a way to skip it from loading the calendar? It may have gotten corrupted...

Or do you know of any specific diagnostic problems to narrow this down even more?

Could I take out a folder in the store folder? Would that screw up things?

[Eclipse]

Fixed the problem, was a permission issue, found it in this thread - [SOLVED] lmtp crash since update to 5.0.9

Run all of these as root

/etc/init.d/zimbra stop
chown -Rf zimbra:zimbra /opt/zimbra
chmod -Rf 755 /opt/zimbra
/opt/zimbra/libexec/zmfixperms
su - zimbra
zmcontrol start

Mark as solve or close if you'd like, web console, email clients, and webmail is now working as it should.

[vavai]

Originally Posted by Eclipse

Fixed the problem, was a permission issue, found it in this thread - [SOLVED] lmtp crash since update to 5.0.9

Run all of these as root

/etc/init.d/zimbra stop
chown -Rf zimbra:zimbra /opt/zimbra
chmod -Rf 755 /opt/zimbra
/opt/zimbra/libexec/zmfixperms
su - zimbra
zmcontrol start

Mark as solve or close if you'd like, web console, email clients, and webmail is now working as it should.

Glad to hear that your problem is all about permission. As Thread Starter, you may mark this thread as Solved to helps anyone who facing similar problem.