Clients stopped receiving emails

[egadinc]

Still down. MTA still collecting email and mailboxes still not getting mail.

[egadinc]

MTA is still running and email is still arriving. Nothing being transferred to mailboxes.


All the email delivery is being deferred.

Scanning message threads, I'm guessing that the mailbox index's are damaged or need reindexing.

I've tried the following:

Shutdown Zimbra and:
chown -R zimbra:zimbra /opt/zimbra/st*
(finished with no errors)

./zmfixperms -extended -verbose
(finished with no errors)

Based on another recent thread, I tried getting convertd running:
su - zimbra
zmprov gs `zmhostname` | grep zimbraServiceInstalled
zmprov gs `zmhostname` | grep zimbraServiceEnabled
(convertd not listed)

zmprov ms `zmhostname` -- +zimbraServiceEnabled convertd
zmconvertctl start
(now convertd listed as service, but not startable)
(zmconvertctl doesn't exist as a command)

I've started Zimbra, went into Admin Console, and reindexed a few accounts.

Result: nothing has changed. Looks like reindexing is not the problem.

NOTE: a week before this problem, users using their webmail UI, were unable to SEARCH. No results came up for every attempt, even when looking at search targets in the email titles, addresses, contents, etc. Perhaps there is a clue here.

[egadinc]

Just in case there is a question about DNS and MX records:

[zimbra@mail libexec]$ zmcontrol start
Host mail.egad.com
Starting ldap...Done.
Starting logger...Done.
Starting convertd...skipped.
/opt/zimbra/bin/zmconvertctl missing or not executable.
Starting mailbox...Done.
Starting memcached...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.


[zimbra@mail libexec]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
#127.0.0.1 mail.egad.com mail localhost.localdomain localhost
127.0.0.1 localhost.localdomain localhost
76.246.226.114 mail.egad.com mail
::1 localhost6.localdomain6 localhost6

[zimbra@mail libexec]$ cat /etc/resolv.conf
search egad.com
nameserver 76.246.226.118
nameserver 76.246.226.118
[zimbra@mail libexec]$ dig egad.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> egad.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41595
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;egad.com. IN A

;; ANSWER SECTION:
egad.com. 1800 IN A 76.246.226.113

;; Query time: 45 msec
;; SERVER: 76.246.226.118#53(76.246.226.118)
;; WHEN: Sun Dec 13 16:17:10 2009
;; MSG SIZE rcvd: 42

[zimbra@mail libexec]$ dig mail.egad.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> mail.egad.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2041
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;mail.egad.com. IN A

;; ANSWER SECTION:
mail.egad.com. 1622 IN A 76.246.226.114

;; Query time: 29 msec
;; SERVER: 76.246.226.118#53(76.246.226.118)
;; WHEN: Sun Dec 13 16:17:32 2009
;; MSG SIZE rcvd: 47

[zimbra@mail libexec]$ dig mail.egad.com any

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> mail.egad.com any
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36347
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;mail.egad.com. IN ANY

;; ANSWER SECTION:
mail.egad.com. 1308 IN A 76.246.226.114

;; Query time: 27 msec
;; SERVER: 76.246.226.118#53(76.246.226.118)
;; WHEN: Sun Dec 13 16:17:44 2009
;; MSG SIZE rcvd: 47

[zimbra@mail libexec]$ host mail.egad.com
mail.egad.com has address 76.246.226.114
[zimbra@mail libexec]$

[egadinc]

mailbox.log still delivering errors at a rapid rate (1.2GB for today so far)

--------------------------------------------------------------------------------------------------------------------------
2009-12-13 17:15:33,266 WARN [LmtpServer-338] [name=joe.vieira@masspolice.com;mid=41;ip=76.246.22 6.114;] indexadd - Caught exception while indexing message id 0 - indexing blocked. Possibly corrupt index?
java.lang.NullPointerException
at com.zimbra.cs.mailbox.IndexHelper.indexingPartOfEn dTransaction(IndexHelper.java:969)
at com.zimbra.cs.mailbox.Mailbox.endTransaction(Mailb ox.java:6669)
at com.zimbra.cs.mailbox.IndexHelper.indexItemList(In dexHelper.java:890)
at com.zimbra.cs.mailbox.IndexHelper.indexDeferredIte msInternal(IndexHelper.java:360)
at com.zimbra.cs.mailbox.IndexHelper.indexDeferredIte ms(IndexHelper.java:278)
at com.zimbra.cs.mailbox.IndexHelper.maybeIndexDeferr edItems(IndexHelper.java:234)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:4233)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:4226)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:4186)
at com.zimbra.cs.filter.IncomingMessageHandler.addMes sage(IncomingMessageHandler.java:116)
at com.zimbra.cs.filter.IncomingMessageHandler.implic itKeep(IncomingMessageHandler.java:109)
at com.zimbra.cs.filter.ZimbraMailAdapter.doDefaultFi ling(ZimbraMailAdapter.java:284)
at com.zimbra.cs.filter.RuleManager.applyRulesToIncom ingMessage(RuleManager.java:363)
at com.zimbra.cs.filter.RuleManager.applyRulesToIncom ingMessage(RuleManager.java:311)
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver MessageToLocalMailboxes(ZimbraLmtpBackend.java:425 )
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver MessageToLocalMailboxes(ZimbraLmtpBackend.java:257 )
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver (ZimbraLmtpBackend.java:162)
at com.zimbra.cs.lmtpserver.LmtpHandler.processMessag eData(LmtpHandler.java:365)
at com.zimbra.cs.lmtpserver.TcpLmtpHandler.continueDA TA(TcpLmtpHandler.java:72)
at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHa ndler.java:350)
at com.zimbra.cs.lmtpserver.LmtpHandler.processComman d(LmtpHandler.java:170)
at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCom mand(TcpLmtpHandler.java:66)
at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:175)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:132)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
2009-12-13 17:15:33,267 WARN [LmtpServer-338] [name=joe.vieira@masspolice.com;mid=41;ip=76.246.22 6.114;] indexadd - Caught exception while flushing index - indexing blocked. Possibly corrupt index?
java.lang.NullPointerException
at com.zimbra.cs.mailbox.IndexHelper.indexingPartOfEn dTransaction(IndexHelper.java:1019)
at com.zimbra.cs.mailbox.Mailbox.endTransaction(Mailb ox.java:6669)
at com.zimbra.cs.mailbox.IndexHelper.indexItemList(In dexHelper.java:890)
at com.zimbra.cs.mailbox.IndexHelper.indexDeferredIte msInternal(IndexHelper.java:360)
at com.zimbra.cs.mailbox.IndexHelper.indexDeferredIte ms(IndexHelper.java:278)
at com.zimbra.cs.mailbox.IndexHelper.maybeIndexDeferr edItems(IndexHelper.java:234)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:4233)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:4226)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:4186)
at com.zimbra.cs.filter.IncomingMessageHandler.addMes sage(IncomingMessageHandler.java:116)
at com.zimbra.cs.filter.IncomingMessageHandler.implic itKeep(IncomingMessageHandler.java:109)
at com.zimbra.cs.filter.ZimbraMailAdapter.doDefaultFi ling(ZimbraMailAdapter.java:284)
at com.zimbra.cs.filter.RuleManager.applyRulesToIncom ingMessage(RuleManager.java:363)
at com.zimbra.cs.filter.RuleManager.applyRulesToIncom ingMessage(RuleManager.java:311)
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver MessageToLocalMailboxes(ZimbraLmtpBackend.java:425 )
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver MessageToLocalMailboxes(ZimbraLmtpBackend.java:257 )
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver (ZimbraLmtpBackend.java:162)
at com.zimbra.cs.lmtpserver.LmtpHandler.processMessag eData(LmtpHandler.java:365)
at com.zimbra.cs.lmtpserver.TcpLmtpHandler.continueDA TA(TcpLmtpHandler.java:72)
at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHa ndler.java:350)
at com.zimbra.cs.lmtpserver.LmtpHandler.processComman d(LmtpHandler.java:170)
at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCom mand(TcpLmtpHandler.java:66)
at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:175)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:132)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
[root@mail log]#
------------------------------------------------------------------------------------------------------------------------------

[egadinc]

MTA still collecting emails, but mailboxes empty.
mailbox.log:
__________________________________________________ ________
2009-12-14 00:03:27,891 INFO [LmtpServer-2144] [name=joe.vieira@masspolice.com;mid=41;ip=76.246.22 6.114;] lmtp - 451 4.0.0 Temporary message delivery failure try again (DATA)
2009-12-14 00:03:30,034 INFO [LmtpServer-2144] [name=joe.vieira@masspolice.com;mid=41;ip=76.246.22 6.114;] lmtp - disconnected without quit
2009-12-14 00:03:30,034 INFO [LmtpServer-2144] [] ProtocolHandler - Handler exiting normally
2009-12-14 00:03:40,328 INFO [LmtpServer-2160] [ip=76.246.226.114;] lmtp - Delivering message: size=12596 bytes, nrcpts=1, sender=matthewsog@santacherry.com, msgid=<20091214004445.ptvjdhjgqs@mx1.santacherry.c om>
2009-12-14 00:03:40,335 WARN [LmtpServer-2160] [name=peter.zaharkiv@egad.com;mid=3;ip=76.246.226.1 14;] lmtp - try again for message peter.zaharkiv@egad.com: exception occurred
java.lang.NullPointerException
at com.zimbra.cs.mailbox.IndexHelper.generateIndexId( IndexHelper.java:178)
at com.zimbra.cs.mailbox.Mailbox.generateIndexId(Mail box.java:1028)
at com.zimbra.cs.mailbox.Message.createInternal(Messa ge.java:417)
at com.zimbra.cs.mailbox.Message.create(Message.java: 360)
at com.zimbra.cs.mailbox.Mailbox.addMessageInternal(M ailbox.java:4446)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:4293)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:4226)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:4186)
at com.zimbra.cs.filter.IncomingMessageHandler.addMes sage(IncomingMessageHandler.java:116)
at com.zimbra.cs.filter.IncomingMessageHandler.implic itKeep(IncomingMessageHandler.java:109)
at com.zimbra.cs.filter.ZimbraMailAdapter.doDefaultFi ling(ZimbraMailAdapter.java:284)
at com.zimbra.cs.filter.RuleManager.applyRulesToIncom ingMessage(RuleManager.java:363)
at com.zimbra.cs.filter.RuleManager.applyRulesToIncom ingMessage(RuleManager.java:311)
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver MessageToLocalMailboxes(ZimbraLmtpBackend.java:425 )
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver MessageToLocalMailboxes(ZimbraLmtpBackend.java:257 )
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver (ZimbraLmtpBackend.java:162)
at com.zimbra.cs.lmtpserver.LmtpHandler.processMessag eData(LmtpHandler.java:365)
at com.zimbra.cs.lmtpserver.TcpLmtpHandler.continueDA TA(TcpLmtpHandler.java:72)
at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHa ndler.java:350)
at com.zimbra.cs.lmtpserver.LmtpHandler.processComman d(LmtpHandler.java:170)
at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCom mand(TcpLmtpHandler.java:66)
at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:175)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:132)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
2009-12-14 00:03:40,335 INFO [LmtpServer-2160] [name=peter.zaharkiv@egad.com;mid=3;ip=76.246.226.1 14;] lmtp - 451 4.0.0 Temporary message delivery failure try again (DATA)
2009-12-14 00:03:40,34
------------------------------------------------------------------------------------------------------------------------
Can't find reason for these 'Temporary Errors'

[egadinc]

Please, is there any thoughts on how to fix this problem???

[ArcaneMagus]

This might be a bit drastic, but you could try this:

• Setting all accounts to maintenance mode (set the domain's status).
• Shutting down Zimbra.
• Renaming /opt/zimbra/index/0 to /opt/zimbra/index/0-back or something
• Starting Zimbra and re-indexing all accounts.

This should recreate the index from scratch for all accounts.


Also convertd is a NE only feature for converting attachments to HTML, you should remove that entry from your configuration.