4.0 RC1 Deferred queue problem

[neilmc]

I'm not sure if this is related to 4.0 RC1 or if it would have happened anyway.

Before the upgrade I only ran into the redo.log permissions problem (Debian 3.1) if the server was shut down without first ..... zmcontrol stop

Now it seems to happen sometimes just by running zmcontrol stop

After such an occurance, I noticed 6 emails in the Deferred queue & requeuing them doesn't clear them. Other email is being delivered successfully while this is happening.

The error for all of them is

Reason: host (hostname ip address) said 451 4.0.0 temporary message delivery failure, try again (in reply to end of data command)

If I look in /var/log/zimbra.log just after requeuing a message I see "Requeued: 1 message"

but nothing else on that message in the log before it goes back to deferred.

This happened earlier & running the installer again cleared the messages, but there must be an easier way?

[anand]

mystery solved: unclean shutdown causes redo.log to be owned by root:root

http://bugzilla.zimbra.com/show_bug.cgi?id=8158

[neilmc]

Thanks anand

I did an OS update last night. I noticed in the redo.log thread that KevinH (I think) said that Debian doesn't have a patch available yet.

I'm not sure if the deferred queue problem is related to the permissions problem.

I've fixed the permissions on redo.log and restarted & there are new messages appearing in deferred both from outside senders and between our own users.

Mail is also being successfully delivered while this is going on.

Is there a list of files that commonly end up with wrong permissions due to the kernal bug? (I think something else might be going on though)

[neilmc]

/opt/zimbra/log/zimbra.log shows a lot of service exception errors relating to indexes

Something seems to be worse with the kernel problem on Debian 3.1 & Zimbra 4.0 RC1.

I just did a zmcontrol stop again and redo.log permissions changed. zmcontrol never used to trigger this for us.

eg of index errors

2006-08-22 10:35:17,285 INFO [LmtpServer-11] [] LmtpHandler - [192.168.1.195] 451 4.0.0 Temporary message delivery failure try again (DATA)
2006-08-22 10:36:04,496 INFO [IndexWritersSweeper] [] MailboxIndex - open index writers sweep: before=1, closed=1, after=0 (0ms)
2006-08-22 10:36:23,549 INFO [LmtpServer-11] [name=username@mdgp.net.au;] cache - Initializing folder and tag caches for mailbox 20
2006-08-22 10:36:23,562 INFO [LmtpServer-11] [name=username@mdgp.net.au;] FileBlobStore - Stored size=27508 wrote=27508 path=/opt/zimbra/store/incoming/1156205738494-26.msg vol=1 digest=bbCdOyKQ56B4DUhc8WoujNCStdY=
2006-08-22 10:36:23,563 INFO [LmtpServer-11] [name=username@mdgp.net.au;] FileBlobStore - Renamed id=980 mbox=20 oldpath=/opt/zimbra/store/incoming/1156205738494-26.msg newpath=/opt/zimbra/store/0/20/msg/0/980-2701.msg
2006-08-22 10:36:23,569 INFO [LmtpServer-11] [name=username@mdgp.net.au;] FileBlobStore - deleting blob 980 in mailbox 20
2006-08-22 10:36:23,570 INFO [LmtpServer-11] [name=username@mdgp.net.au;] ZimbraLmtpBackend - try again for message username@mdgp.net.au: exception occurred
com.zimbra.cs.service.ServiceException: system failure: indexMessage caught IOException
at com.zimbra.cs.service.ServiceException.FAILURE(Ser viceException.java:174)
at com.zimbra.cs.index.Indexer.indexMessage(Indexer.j ava:161)
at com.zimbra.cs.mailbox.Message.reindex(Message.java :439)
at com.zimbra.cs.mailbox.Mailbox.endTransaction(Mailb ox.java:4303)
at com.zimbra.cs.mailbox.Mailbox.addMessageInternal(M ailbox.java:3262)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:3042)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:3026)
at com.zimbra.cs.filter.ZimbraMailAdapter.addMessage( ZimbraMailAdapter.java:325)
at com.zimbra.cs.filter.ZimbraMailAdapter.doDefaultFi ling(ZimbraMailAdapter.java:319)
at com.zimbra.cs.filter.RuleManager.applyRules(RuleMa nager.java:182)
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver MessageToLocalMailboxes(ZimbraLmtpBackend.java:308 )
at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver (ZimbraLmtpBackend.java:122)
at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHa ndler.java:420)
at com.zimbra.cs.lmtpserver.LmtpHandler.processComman d(LmtpHandler.java:197)
at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:231)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:198)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Could not create index /opt/zimbra/index/0/20/index/0 (directory already exists)
at com.zimbra.cs.index.MailboxIndex.openIndexWriter(M ailboxIndex.java:906)
at com.zimbra.cs.index.MailboxIndex.addDocument(Mailb oxIndex.java:349)
at com.zimbra.cs.index.Indexer.addDocument(Indexer.ja va:354)
at com.zimbra.cs.index.Indexer.addDocument(Indexer.ja va:324)
at com.zimbra.cs.index.Indexer.indexMessage(Indexer.j ava:156)
... 16 more
Caused by: java.io.IOException: Lock obtain timed out: Lock@/opt/zimbra/apache-tomcat-5.5.1...89b-write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:58)
at org.apache.lucene.index.IndexWriter.<init>(IndexWr iter.java:223)
at org.apache.lucene.index.IndexWriter.<init>(IndexWr iter.java:173)
at com.zimbra.cs.index.MailboxIndex.openIndexWriter(M ailboxIndex.java:887)
... 20 more

[neilmc]

/opt/zimbra/index/0/4/index/0/segments
/opt/zimbra/index/0/4/index/0/deletable
/opt/zimbra/index/0/4/index/0/_bsz.cfs

all had ownership set to root (also happened in other areas in the index directory structure).

chown -R zimbra:zimbra /opt/zimbra/index

caused the queue to clear.

This will likely happen everytime our backup script shuts zimbra down.

I'll probably shedule a chown for redo.log and the indexes before the backup script tries to start Zimbra again.

Can anyone think of a better solution for now?

Has anyone tried manually installing a new kernel on Debian 3.1 to get around this sort of thing? (nothing available by apt update).

[anand]

unfortunately, many data areas can have wrong permissions because of the kernel bug. Best to find yourself a kernel without this problem.

There maybe something we can do on our end, but I wouldn't say it is a priority for us because fixing the kernel problem is the right thing. I'm *guessing* that we create the thread on which crash recovery will run way too early. If we delayed the creation of that thread, then it won't be bit by this setuid() kernel bug. Our code is correct mind you - we don't do anything meaningful at all before setuid() has been called - but the JVM runs some static initializers which probably end up creating the thread early (though no code runs on it until after setuid()).

[neilmc]

Originally Posted by anand

unfortunately, many data areas can have wrong permissions because of the kernel bug. Best to find yourself a kernel without this problem.

.

My next step is probably to take an image backup of the machine and do a manual upgrade of the kernel & test it.

I'd like to stick with official debian updates, but they might be too slow with this one.

I'll also consider changing to the server release of Ubuntu Dapper (6.06). It's had a couple of kernel updates in the official updates and is now up to 2.6.15-25 where Debian is using 2.6.8

Thanks for the feedback.

[mintra]

Hi

I am getting deffered mail q issues on network edition 3.14 the data was moved from another server with open souece edition and uograde was run, I had a problem caused by me with mysql permissions Zimbra support helped me fix this.

But Deffered mail is really causing end user aggro.

I have looked through the forums and have done.

chown -R zimbra:zimbra /opt/zimbra/index

This solved it for someone else but not for me.

I have a DNS set up on the Zimbra server, but I do not knopw how to configure reverse DNS. (I use webmin and I just assumed that when you create a domain that it will set up reverse DNS as tick box to do so is ticked)

Then their is another thread which indicated that it is bug 5027

LMTP issue after upgrade to M3

Two things would help a permeant fix or a temp work around.

[phoenix]

Have you got other permissions set incorrectly in the /opt/zimbra directory? You could try running zmfixperms and then rebooting.

As this is the Network Edition a call to tech support may be the best option for your situation.