Results 1 to 9 of 9

Thread: 4.0 RC1 Deferred queue problem

  1. #1
    neilmc is offline Active Member
    Join Date
    Nov 2005
    Posts
    44
    Rep Power
    9

    Default 4.0 RC1 Deferred queue problem

    I'm not sure if this is related to 4.0 RC1 or if it would have happened anyway.

    Before the upgrade I only ran into the redo.log permissions problem (Debian 3.1) if the server was shut down without first ..... zmcontrol stop

    Now it seems to happen sometimes just by running zmcontrol stop

    After such an occurance, I noticed 6 emails in the Deferred queue & requeuing them doesn't clear them. Other email is being delivered successfully while this is happening.

    The error for all of them is

    Reason: host (hostname ip address) said 451 4.0.0 temporary message delivery failure, try again (in reply to end of data command)

    If I look in /var/log/zimbra.log just after requeuing a message I see "Requeued: 1 message"

    but nothing else on that message in the log before it goes back to deferred.

    This happened earlier & running the installer again cleared the messages, but there must be an easier way?
    Last edited by neilmc; 08-21-2006 at 05:10 PM.

  2. #2
    anand is offline Zimbra Employee
    Join Date
    Sep 2005
    Posts
    274
    Rep Power
    9

  3. #3
    neilmc is offline Active Member
    Join Date
    Nov 2005
    Posts
    44
    Rep Power
    9

    Default

    Thanks anand

    I did an OS update last night. I noticed in the redo.log thread that KevinH (I think) said that Debian doesn't have a patch available yet.

    I'm not sure if the deferred queue problem is related to the permissions problem.

    I've fixed the permissions on redo.log and restarted & there are new messages appearing in deferred both from outside senders and between our own users.

    Mail is also being successfully delivered while this is going on.

    Is there a list of files that commonly end up with wrong permissions due to the kernal bug? (I think something else might be going on though)

  4. #4
    neilmc is offline Active Member
    Join Date
    Nov 2005
    Posts
    44
    Rep Power
    9

    Default Does look like permissions related.... indexes

    /opt/zimbra/log/zimbra.log shows a lot of service exception errors relating to indexes

    Something seems to be worse with the kernel problem on Debian 3.1 & Zimbra 4.0 RC1.

    I just did a zmcontrol stop again and redo.log permissions changed. zmcontrol never used to trigger this for us.

    eg of index errors

    2006-08-22 10:35:17,285 INFO [LmtpServer-11] [] LmtpHandler - [192.168.1.195] 451 4.0.0 Temporary message delivery failure try again (DATA)
    2006-08-22 10:36:04,496 INFO [IndexWritersSweeper] [] MailboxIndex - open index writers sweep: before=1, closed=1, after=0 (0ms)
    2006-08-22 10:36:23,549 INFO [LmtpServer-11] [name=username@mdgp.net.au;] cache - Initializing folder and tag caches for mailbox 20
    2006-08-22 10:36:23,562 INFO [LmtpServer-11] [name=username@mdgp.net.au;] FileBlobStore - Stored size=27508 wrote=27508 path=/opt/zimbra/store/incoming/1156205738494-26.msg vol=1 digest=bbCdOyKQ56B4DUhc8WoujNCStdY=
    2006-08-22 10:36:23,563 INFO [LmtpServer-11] [name=username@mdgp.net.au;] FileBlobStore - Renamed id=980 mbox=20 oldpath=/opt/zimbra/store/incoming/1156205738494-26.msg newpath=/opt/zimbra/store/0/20/msg/0/980-2701.msg
    2006-08-22 10:36:23,569 INFO [LmtpServer-11] [name=username@mdgp.net.au;] FileBlobStore - deleting blob 980 in mailbox 20
    2006-08-22 10:36:23,570 INFO [LmtpServer-11] [name=username@mdgp.net.au;] ZimbraLmtpBackend - try again for message username@mdgp.net.au: exception occurred
    com.zimbra.cs.service.ServiceException: system failure: indexMessage caught IOException
    at com.zimbra.cs.service.ServiceException.FAILURE(Ser viceException.java:174)
    at com.zimbra.cs.index.Indexer.indexMessage(Indexer.j ava:161)
    at com.zimbra.cs.mailbox.Message.reindex(Message.java :439)
    at com.zimbra.cs.mailbox.Mailbox.endTransaction(Mailb ox.java:4303)
    at com.zimbra.cs.mailbox.Mailbox.addMessageInternal(M ailbox.java:3262)
    at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:3042)
    at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:3026)
    at com.zimbra.cs.filter.ZimbraMailAdapter.addMessage( ZimbraMailAdapter.java:325)
    at com.zimbra.cs.filter.ZimbraMailAdapter.doDefaultFi ling(ZimbraMailAdapter.java:319)
    at com.zimbra.cs.filter.RuleManager.applyRules(RuleMa nager.java:182)
    at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver MessageToLocalMailboxes(ZimbraLmtpBackend.java:308 )
    at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver (ZimbraLmtpBackend.java:122)
    at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHa ndler.java:420)
    at com.zimbra.cs.lmtpserver.LmtpHandler.processComman d(LmtpHandler.java:197)
    at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:231)
    at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:198)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:595)
    Caused by: java.io.IOException: Could not create index /opt/zimbra/index/0/20/index/0 (directory already exists)
    at com.zimbra.cs.index.MailboxIndex.openIndexWriter(M ailboxIndex.java:906)
    at com.zimbra.cs.index.MailboxIndex.addDocument(Mailb oxIndex.java:349)
    at com.zimbra.cs.index.Indexer.addDocument(Indexer.ja va:354)
    at com.zimbra.cs.index.Indexer.addDocument(Indexer.ja va:324)
    at com.zimbra.cs.index.Indexer.indexMessage(Indexer.j ava:156)
    ... 16 more
    Caused by: java.io.IOException: Lock obtain timed out: Lock@/opt/zimbra/apache-tomcat-5.5.1...89b-write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:58)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWr iter.java:223)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWr iter.java:173)
    at com.zimbra.cs.index.MailboxIndex.openIndexWriter(M ailboxIndex.java:887)
    ... 20 more

  5. #5
    neilmc is offline Active Member
    Join Date
    Nov 2005
    Posts
    44
    Rep Power
    9

    Default index files owned by root

    /opt/zimbra/index/0/4/index/0/segments
    /opt/zimbra/index/0/4/index/0/deletable
    /opt/zimbra/index/0/4/index/0/_bsz.cfs

    all had ownership set to root (also happened in other areas in the index directory structure).

    chown -R zimbra:zimbra /opt/zimbra/index

    caused the queue to clear.

    This will likely happen everytime our backup script shuts zimbra down.

    I'll probably shedule a chown for redo.log and the indexes before the backup script tries to start Zimbra again.

    Can anyone think of a better solution for now?

    Has anyone tried manually installing a new kernel on Debian 3.1 to get around this sort of thing? (nothing available by apt update).

  6. #6
    anand is offline Zimbra Employee
    Join Date
    Sep 2005
    Posts
    274
    Rep Power
    9

    Default

    unfortunately, many data areas can have wrong permissions because of the kernel bug. Best to find yourself a kernel without this problem.

    There maybe something we can do on our end, but I wouldn't say it is a priority for us because fixing the kernel problem is the right thing. I'm *guessing* that we create the thread on which crash recovery will run way too early. If we delayed the creation of that thread, then it won't be bit by this setuid() kernel bug. Our code is correct mind you - we don't do anything meaningful at all before setuid() has been called - but the JVM runs some static initializers which probably end up creating the thread early (though no code runs on it until after setuid()).
    Bugzilla - Wiki - Downloads - Before posting... Search!

  7. #7
    neilmc is offline Active Member
    Join Date
    Nov 2005
    Posts
    44
    Rep Power
    9

    Default

    Quote Originally Posted by anand
    unfortunately, many data areas can have wrong permissions because of the kernel bug. Best to find yourself a kernel without this problem.

    .
    My next step is probably to take an image backup of the machine and do a manual upgrade of the kernel & test it.

    I'd like to stick with official debian updates, but they might be too slow with this one.

    I'll also consider changing to the server release of Ubuntu Dapper (6.06). It's had a couple of kernel updates in the official updates and is now up to 2.6.15-25 where Debian is using 2.6.8

    Thanks for the feedback.
    Last edited by neilmc; 08-21-2006 at 11:51 PM.

  8. #8
    mintra is offline Special Member
    Join Date
    Nov 2005
    Location
    UK
    Posts
    117
    Rep Power
    9

    Default I am having defferd queue

    Hi

    I am getting deffered mail q issues on network edition 3.14 the data was moved from another server with open souece edition and uograde was run, I had a problem caused by me with mysql permissions Zimbra support helped me fix this.

    But Deffered mail is really causing end user aggro.

    I have looked through the forums and have done.

    chown -R zimbra:zimbra /opt/zimbra/index

    This solved it for someone else but not for me.

    I have a DNS set up on the Zimbra server, but I do not knopw how to configure reverse DNS. (I use webmin and I just assumed that when you create a domain that it will set up reverse DNS as tick box to do so is ticked)

    Then their is another thread which indicated that it is bug 5027

    LMTP issue after upgrade to M3

    Two things would help a permeant fix or a temp work around.

  9. #9
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,480
    Rep Power
    56

    Default

    Have you got other permissions set incorrectly in the /opt/zimbra directory? You could try running zmfixperms and then rebooting.

    As this is the Network Edition a call to tech support may be the best option for your situation.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 3
    Last Post: 07-19-2007, 02:00 AM
  2. resolving deferred email problem
    By DVan in forum Administrators
    Replies: 1
    Last Post: 03-26-2007, 07:45 PM
  3. queue problem w/ deferred messages
    By alderion in forum Administrators
    Replies: 10
    Last Post: 10-26-2006, 07:51 AM
  4. ZCS 4.0 RC1 to launch this week
    By KevinH in forum Announcements
    Replies: 1
    Last Post: 08-14-2006, 09:37 PM
  5. Outbound queue problem & fetchmail question
    By phoenix in forum Administrators
    Replies: 7
    Last Post: 01-25-2006, 11:59 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •