zmlmtpinject sometimes hangs on errors

[jigi]

Hello,

I am trying to automate the injection of emails into my intranet from a remote zimbra server to a local one. If the email are valid it works flawlessly.

If I have a configuration error and say, the email does not exist on the server anymore, 1 time over 3-5 tries zmlmtpinject will hang without returning. Even if I kill the zmlmtpinject pid, the process never comes back because it seems I cannot kill the Java sub-process thread that zmlmtpinject spans.

Normaly my script goes like that:

[zimbra@agora]# /usr/local/bin/getmail_inject_zimbra.py -d /var/spool/getmail/Maildir/new -r test@mydomain.net -s test@mydomain.net
file to process: 1137616914.M524569P9377Q1R8c449952ba23ef8b.agora
Exit-Status: 1
Data: [] INFO: connections=1 host=localhost port=7025
[] WARN: Delivery failed for 1137616914.M524569P9377Q1R8c449952ba23ef8b.agora:
com.zimbra.cs.lmtpserver.LmtpProtocolException: 503 5.5.1 No recipients
at com.zimbra.cs.lmtpserver.utils.LmtpClient.sendMess age(LmtpClient.java:168)
at com.zimbra.cs.lmtpserver.utils.LmtpInject$NamedLmt pClient.sendMessage(LmtpInject.java:591)
at com.zimbra.cs.lmtpserver.utils.LmtpInject$LmtpInje ctTask.run(LmtpInject.java:497)
at com.zimbra.cs.util.ThreadPool$CountedTask.run(Thre adPool.java:302)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

LmtpInject Finished
submitted=0 failed=1
maximum concurrent active connections: 1
0.017s, 0.0ms/msg, 0.0mps
average message size = 0.0KB
[zimbra@agora]#

When the process hangs it does something like that:

[zimbra@agora .getmail]# /usr/local/bin/getmail_inject_zimbra.py -d /var/spool/getmail/Maildir/new -r test@mydomain.net -s test@mydomain.net
file to process: 1137616914.M524569P9377Q1R8c449952ba23ef8b.agora
PID: 1563
Timeout Encountered!
Exit-Status: None
Data: [] INFO: connections=1 host=localhost port=7025
[] WARN: Delivery failed for 1137616914.M524569P9377Q1R8c449952ba23ef8b.agora:
com.zimbra.cs.lmtpserver.LmtpProtocolException: 503 5.5.1 No recipients
at com.zimbra.cs.lmtpserver.utils.LmtpClient.sendMess age(LmtpClient.java:168)
at com.zimbra.cs.lmtpserver.utils.LmtpInject$NamedLmt pClient.sendMessage(LmtpInject.java:591)
at com.zimbra.cs.lmtpserver.utils.LmtpInject$LmtpInje ctTask.run(LmtpInject.java:497)
at com.zimbra.cs.util.ThreadPool$CountedTask.run(Thre adPool.java:302)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

..and it nevers come back to the prompt. It just hangs there.

In these cases, my script cannot complete its cleanup and therefor it becomes impossible to automate the task in question.

I hope there would be a way to be sure that zmlmtpinject in itself terminates properly in all cases, but I'm surely scared that it's more like a Sun's JDK thread hanging problem.

One way to make-it work would be to insure that the account exists beforehand, obviously. But I wanted to ask the experts before having to code-around this since I'm worried that zmlmtpinject may hang for other reasons and then I would be cooked!

Thanks much for replying.

Keep-up the good work.

-jeff

[KevinH]

Try a "kill -QUIT <pid>" to get a stack dump. This should at least point to where the code or Java is hanging.

[jigi]

Full thread dump Java HotSpot(TM) Client VM (1.5.0_05-b05 mixed mode, sharing):

"LmtpInject-1" prio=1 tid=0x08771560 nid=0x15a2 in Object.wait() [0xb185e000..0xb185f0c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x88cc8b20> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
at java.lang.Object.wait(Object.java:474)
at EDU.oswego.cs.dl.util.concurrent.SynchronousChanne l.take(Unknown Source)
- locked <0x88cc8b20> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.ge tTask(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

"Timer-0" daemon prio=1 tid=0x085358e0 nid=0x15a0 in Object.wait() [0xb18f4000..0xb18f4f40]
at java.lang.Object.wait(Native Method)
- waiting on <0x89171690> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0x89171690> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)

"Low Memory Detector" daemon prio=1 tid=0x0831edc0 nid=0x159e runnable [0x00000000..0x00000000]

"CompilerThread0" daemon prio=1 tid=0x0831d828 nid=0x159d waiting on condition [0x00000000..0xb23798a8]

"Signal Dispatcher" daemon prio=1 tid=0x0831c8e0 nid=0x159c runnable [0x00000000..0x00000000]

"Finalizer" daemon prio=1 tid=0x083159a0 nid=0x159b in Object.wait() [0xb267c000..0xb267c1c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x8914ca00> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue .java:116)
- locked <0x8914ca00> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue .java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finali zer.java:159)

"Reference Handler" daemon prio=1 tid=0x08314c58 nid=0x159a in Object.wait() [0xb26fc000..0xb26fd040]
at java.lang.Object.wait(Native Method)
- waiting on <0x8914ca80> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Refer ence.java:116)
- locked <0x8914ca80> (a java.lang.ref.Reference$Lock)

"main" prio=1 tid=0x082d79f0 nid=0x158a in Object.wait() [0xbfd7f000..0xbfd7f5b8]
at java.lang.Object.wait(Native Method)
- waiting on <0x88ce4710> (a java.lang.Object)
at java.lang.Object.wait(Object.java:474)
at com.zimbra.cs.lmtpserver.utils.LmtpInject.main(Lmt pInject.java:242)
- locked <0x88ce4710> (a java.lang.Object)

"VM Thread" prio=1 tid=0x08312118 nid=0x1599 runnable

"VM Periodic Task Thread" prio=1 tid=0x08320298 nid=0x159f waiting on condition

[jigi]

Hello again,

if you simply use from the command line the command:

/opt/zimbra/bin/zmlmtpinject -d /var/spool/getmail/Maildir/new -r test@mydomain.net -s test@mydomain.net

..with a valid domain but an invalid email box, I'm sure you will get the same results: within 5 times of issuing the command, it will eventually hang.. ;(

Tx much for looking into-it.

-jeff

[tim brennan]

yah, a quick look turns up a simple concurrency bug in the LMTPInject code.

I'll go fix it.

[tim brennan]

http://bugzilla.zimbra.com/show_bug.cgi?id=5532

Unfortunately the fix missed the M4 cutoff, so you'll have to get it from one of the nightly builds or wait a few weeks for the next official build.

[jigi]

You guys will never cease to impress me. It took me twice the time to document and understand the problem than for you to get-it resolved!

Soooo nice of you..

Thanks a lot!

-jeff