Gargantuan deletion - How long?

[EdMartin]

Sunday afternoon, one of our database programmers was debugging some automessaging code on our websites and inserted a piece of code that, unbeknownst to him, resulted in an infinite loop continually sending messages to an info mailbox. By the time I noticed it this morning, almost half a million messages had been sent. This kinda clogged the system!

We stemmed the flow and I was left with an incoming queue containing 474,323 messages, 439,985 of them resulting from the infinite loop. In the Admin GUI, I scanned the entire incoming queue (it took 4 hours). I selected the appropriate sender address, right-clicked and selected Delete. I was asked if I really wanted to delete 439,985 messages, to which I responded "You betcha!" And now the Admin GUI and I are just sitting there looking at each other.

Is the deletion likely to complete? If so, how long is it likely to take? How will I know when it has completed? Should I click the Refresh button on the Incoming queue tab periodically, or will that result in another 4-hour scan job?

Should I have gone about this another way? (Not that I'm planning a repeat performance.)

On the flip side, I must say that it's very impressive that Zimbra appears to have withstood the onslaught, albeit suffering chronic indigestion of the incoming queue. All the services are running, but the hamsters working the treadmills must be getting pretty tired by now ... :-)

Ed

[EdMartin]

Almost an hour later, the deletion appears to be ongoing -- leastways, zimbra.log reports several messages "removed" by postfix/qmgr every second. But "several" per second isn't going to cut it when dealing with almost half a million items.

I'm a long way from being a command line aficionado, but is there a way to use the postfix command line to

* stop the wholesale deletion I ordered,
* temporarily stop the addition of any new messages into the incoming queue,
* force the delivery (presumably by jumping the queue) of all the messages I DON'T want deleted (there are far fewer of them, obviously),
* delete the entire contents of the incoming queue once those deliveries have been made,
* and finally start once again to accept new messages into the incoming queue.

If so, please give me a blow by blow, noting in particular what user I need to be for each action suggested.

Thanks in advance.

Ed

[Dirk]

On the positive side, you could call it a realtime stresstest of the equipment I'm glad that the server is still operational but I cant help with a better way to delete the messages I'm afraid. I'm just posting as I notice some time has passed. I'm sure someone here will know of a way to get this done in a more efficient manner though. Good luck with it.

[uxbod]

You could stop the MTA using

su - zimbra
zmmtactl stop

and then check in /opt/zimbra/postfix/spool/incoming/ for any mails that have a certain subject line which you could then delete before they are processed.

[EdMartin]

Thanks for the moral support, Dirk, and thanks, Uxbod, for a possible strategy. It lines up with what I expected would be a good approach. However, this is where my lack of command line prowess is revealed. I don't know how to check the incoming queue for messages with a particular subject line and then delete them. (By the way, I do know the exact subject line of all the messages I want to delete.)

I found the following on the postfix postsuper man page:

mailq | tail +2 | grep -v '^ *(' | awk 'BEGIN { RS = "" }
# $7=sender, $8=recipient1, $9=recipient2
{ if ($8 == "user@example.com" && $9 == "")
print $1 }
' | tr -d '*!' | postsuper -d -

Apparently, this will delete all mail with exactly one recipient (to wit, user@example.com). But this doesn't tell me what the placeholder is for the subject line. Furthermore, when I tried the following edited version of it

mailq | tail +2 | grep '^ *(' | awk 'BEGIN { RS = "" }
# $7=sender, $8=recipient1, $9=recipient2
{ if ($7 == "sender@domain.com" && $8 == "receiver@domain.com")
print $1 }
' | tr -d '*!' | postsuper -d incoming

I was informed that the "tail +2" syntax is deprecated, the command completed far too quickly to have had the desired effect, and the incoming queue seems not to have diminished appreciably.

In addition, this surely can't be an efficient way to achieve the desired goal, with all those pipes in there.

Does anyone have an further command line wisdom for me?

Thanks in advance.

Ed

[EdMartin]

Crisis over!

First of all, it took me an appreciable amount of time to discover that ZCS v5.0.6 has the incoming queue at

/opt/zimbra/data/postfix/spool/incoming

rather than the URL Uxbod provided. Armed with that knowledge, we then did the following:

1. As zimbra, stop MTA using "zmmtactl stop".

2. As root, cd to the above incoming queue directory.

3. Execute the following command:

find . -exec grep -q "KNOWN SUBJECT LINE" '{}' \; -print0 | tr -d './' | xargs -0 rm

The find command returns a list of file names starting with ./
The tr command strips off those two characters

It took close to 45 minutes, but this deleted all the unwanted messages from the incoming queue, leaving only 922 wanted ones.

I then restarted the server. All the Zimbra services came back up with the exception of MTA, but that started without incident when I executed "zmmtactl start" as zimbra.

There then followed about a 20 - 30 minute period while the contents of the active queue was gradually dealt with, while being topped up as circumstances allowed by the contents of the (now immensely diminished) incoming queue.

Now all the queues are down to zero and everything seems to be functioning normally. Phew! I don't want to have to go through that again any time soon!

Ed