I to am having the same issues on 3.2. It all starts at about 2300 at night when the spam training runs. Zimbra had me disable some lines in the zmtrainsa command to see if it helps. I come in every morning and there are over 2000 messages in the deferred queue. I flush the queues and the messages do eventually get delivered but it creates a delay of about 2 hours in mail sending.