When this happens you will also see that ClamAV is working very hard aswell.
Have a read of
Improving Anti-spam system - Zimbra :: Wiki as you could easily reduce the amount of work Amavis is performing by implementing a few of the Postfix tweaks. This can be enhanced further by implementing RBL checking at the Postfix level aswell.
Hope this helps.