Adding this to main.cf in postfix/conf seemed to speed everything up
Code:
lmtp_destination_concurrency_limit = 40
default_destination_concurrency_limit = 40
I still get that no socks ready when using the spamassassin script, but the active queue stays pretty low now. Huge difference than before