Well you cannot have it both ways
If you add a catch all account then there is the potential for spams to get through. By adding the catch all you remove the functionality of validating the to: account prior to delivery. Sorry; no real way around that. If you SpamAssassin is tuned then that should catch most most of the spams.