OK found the answer to my issue here:
http://www.zimbra.com/forums/install...happening.html
Basically it takes 3 minutes for clamd to read in the db file. Once it does it will bind to the port and start normally. I changed the interval in this file:
/opt/zimbra/libexec/zmmtaconfig
from 120 to 180. Now it starts just fine.