Using the whitelist fixed my issue. Originally, when I setup the whitelist I added the whitelist info into AMASVISD.CONF but following the instructions on the
wiki the info goes into the AMASVISD.CONF
.IN Once I put the info into the correct file and then restarted both Postfix AND amasvisd it worked perfectly.