If you have "TLS" only checked and SMTP Auth is on by default, you should be good to go security-wise anyway.
We have gone a step further and blocked everything except for tcp-25 ( SMTP ), DNS and VPN, on the firewalls for incoming connections from anything that is not based in US so, you could do that can gain some extra security if you want. Why block at one device when you can block for all your devices at a single point?
YMMV I guess. :-)
I tried to do what you are trying to do on our mail servers at one point in time and it got too messy after while, as there was always someone on vacation, at a hotel lobby, etc., that needed access and I would have to add their IP Address or network, to the mail server for access and then restart the service EACH TIME I made a change. Got old after a while.
Plus by doing this on the main border firewall anyway, you can track and block script kiddies for the whole internal network easier anyway.
I agree with the VPN suggestion if you really want to get double authentication and stop the "pass-by scans" by script kiddies but it does take more work in both knowing how to run it and client software support issues always arrise so, it's not a "white horse" by any means.
We block all non-us network ranges at the main firewall for everything except for SMTP, DNS and VPN stuff. If someone goes to another country, they can still VPN in using AES-256 level encryption and get everything that way. So far it's worked out OK and no more "mad have-to-have-it-now" changes to the mail server in the middle of the business day kinda stuff.
Looking at the whole Zimbra package I'm not sure why, assuming TLS only checked and SMTP auth is not circumvented, you would want to restrict it in that manner anyway really, other than the way we do it above.
Net Ranges starting point for weeding out by country can be found here:
http://www.completewhois.com/statistics/index.htm
and of course
http://www.arin.net
Anyway, all IMHO, YMMV, etc., etc. ;-)
Scotty