I haven't used CardDAV due to issues with virtual domains, but CardDAV over SSL "just works" for me. For example, in the account setup for iCal under Mac OS, I just have it set to use Port 443 and the "Use SSL" checkbox is checked.
If you use the iSync connector on a Mac to have it automatically configure CardDav, you do the same in the Account tab: Server port 443 and check "Use Secure Sockets Layer (SSL)".
The only possible prerequisites I can think of are (a) having a commercial certificate instead of self-signed, and (b) having Zimbra configured to use https. For the latter see
CLI zmtlsctl to set Web Server Mode - Zimbra :: Wiki and use any mode (as described in the command) other than
http.
Not sure if (a) and (b) are necessary, but I configured them very early in our installation, so I can't say what would happen if you don't have them.