I discovered that, at least in my installation, the problem simply was that the web.xml file was not being updated from its template, /opt/zimbra/jetty/etc/zimbra.web.xml.in. (note, I was also getting the zmmailbox error) If for some reason, the changes below do not work for you, you may consult the generated zimbra.web.xml.in file and see what settings your desired mode might require.
Note that this is a workaround only, it does not fix the zmtlsctl command, only "finishes the job".
To fix, simply:
Edit /opt/zimbra/jetty/webapps/zimbra/WEB-INF/web.xml
Locate "protocolMode", and edit
<env-entry-value>http</env-entry-value><!-- PROTOCOL MODE -->
to read as the mode you would like to set, example:
<env-entry-value>both</env-entry-value><!-- PROTOCOL MODE -->
Furthermore, if redirects will be performed (for HTTPS mode, and perhaps others..), go to the bottom, change:
<!-- REDIRECTBEGIN
<security-constraint>
<web-resource-collection>
<web-resource-name>force https</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
REDIRECTEND -->
to...
<!-- REDIRECTBEGIN -->
<security-constraint>
<web-resource-collection>
<web-resource-name>force https</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
<!-- REDIRECTEND -->
Finally,
Run:
su - zimbra
zmmailboxdctl stop
zmmailboxdctl start