Hmmm, was able to use a hostname just fine here. Try setting it via the command line
Code:
su - zimbra
zmprov gs server.domain.com | grep zimbraMtaRelayHost
just to check it first, and then to change it
Code:
zmprov ms server.domain.com zimbraMtaRelayHost external.isp.net:25