I had a few problems with this too but maybe not the same ones.
However, just to check, are you using External LDAP or Active Directory Authentication?
I am and when I tried to connect Sugar using my Windows Logon password it would not connect or would give a 'bad user or password message'. I had to use the original password I had entered for my user in Zimbra.
This may not be of any use but thought I'd mention it.
Oh, also, because I have 'Virtual Domains' set up on Zimbra I have to put my IMAP username in Sugar as just 'user' as opposed to before the Virtual Domain where I had to put 'user@domain' for Sugar to connect.
Hope this helps but it probably won't!
Russ
Fingers crossed by a Zimbra/Sugar connector sometime in the future!!