Z-Push Zimbra Backend (Push Email, Contacts, Appointments & Tasks)

[YakkoWarner]

Try setting PROVISION to true, AND setting LOOSE_PROVISIONING to true also.

LOOSE_PROVISIONING brought me back to the first error.

EDIT: It might also be an issue with how you configured your server Alias command - or with the installation of a client certificate on the WM7 phone.

Already have the cert installed and the Alias command set up as specified. (The errors they describe are different codes than what I am getting.)

What else would be helpful in debugging this?

[liverpoolfcfan]

LOOSE_PROVISIONING brought me back to the first error.

Already have the cert installed and the Alias command set up as specified. (The errors they describe are different codes than what I am getting.)

What else would be helpful in debugging this?

Have you gotten any further with this testing ?

What server configuration do you have ?

[liverpoolfcfan]

Release 51 of the z-push zimbra backend has just been uploaded to SourceForge (https://sourceforge.net/projects/zimbrabackend/)

NOTE: For those interested in HTML email - There is an as12.1 branch of the backend checked in to the SVN at the site under /zimbra-backend/branches/as12.1 - It supports HTML email on Apple/WM/Android and some Nokia phones. Please read the CRITICAL README.TXT in the directory as the data store for device states changes completely with this version. It requires the as12.1 SVN branch of z-push to work. I am not making 'release packages' for this version as it is not an official z-push release. Official support for HTML and other ActiveSync protocol 14 features will come in z-push 2 - and once that is reasonably stable we will work to support it fully from the backend.


Here is a summary of the changes.

THESE RELEASE NOTES ARE FOR RELEASE 51 CHANGES 

PLEASE READ THESE RELEASE NOTES COMPLETELY BEFORE IMPLEMENTING RELEASE 51
NOTE THAT THE RELEASE NOTES FOR 49 & 50 ARE ALSO CONTAINED BELOW AND SHOULD 
ALSO BE READ FULLY WHEN UPGRADING FROM A RELEASE 48 OR EARLIER.
THIS IS EXPECIALLY CRITICAL IF YOU ARE TURNING ON ANY OF THE NEW FEATURES
AND ARE UPGRADING AN EXISTING INSTALLATION FROM RELEASE 48 OR EARLIER. 


Changes Made To Revision 51:
 - Added attendee status/type information to appointments 
 - Changed GlobalObjID to be correct non-outlook encoded value so that Apple 
   and possibly other devices will update appointments correctly 
 - Changed handling of invitations for client generated meetings - zimbra will 
   always send out invitations now (even for Apple/Android) - but if zimbra is
   the default email client on the device, we will drop the device originated 
   invitation once it is received by SendMail. If another email account is your 
   default account then the device will send a "duplicate" meeting invitation 
   directly through that email account. Replies to the device originated  
   invitation may not correctly update the attendee status of the appointment 
   on the original device 
 - Fixed as12+ body handling for Appointment/Contact/Task
 - Changed handling of birthday/anniversary field to try to get a solution 
   that will work for most scenarios 
 - android 2.3.3 seems to have a serious problem with the birthday field that 
   causes it to spin constantly trying to resync the calendar - limiting 
   birthday to known UserAgents for now (Apple and MailforExchange)
 - Experimental fix in Logon to delay and retry logon attempts when the
   zimbra server is not available (eg. zimbra FOSS shut down for backup)
   See ZIMBRA_RETRIES_ON_HOST_CONNECT_ERROR in the notes below.
 - Fix to add missing $this-> prefix to ScrubHtmlText in function 
   GetMpBodyRecursive
 - Added fix to AlterPingChanges to detect changes in Virtual Folders
 - Added output the Conversation ID field to allow view by conversation
   on some Android devices (WM appears to need more to work)
 - Cosmetic fix in GetSearchResults to avoid logging error when GAL search
   returns zero matches
 - Added 'wiki' folder type for SmartFolder setup to accomodate servers that
   are running, or once ran, version 5.0.x of zimbra
 - Removed Nokia only filter on adding FakeOutbox folder, as Android 2.3.3  
   seems to need it too, and it doesn't appear to break other phone syncs
   and renamed the devid to FakeOutbox from NokiaFakeOut
 - Added path to fake outbox structure to fix debug log error
 - Save SOAP error extracted from response to $this->_soapError for 
   decision making in calling routine
 - Removed 'Too big for inline?' debug message
 - Reworked Sendmail address checking
 - Revised non-plain text email handling in GetMessage
 - Reverted text/html encoding to base64 from quoted-printable for mime body
   as the majority of incoming messages use base64
 - Added check for 'profile' in XML file to fix debug log error
 - Added quotes around rtf for class_exists checks to fix debug log error
 - Added check for empty $response content in ExtractSessionID to fix debug
   log error
 - Added Date range handling to Out-Of-Office logic in getSettings and
   setSettings, and fixed $oofstate logic.
 - Changed function arguments to work with as12.1 branch - Fetch, GetMessage,
   SendMail
 - Changed handling of flags to work with as12.1 branch
 - Removed class class ExportChangesZimbra as not needed for as12.1
 - Removed class class ImportHierarchyChangesZimbra as not needed for as12.1

IF YOU ARE UPGRADING FROM RELEASE 48 OR EARLIER - IT IS CRITICAL THAT YOU READ THE
RELEASE NOTES FOR RELEASES 49 & 50. SOME MAJOR CHANGES WERE IMPLEMENTED IN THOSE
RELEASES (ROLLED UP INTO RELEASE 51 ALSO) THAT WILL AFFECT YOUR SETUP

[bofh]

Hey just a Question - smartfolder
are they supposed to work with calendars too ?

[liverpoolfcfan]

Hey just a Question - smartfolder
are they supposed to work with calendars too ?

Yes. Should do.

[rfr]

Just a little note to tell you that I had to delete/recreate my account to sync my contacts (I had a mismatch tag error in z-push). I also cleaned the state directory and now everything seems to work (Samsung GS2).

[bofh]

1. about Smartfolder - thats strange - ife activated smartfolder from the first day but it doesnt have any affect on the caledarsync
i shared one folder form another account and renamed it to something like "name -" still syncs
any way to debug this ?


EDIT: Sorry my Bad or better ZDesktops Bad - for Some Reason renaming Folder from the Desktop do not Sync properly to the Server
so the Change was never aplied

2. as i wrote earlier - no need - absolute no need - to clean statedirectory - this is important special for running medium to large installs
its like a realexchange active sync (theres also kinda statedirectory) you cant really clean

android always gets a new idea everytime you add an account so you get new statefiles

however clean old unused files might not be wrong - something like older than 8 months or so should work

regarding android sync we discovered a LOOOT of glitches there today
we where able to test with
2.2.1 / 2.2.2 /2.3.4 / 2.3.3 / 3.0 / 3.1
devices where phones and tablets (most samsung devices and one acer)

turns out that every dam firmware behaves differently
even we got now some very strange errors on tablet (while phones wokred very fine)
the error was only for email sync
something like "security error - and he cant connect to the server" so no new mails where synced while contacts and calendar running fine

even worse on the most actual android version you can jsut sync calendar and or contacts without mail - mail is always in
which is a problem on tablets -

theres absolute no sufficent mail application for tablets except built in but that one always syncs then - so the option no mailsync BUT imap isnt one

alternative is forget active sync - use caldav and cardav and imap - which is a way better idea for many reasons - most important you can determine which shared calendar you wanna use

downside - no real caldav client yet for bidirectional sync
so situation is kinda messy

just to be clear this ISNT THE FAULT of z-push or zimbra backend or Zimbra

you will have those issues with
zimbra netwokr edition
zimbra foss with zpush or funambol
exchange no matter which version

this is a problem of the active sync implementation of android reported hundred of times on different systems all over the web

and it looks its different on phones and tablets even on the same android version different cause of manufacturers modifed roms

only solution i see is get a caldav working or ry custom roms
but here agian problem with different devices (you not gonna get one rom version for every device) so you cant be shure you get a straight behavaiour


current solution tot he security error seems to clear cache and if that not help - clear cache and readd exchange which is a mess

Conclusion: active sync is messy by default - activesync and android work on phones well but not on tablets
activesync and iphone seems to work but i only got 5 devices running on that yet
and final - forget cleant statedirectory - at least android and iphones always make a new id so its not nessesary

[liverpoolfcfan]

1. about Smartfolder -
i shared one folder form another account and renamed it to something like "name -" still syncs
any way to debug this ?

Have you made a change to the calendar that will trigger a re-sync - or manually synced it ?
I have just done the same as you described here - added a "-" to the name of a shared folder. Then I created a new dummy appointment on my normal calendar to cause a resync. As soon as it synced, it has cleared all the shared Calendar appointments. The reverse also works. Removing the "-" and causing a sync brought them all back.

2. as i wrote earlier - no need - absolute no need - to clean statedirectory - this is important special for running medium to large installs
its like a realexchange active sync (theres also kinda statedirectory) you cant really clean

This is "almost true" at this point in time. The actual folders for email, calendar, etc. always get created as new GUID's with each fresh sync - this part is true - but the overall "syncstate" that stores the current state of the folders on the phone is always named for the device itself. If the new zimbra backend release does not change the structure of that state file - then you will be fine without clearing out the state directory. If, however, the structure of the file is changed by the release, then you will get errors trying to sync. The last 4 or 5 releases have not altered the structure - so for anyone keeping up to date with releases - clearing out the files is probably not necessary. For someone coming from a very old release to a recent one, they will need to clear the old files.

For simplicity - if you have a problem device - look for it's device ID in the debug.txt file. Once you have found it - remove the activesync account from the device - then delete the file named the same as the device ID from the state directory. This will leave all the old folder state files behind for that device - but when you add the device back it will get a clean "syncstate".

android always gets a new idea everytime you add an account so you get new statefiles

This is not true of all phones and not even of all android phones. We have Samsung Galaxy S phones here, and some (not even all of them) got a new device ID when we applied a firmware update. But - their activesync ID always matched the phone's ID - and during testing I have removed and re-added accounts hundreds of times.

Hopefully z-push2 will adopt the same naming scheme as the as12.1 branch - which creates a folder per device - as this makes cleanup really simple.

regarding android sync we discovered a LOOOT of glitches there today
we where able to test with
2.2.1 / 2.2.2 /2.3.4 / 2.3.3 / 3.0 / 3.1
devices where phones and tablets (most samsung devices and one acer)

turns out that every dam firmware behaves differently

Couldn't agree more. It is a mess.

even we got now some very strange errors on tablet (while phones wokred very fine)
the error was only for email sync
something like "security error - and he cant connect to the server" so no new mails where synced while contacts and calendar running fine

Do you have error reporting turned on in PHP ? It is possible that the device is sending in it's request with fields in an unexpected order - or with unexpected content. With wbxml_debug turned on, look to see if you have a session from the tablet that begins to process an incoming request from the tablet (lines start with "I" in column 1) and then just stops. It might be crashing. With PHP error reporting turned on it gives a line number (usually in request.php) where it crashed. You might also get a line number quoted in an error message in your apache log file.

even worse on the most actual android version you can jsut sync calendar and or contacts without mail - mail is always in
which is a problem on tablets

You could try just setting a SmartFolders directive - i.e. putting a folder named "message&active=false" in your "*SyncConfig*" folder - to see if that gets you over the issue. However, as mentioned it the answer above - if the problem is the sequence of fields in the request message then this won't help you.

[Stuart]

Release 51 of the z-push zimbra backend has just been uploaded to SourceForge (https://sourceforge.net/projects/zimbrabackend/)

NOTE: For those interested in HTML email - There is an as12.1 branch of the backend checked in to the SVN at the site under /zimbra-backend/branches/as12.1 - It supports HTML email on Apple/WM/Android and some Nokia phones.....

And it works better than Network Edition Great work, HTML works very well for SGS2 and some custom ROMS I've tried. And at last a contact sync that doesn't change the "file as" field

[bofh]

This is "almost true" at this point in time. The actual folders for email, calendar, etc. always get created as new GUID's with each fresh sync - this part is true - but the overall "syncstate" that stores the current state of the folders on the phone is always named for the device itself. If the new zimbra backend release does not change the structure of that state file - then you will be fine without clearing out the state directory. If, however, the structure of the file is changed by the release, then you will get errors trying to sync. The last 4 or 5 releases have not altered the structure - so for anyone keeping up to date with releases - clearing out the files is probably not necessary. For someone coming from a very old release to a recent one, they will need to clear the old files.

You could try just setting a SmartFolders directive - i.e. putting a folder named "message&active=false" in your "*SyncConfig*" folder - to see if that gets you over the issue. However, as mentioned it the answer above - if the problem is the sequence of fields in the request message then this won't help you.

Hello,

So first of all - sorry for the Smartfolder thing - it was Zdesktop fault
i already files a Bugreport

ATM you cannot rename Shared Folder FROM Zdesktop you can only Rename em in the Webinterface - otherwise new Name apears in the Desktop but never getting Synced to the Server

So Smartfolder work fine - just my Renaming didnt hehe

---


About Statedirectory - ahm yes i didnt talked about a new Revision of the Backend - i was talking about Sync errors on the Device
A New Version of the Zimbra Backend is the only real reason to clear state Directory

BTW debug.txt and manually cleaning id files are not a solution - not a practical one it works for 5 Devices but if you get more youre in trouble

So honestly, even if im still in testing Operation mode, im never gonna update the Backend to a version which need the clearing of statedirectory

i simply cant expect so many users delete their exchange - wait all have it done - make the upgrade - recall all user to add it again - no way

I personally just wait that a proper CALDAV App comes to final stage (theres something promising) - then i will switch many user as possible to carddav (works already) /caldav/imap
From my Point of view way more relyable technology than active sync

if a nessesary backend upgrade comming which need a clear statedirectory OR upgrade on current statefile (which should work too)
we may can and should workout an updatemethod for the statefiles instead of clearing em

again - remove 10 or 20 or 50 devices jsut for a upgrade = nogo

but i think it should be possible to upgrade those files - they are structured so if we need one more field in i think it can be done by linux basic tools such a sed to generate new updated files




About the Security error - ahm its a wierd thing - and one thats not related to backend - it apears to any activesync provider - and even worse its not device related its almsot random

devices that work for ages stop it from one second to another - some get the error (just for a short moment you cant even proper read it) but then do the job some other devices stop to sync at all (mail only - calendar and contacs never make an error and work even when mail recieve the error)

and now the fun part : since the lates update i made yesterday (upgrade zimbra upgrade backend) the error dissapeared again (we will see how long it last)

i think it could be the nokia /outbox thing that is on by default now - and just the error apears wrong (so no securtiy error but no outbox found)


---

actually its not a real problem for me - every user know he should use imap instead of activesync for mail - in all new androids you can turn of email sync for exchange so this isnt a problem - only a few not having hat option are bit messy now

and again i really look forward to the day where most mobile users can use caldav and cardav - and never look back to active sync at least for most users


ps. forgot to mention that i only need android and iphone support because of the Cellprovider Situation here in my Country - it apears no user with a normal phone (almost no) got a data contract so they never can and want use email and other sync
the only ones that do are androids and iphones (there other smartphones with internet contract but now they all switch to droid which make the situation way better


hey btw i know theres now that switch in case zimbry is offline but what happen if the zpush server is offline
wiill the device make an error ???

its because i waked up a few people during the zimbra upgrade - android instantly feeps every minute and ask for correct user and password
so i use now the new function but maybe its better to turn off the zpush server itself during system upgrades