I am not sure how to set this up since I have only one group (All accounts). The shared calendars I have belong to this group as well and I don't see any configuration option in case I have only one account.
Most likely I'm hitting this bug:
Bug 30991 – ZCO takes a long time to load an ics with lots of attendees
We had imported Oracle calendar into Zimbra and are now experiencing serious challenges. The ZCO clients are particularly hit hard...