Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Importing large mailboxes

  1. #1
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    141
    Rep Power
    6

    Default Importing large mailboxes

    Issues arrised, when I tried to import large mailboxes - over 1 GB. Curently migrating from v6, to v7 OS. I wanted to achieve fresh ZCS server work, so migration is done with mailbox by mailbox basis, using user Preferences and import options, or REST URL options from CLI. Migration, besides v6->v7, is done from 32-bit->64-bit.

    All times, when import is more than 100Mb (tgz file), I run in timeout problems or archive formatter problems on v7 machine. It seamed to me, that this could be tied up with max file upload settings on ZCS configuration. Increased this amount, but still get these errors. Nor CLI, nor webclient based imports rarely end with success. Can anybody give a hint on these issues? Almost all our mailboxes are more than 1G or 2G in size. IMAP syncing too get problems with read timeout and UID fetch failed.

  2. #2
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    141
    Rep Power
    6

    Default

    Regarding UID FETCH failed:
    I added configuration for zimbraDataSourceConnectTimeout and zimbraDataSourceReadTimeout, by increasing values to 120. It seems to be more stable, than without these settings for exact datasource, but still it has the same issue after longer time. This was done according to these bugs:
    Bug 44532 – IMAP/POP timeout should be per data source instead of global
    Bug 48247 – Error:system failure: Synchronization of folder '/GMX IMAP Account/INBOX' failed, UID FETCH failed

    The most annoying thing is that user has to move to Preferences, to input External account password again and again, if this timeout incurres. And, if user has logged out with this error, he gets a infobox on issues regarding the same, when he logs in again. Is it ok just to increase these timeouts more?

  3. #3
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    141
    Rep Power
    6

    Default

    Even with increased timeouts the same happens. During longer sync periods, error comes out informing, that UID FETCH failed. Are there any other settings should be applied, to let sync run without any problems and rememeber user password for external account?

  4. #4
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    141
    Rep Power
    6

    Default

    Guys, I really have to solve this issue. Can anybody ellaborate on it with me, as it gets tough. As I decided to migrate accounts one by one via linkage to old server IMAP account on new ZCS v7, there is the same problem - UID Fetch failed with timeout. Even increasing Read and Connect timeouts does not help at all. It just waits for longer period, and throws out an error. The odd thing is that even logs say, that there are errors, for some time after an error, mails keep syncing, but then stop without any notice. The other - after an error dialog for external account, it gets blocked, and the only way is to go to Preferences > Accounts and re-enter password and save. Then again - arround hundred mails synced, and the same problem again.

    Where does ZCS server store UIDs? MySQL or any other place?
    Does ZCS IMAP functionality is provided by Cyrus or there is some mix of things?
    During last week I managed to transfer only 6 accounts, due to these problems. May be this is connected with corrupted mailboxes? How can I fix them?

  5. #5
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    141
    Rep Power
    6

    Default

    Just found parameter: zimbraMailContentMaxSize, which is different from zimbraFileUploadMaxSize. Can this influence potential timeouts for syncing large IMAP mailboxes? I can not find any documentation on issue, whether this parameter should be stated in Bytes or Kilobytes, nor if it has to be increased, then on what servers (multiserver installation) Mailbox or MTA? Does separate Zimbra Proxy server has to do something with this setting?

    Anybody? !!!

  6. #6
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    141
    Rep Power
    6

    Default

    For a while, I got some progress, but now, again all efforts stop with next error.

    I modified zimbraMailContentMaxSize, which per default is 10Mb (in bytes). I increased it to 25GB, restarted mailbox and coud finish imap account sync between two zimbra servers (v7 & v6), but still with resets of External account passwords. I modified this attribute in two cli commands:
    $ zmprov ms my.mailbox.com zimbraMailContentMaxSize 26843545600
    and the same with:
    $ zmprov mcf zimbraMailContentMaxSize 26843545600
    I could not test it with zmprov gcf, as documentation states syntax: zmprov gcf {name}, which for now does not makes a sense to me, e.g. which name?

    But after mcf modifications, I could at least find these settings with zmprov gacf | grep ...

    The account I was trying to migrate tonight is 1.6GB large. After these modifications Read timeout on UID FETCH dissapeared, and new one appeared:
    system failure: Folder sync failed, system failure: Synchronization of folder '/Ext-account/Inbox' failed, system failure: UID FETCH failed: com.zimbra.cs.mailclient.CommandFailedException: UID FETCH failed: Error in response, UID FETCH failed: Error in response, Zero-length attribute
    Sync imported arround 1300 (a bit less, than 50%) messages and stopped with the following from mailbox.log:
    2011-03-25 23:55:12,815 ERROR [ImportData-1] [name=user.name@domain.com;mid=18;ds=Ext-account;] datasource - UID FETCH failed: com.zimbra.cs.mailclient.CommandFailedException: UID FETCH failed: Error in response
    com.zimbra.cs.mailclient.CommandFailedException: UID FETCH failed: Error in response
    at com.zimbra.cs.mailclient.imap.ImapRequest.failed(I mapRequest.java:210)
    at com.zimbra.cs.mailclient.imap.ImapRequest.send(Ima pRequest.java:103)
    at com.zimbra.cs.mailclient.imap.ImapRequest.sendChec kStatus(ImapRequest.java:111)
    at com.zimbra.cs.mailclient.imap.ImapConnection.uidFe tch(ImapConnection.java:334)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.fetch Messages(ImapFolderSync.java:843)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.fetch Messages(ImapFolderSync.java:750)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.syncM essages(ImapFolderSync.java:341)
    at com.zimbra.cs.datasource.imap.ImapSync.syncMessage s(ImapSync.java:330)
    at com.zimbra.cs.datasource.imap.ImapSync.syncFolders (ImapSync.java:243)
    at com.zimbra.cs.datasource.imap.ImapSync.importData( ImapSync.java:163)
    at com.zimbra.cs.datasource.imap.ImapSync.importData( ImapSync.java:135)
    at com.zimbra.cs.datasource.DataSourceManager.importD ata(DataSourceManager.java:312)
    at com.zimbra.cs.datasource.DataSourceManager.importD ata(DataSourceManager.java:271)
    at com.zimbra.cs.datasource.DataSourceManager$1.run(D ataSourceManager.java:258)
    at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(Futu reTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.jav a:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: com.zimbra.cs.mailclient.ParseException: Zero-length attribute
    at com.zimbra.cs.mailclient.imap.MessageData.readAttr ibute(MessageData.java:83)
    at com.zimbra.cs.mailclient.imap.MessageData.read(Mes sageData.java:74)
    at com.zimbra.cs.mailclient.imap.MessageData.read(Mes sageData.java:63)
    at com.zimbra.cs.mailclient.imap.ImapResponse.readUnt agged(ImapResponse.java:106)
    at com.zimbra.cs.mailclient.imap.ImapResponse.readRes ponse(ImapResponse.java:51)
    at com.zimbra.cs.mailclient.imap.ImapResponse.read(Im apResponse.java:38)
    at com.zimbra.cs.mailclient.imap.ImapConnection.readR esponse(ImapConnection.java:648)
    at com.zimbra.cs.mailclient.imap.ImapConnection.waitF orResponse(ImapConnection.java:641)
    at com.zimbra.cs.mailclient.imap.ImapConnection.sendR equest(ImapConnection.java:518)
    at com.zimbra.cs.mailclient.imap.ImapRequest.send(Ima pRequest.java:95)
    ... 18 more
    2011-03-25 23:55:12,815 ERROR [ImportData-1] [name=user.name@domain.com;mid=18;ds=Ext-account;] datasource - Synchronization of folder '/Ext-account/Inbox' failed
    com.zimbra.common.service.ServiceException: system failure: UID FETCH failed: com.zimbra.cs.mailclient.CommandFailedException: UID FETCH failed: Error in response
    ExceptionId:ImportData-1:1301090112815:71685e8d5f97239c
    Code:service.FAILURE
    at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:248)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.check CanContinue(ImapFolderSync.java:1187)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.fetch Messages(ImapFolderSync.java:846)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.fetch Messages(ImapFolderSync.java:750)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.syncM essages(ImapFolderSync.java:341)
    at com.zimbra.cs.datasource.imap.ImapSync.syncMessage s(ImapSync.java:330)
    at com.zimbra.cs.datasource.imap.ImapSync.syncFolders (ImapSync.java:243)
    at com.zimbra.cs.datasource.imap.ImapSync.importData( ImapSync.java:163)
    at com.zimbra.cs.datasource.imap.ImapSync.importData( ImapSync.java:135)
    at com.zimbra.cs.datasource.DataSourceManager.importD ata(DataSourceManager.java:312)
    at com.zimbra.cs.datasource.DataSourceManager.importD ata(DataSourceManager.java:271)
    at com.zimbra.cs.datasource.DataSourceManager$1.run(D ataSourceManager.java:258)
    at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(Futu reTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.jav a:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: com.zimbra.cs.mailclient.CommandFailedException: UID FETCH failed: Error in response
    at com.zimbra.cs.mailclient.imap.ImapRequest.failed(I mapRequest.java:210)
    at com.zimbra.cs.mailclient.imap.ImapRequest.send(Ima pRequest.java:103)
    at com.zimbra.cs.mailclient.imap.ImapRequest.sendChec kStatus(ImapRequest.java:111)
    at com.zimbra.cs.mailclient.imap.ImapConnection.uidFe tch(ImapConnection.java:334)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.fetch Messages(ImapFolderSync.java:843)
    ... 15 more
    Caused by: com.zimbra.cs.mailclient.ParseException: Zero-length attribute
    at com.zimbra.cs.mailclient.imap.MessageData.readAttr ibute(MessageData.java:83)
    at com.zimbra.cs.mailclient.imap.MessageData.read(Mes sageData.java:74)
    at com.zimbra.cs.mailclient.imap.MessageData.read(Mes sageData.java:63)
    at com.zimbra.cs.mailclient.imap.ImapResponse.readUnt agged(ImapResponse.java:106)
    at com.zimbra.cs.mailclient.imap.ImapResponse.readRes ponse(ImapResponse.java:51)
    at com.zimbra.cs.mailclient.imap.ImapResponse.read(Im apResponse.java:38)
    at com.zimbra.cs.mailclient.imap.ImapConnection.readR esponse(ImapConnection.java:648)
    at com.zimbra.cs.mailclient.imap.ImapConnection.waitF orResponse(ImapConnection.java:641)
    at com.zimbra.cs.mailclient.imap.ImapConnection.sendR equest(ImapConnection.java:518)
    at com.zimbra.cs.mailclient.imap.ImapRequest.send(Ima pRequest.java:95)
    ... 18 more
    2011-03-25 23:55:12,905 INFO [ImportData-1] [name=user.name@domain.com;mid=18;ds=Ext-account;] datasource - Import completed for data source 'Ext-account'
    2011-03-25 23:55:12,905 WARN [ImportData-1] [name=user.name@domain.com;mid=18;ds=Ext-account;] datasource - On-demand DataSource import failed.
    com.zimbra.common.service.ServiceException: system failure: Folder sync failed
    ExceptionId:ImportData-1:1301090112845:71685e8d5f97239c
    Code:service.FAILURE
    at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:248)
    at com.zimbra.cs.datasource.imap.ImapSync.importData( ImapSync.java:168)
    at com.zimbra.cs.datasource.imap.ImapSync.importData( ImapSync.java:135)
    at com.zimbra.cs.datasource.DataSourceManager.importD ata(DataSourceManager.java:312)
    at com.zimbra.cs.datasource.DataSourceManager.importD ata(DataSourceManager.java:271)
    at com.zimbra.cs.datasource.DataSourceManager$1.run(D ataSourceManager.java:258)
    at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(Futu reTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.jav a:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: com.zimbra.common.service.ServiceException: system failure: Synchronization of folder '/Ext-account/Inbox' failed
    ExceptionId:ImportData-1:1301090112845:71685e8d5f97239c
    Code:service.FAILURE
    at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:248)
    at com.zimbra.cs.datasource.imap.ImapSync.syncFailed( ImapSync.java:392)
    at com.zimbra.cs.datasource.imap.ImapSync.syncMessage s(ImapSync.java:333)
    at com.zimbra.cs.datasource.imap.ImapSync.syncFolders (ImapSync.java:243)
    at com.zimbra.cs.datasource.imap.ImapSync.importData( ImapSync.java:163)
    ... 10 more
    Caused by: com.zimbra.common.service.ServiceException: system failure: UID FETCH failed: com.zimbra.cs.mailclient.CommandFailedException: UID FETCH failed: Error in response
    ExceptionId:ImportData-1:1301090112815:71685e8d5f97239c
    Code:service.FAILURE
    at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:248)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.check CanContinue(ImapFolderSync.java:1187)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.fetch Messages(ImapFolderSync.java:846)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.fetch Messages(ImapFolderSync.java:750)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.syncM essages(ImapFolderSync.java:341)
    at com.zimbra.cs.datasource.imap.ImapSync.syncMessage s(ImapSync.java:330)
    ... 12 more
    Caused by: com.zimbra.cs.mailclient.CommandFailedException: UID FETCH failed: Error in response
    at com.zimbra.cs.mailclient.imap.ImapRequest.failed(I mapRequest.java:210)
    at com.zimbra.cs.mailclient.imap.ImapRequest.send(Ima pRequest.java:103)
    at com.zimbra.cs.mailclient.imap.ImapRequest.sendChec kStatus(ImapRequest.java:111)
    at com.zimbra.cs.mailclient.imap.ImapConnection.uidFe tch(ImapConnection.java:334)
    at com.zimbra.cs.datasource.imap.ImapFolderSync.fetch Messages(ImapFolderSync.java:843)
    ... 15 more
    Caused by: com.zimbra.cs.mailclient.ParseException: Zero-length attribute
    at com.zimbra.cs.mailclient.imap.MessageData.readAttr ibute(MessageData.java:83)
    at com.zimbra.cs.mailclient.imap.MessageData.read(Mes sageData.java:74)
    at com.zimbra.cs.mailclient.imap.MessageData.read(Mes sageData.java:63)
    at com.zimbra.cs.mailclient.imap.ImapResponse.readUnt agged(ImapResponse.java:106)
    at com.zimbra.cs.mailclient.imap.ImapResponse.readRes ponse(ImapResponse.java:51)
    at com.zimbra.cs.mailclient.imap.ImapResponse.read(Im apResponse.java:38)
    at com.zimbra.cs.mailclient.imap.ImapConnection.readR esponse(ImapConnection.java:648)
    at com.zimbra.cs.mailclient.imap.ImapConnection.waitF orResponse(ImapConnection.java:641)
    at com.zimbra.cs.mailclient.imap.ImapConnection.sendR equest(ImapConnection.java:518)
    at com.zimbra.cs.mailclient.imap.ImapRequest.send(Ima pRequest.java:95)
    ... 18 more
    So, how to interpret this error? From one side, it says, that there is error in response, then it says, that import completed, but then again - import failed.

    Are there really nobody, who could help me to find the right way of dealing with this issue?

  7. #7
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    141
    Rep Power
    6

    Default

    And in addition, after this last error, I do not have a possibility to resave external account password. Going to Preferences, writing down passowrd again, saving with successfull test - nothing happens according to logs, but modal box says, that preferences are saved. Going back to preferences again - the same error without any activity.

  8. #8
    brickjenks101 is offline Intermediate Member
    Join Date
    Feb 2011
    Posts
    17
    Rep Power
    4

    Default

    I am having same issue on mailbox larger than 6gb

    we are on ZCS OSE 7

    zimbraDataSourceLastError: system failure: Folder sync failed, system failure: Synchronization of folder '/Cliff China/INBOX' failed, system failure: UID FETCH failed: com.zimbra.cs.mailclient.CommandFailedException: UID FETCH failed: Error in respon se, UID FETCH failed: Error in response, Exception in data handler, Unexpected end of stream

  9. #9
    windyandy is offline Starter Member
    Join Date
    Aug 2011
    Posts
    1
    Rep Power
    3

    Default

    Hello is there any progress on solving this issue? I am having very similar problems with a new ZCS OSE 7.1.2 and having connected 2 external IMAP accounts >6GB each.
    Here the errors :
    1.) standard IMAP at lunarpages.com: >6 GB
    ...synchonization of folder .../INBOX failed, ....
    UID FETCH failed: com.zimbra.cs.mailclient.CommandFailedException... ..
    Timeout waiting for response, Read timed out


    2.) [the @gmail.com account] > 6 GB
    ..Synchronization of folder INBOX failed.... unable to rename remote folder to Inbox, RENAME failed: Duplicate folder name Inbox (Failure)
    ->there is no duplicate....
    Since yesterday when I installed ZCS I have seen this several times although the gmail account seems sometimes to work but has transfered less data (<500 MB) the 1st one already has 3 GB pulled over but cannot be used for sending and receiving ==> same problems as j2b described.
    Where / how can I increase TIMEOUTS and MESSAGESIZE etc.. in case it would help (some other posts had success others not). I am new to Zimbra.
    [the timeouts may be related to my location in a very large country in Asia where there seems to be an IT-policy to discourage out-of-country mail-systems - so eg gmail is artificially sloooowed down. In order to have a working mail system for our small workgroup I want to use ZCS in a local data center]
    ==>the aim is just to get those messages from IMAP over and put them in the respective Zimbra standard account = is maybe an import via Zimbra Desktop or Thunderbird better? Thanks.

  10. #10
    brickjenks101 is offline Intermediate Member
    Join Date
    Feb 2011
    Posts
    17
    Rep Power
    4

    Default

    My issue was caused by trying to import a mailbox from a server in china, I believe the problem was the connection was slow, for example webmail took minutes to load. ZCS should be able to handle poor connections without breaking so this is still a bug.

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Error importing Mailboxes from Exchange/PST
    By mtanzer in forum Migration
    Replies: 23
    Last Post: 09-18-2012, 09:49 PM
  2. Outlook Connector with realy large Mailboxes
    By yogg in forum Zimbra Connector for Outlook
    Replies: 1
    Last Post: 11-15-2010, 09:11 AM
  3. Importing Mailboxes
    By de_lucca in forum Installation
    Replies: 6
    Last Post: 02-26-2008, 06:25 AM
  4. Importing mailboxes from Apple Mail
    By swu in forum Migration
    Replies: 10
    Last Post: 07-11-2007, 07:48 AM
  5. large mailboxes
    By bakers in forum Administrators
    Replies: 1
    Last Post: 12-22-2006, 11:27 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •