Help needed migrating from FirstCLass

[jsnapp]

I'm not sure if I'm missing something but I know I didn't need a lot of storage space for my migration process. At any point in time I would only need storage space to hold a single folder from a single user's account. This is a small issue compared to what follows but I just thought I'd include it.

The big issue is that you're saying imap is fairly reliable.
That would be fantastic news as I'm sure others would be glad to know there is an easier migration path from FirstClass than there used to be.

I wonder if we can be sure that it's more reliable.
I often tested with small accounts only to find the problems later while migrating larger accounts. If you're not already, then you might try a medium to large account with many folders and subfolders as a test every now and then.

This brings up one of the many issues I ran into with FirstClass. It allowed multiple folders at the same level to have the same name. Zimbra doesn't (or at least didn't) allow this. FirstClass allowed the use of certain characters in folder names that Zimbra didn't allow. This second problem might be able to be resolved by the use of imapsync's --regextrans2 parameter. My migration method took care of both of these problems.

Here's a quote from a May 2009 post where someone claims imap is broken with no hope to be fixed.
FirstClass 9 to Zimbra

"Don't bother with imapsync, it won't work. FC IMAP is broken. Period. Even their product manager Scott Welch agrees that IMAP is broken (his suggestion for using IMAP is to have two separate FC accounts, one you access via the client, one you access via IMAP, and don't ever mix the two). They have no plans to fix IMAP, but will gladly advertise it as working in all their product literature."

Here's another post in June 2010 from the same user.
imapsync both ways?

I don't think the user indicates what version of FirstClass they are using.

On the upside I did find this post July 2010 that indicates someone is migrating large mailboxes from FirstClass using imapsync.
Issue migrating large mailboxes from FirstClass to Dovecot
This thread is a little long but well worth the read if you are trying to use imap to migrate FirstClass. The short version of the story is that using the latest versions of imapsync (1.359) and Mail::IMAPClient (3.25) you may very well be able to migrate from FirstClass using imap. You may need to include some special parameters for imapsync as indicated in some of the thread posts though.

I think this information would be invaluable to administrators who are looking to migrate from FirstClass so if you are able to confirm that it works please post here.
Of course, you still won't get the "sent" email unless you do as you suggested by copying them and "unsending" them.

I still think there is nothing imap can do to migrate things like "uploaded files" or "collaborative documents" but please correct me if I'm wrong.

I think imapsync is a very powerful tool and would be better suited to your task than the script you mentioned. I'm probably not understanding your requirements though. Does the script you are using handle syncing or will you just disable a user's account and migrate in one shot?

I hope you understand that I don't have a particular interest in my method being used.
I just don't want anyone to go too far down a path (imap) if it ends up failing in someway.
I spent many months working around FirstClass issues and I don't wish that on anyone else.

Please let me know what your results are with imap reliability.

Thanks

[sglewis]

I'm not sure if I'm missing something but I know I didn't need a lot of storage space for my migration process. At any point in time I would only need storage space to hold a single folder from a single user's account. This is a small issue compared to what follows but I just thought I'd include it.

Brain fart, you are correct. Although our snapshot reserves might get hit pretty hard on the SAN, but that's easily mitigated.

The big issue is that you're saying imap is fairly reliable.
That would be fantastic news as I'm sure others would be glad to know there is an easier migration path from FirstClass than there used to be.

I wonder if we can be sure that it's more reliable.
I often tested with small accounts only to find the problems later while migrating larger accounts. If you're not already, then you might try a medium to large account with many folders and subfolders as a test every now and then.

I've been testing on mailboxes with 1500 messages to a folder, 3-4 folders at a time, while I refine the script a bit. I plan on adding many, many more folders and will report back if we hit great success. Understand that this is not without caveats, so far we've decided to:

1) Force users to MOVE folders of importance to a sub-folder called "Zimbra Migrations".
2) Not have any folders deeper than 1-2 levels.
3) COPY sent mail and UNSEND it.
4) SAVE attachments over 15mb as we will be skipping them unilaterally due to socket time outs that we can't seem to beat. Their other choice is to "forward them in to Zimbra".

This brings up one of the many issues I ran into with FirstClass. It allowed multiple folders at the same level to have the same name. Zimbra doesn't (or at least didn't) allow this. FirstClass allowed the use of certain characters in folder names that Zimbra didn't allow. This second problem might be able to be resolved by the use of imapsync's --regextrans2 parameter. My migration method took care of both of these problems.

I've hit that already. In both cases, I'm working around it. The script will alert me to any user with multiple, identically named folders, such that we can yell at them to rename one to be unique. I'm also mapping ANY and ALL non-kosher characters to Zimbra-friendly ones during the import process.

Here's a quote from a May 2009 post where someone claims imap is broken with no hope to be fixed.
FirstClass 9 to Zimbra

"Don't bother with imapsync, it won't work. FC IMAP is broken. Period. Even their product manager Scott Welch agrees that IMAP is broken (his suggestion for using IMAP is to have two separate FC accounts, one you access via the client, one you access via IMAP, and don't ever mix the two). They have no plans to fix IMAP, but will gladly advertise it as working in all their product literature."

Here's another post in June 2010 from the same user.
imapsync both ways?

I don't think the user indicates what version of FirstClass they are using.

I'm familiar with both threads, and a few others here (and on other forums). I have found imapsync to be VERY unreliable... specifically for FirstClass. Which is a shame, I've used the tool quite a bit in the past, and it's a very solid tool. It's biggest problems are finding messages with incomplete headers (or so it complains) and skipping them.

I've found getmail ( getmail version 4 ) to be a lifesaver to this end. It's a python based replacement for fetchmail that works fairly fast, fairly clean, and pulls in all of my test mail (can't wait to try this out on some real mailboxes soon).

I *LOSE* the ability to IMAP from FirstClass right into Zimbra, but the time I save by not using the EXPORT script command and then running off the resulting returned script to setup the migration account's inbox over and over again should outweigh that easily. Now, I'm basically IMAPing out, then zmmailbox addMessageing in. The other method is export, import, pop, import to zimbra.

What my script will be doing (still working on it, although bits and pieces for sure work):

1) Read a master list of users.
2) Scan INBOX/Zimbra Migration/ for any and all sub-folders.
3) STOP if there are duplicate named folders or too many nested folders, and move on to the next user.
4) Build folder-by-folder Maildir on file storage. For speed, I'm going to use a separate LUN on our SAN from Zimbra's mail store, so I can read from one set of spindles and write to another. Once I get the script to be threaded (run multiple mailboxes at a time) this should greatly help performance, or so I believe.
5) Build a getmailrc for INBOX, plus all of the designated folders, one by one, and then run getmail to IMAP out the folder into the appropriate maildir.
6) Use zmmailbox to create matching destination folders, translating illegal characters as needed.
7) Once all of a user's content is pulled, use zmmailbox to import the now RFC-822 compliant messages into the user's account.

I should mention that each step has a substep, for example 1a) pray, 2a) pray.

On the upside I did find this post July 2010 that indicates someone is migrating large mailboxes from FirstClass using imapsync.
Issue migrating large mailboxes from FirstClass to Dovecot
This thread is a little long but well worth the read if you are trying to use imap to migrate FirstClass. The short version of the story is that using the latest versions of imapsync (1.359) and Mail::IMAPClient (3.25) you may very well be able to migrate from FirstClass using imap. You may need to include some special parameters for imapsync as indicated in some of the thread posts though.

This is where getmail has been invaluable. It seems to pull in more junk from FirstClass than IMAPsync that thread indicates IMAPsync might be tunable to have similar success.

Please note: this is not a knock on IMAPsync. It's a continuing knock on the lack of standards implemented by FirstClass, plus the general unreliable nature of just about every external facing service it has, except interaction with it's own fat client.

I think this information would be invaluable to administrators who are looking to migrate from FirstClass so if you are able to confirm that it works please post here.
Of course, you still won't get the "sent" email unless you do as you suggested by copying them and "unsending" them.

I still think there is nothing imap can do to migrate things like "uploaded files" or "collaborative documents" but please correct me if I'm wrong.

Luckily, we use almost no features outside of conferences, email, contacts, calendars and group calendars. We use some forms, but they have to live with that. We use the history feature, but we're exploring recreating that functionality, perhaps through a custom zimlet. All the other stuff we don't use, except perhaps the resume, but not to any extent that the internal Zimbra GAL can't handle as we use it for contact information essentially.

I think imapsync is a very powerful tool and would be better suited to your task than the script you mentioned. I'm probably not understanding your requirements though. Does the script you are using handle syncing or will you just disable a user's account and migrate in one shot?

My current thought is to introduce a couple of load balanced postfix servers in between my external MTAs which process inbound mail for spam and the FirstClass server + zimbra cluster we're building. Using a simple transport file, we'll direct mail to EITHER FirstClass OR Zimbra.

Basically, we'll be dumping users (after training) into Zimbra a few hundred or so at a time. They'll continue to have access to FirstClass to look up old data, to cover the gap between the beginning of that batch of migrations and the completion. But new mail will flow into Zimbra and not FirstClass for them.

This leaves the question of how users will email each other using native email addresses between the two systems, which will be simple enough I just haven't spent a second thinking about it. Perhaps using a smarthost alone will solve that issue. Trying to hook the GAL up to FirstClass's FAKE LDAP server has been a nightmare by the way. And it is FAKE, no matter what base search path you use, including aklsjdaklsjdaklsd you get all users back. And queries seem inconsistent in their return. These are minor issues, as I'll BUILD an LDAP server myself if I need to.

We just have too much data to do regular syncs between the two systems, quite honestly. You said yourself in a prior email that it took on the order of a couple of days the first time you moved a guy with 600mb or so of mail, that just won't work for us, we have half a tb of data.

I hope you understand that I don't have a particular interest in my method being used.
I just don't want anyone to go too far down a path (imap) if it ends up failing in someway.
I spent many months working around FirstClass issues and I don't wish that on anyone else.

Please let me know what your results are with imap reliability.

Thanks

The same mail retrieval product I'm using also supports POP3. That would only involve me rewriting say 70% of my script if I had to go back to exports/imports and migration accounts.

I *WILL* be reporting back, if anyone saves some time and headaches as a result of your work, and possibly as a result of mine, that would be awesome!

[jsnapp]

Thanks for such a thorough write up. I'm certain that this kind of information will be helpful to others.

I'm very glad to hear there is an easier and faster way to migrate. This is definitely good news.

When I migrated I only required the user to move everything under their Mailbox folder in FirstClass. I didn't want to rely on them to do much else but it sounds like I could have saved myself a lot of work by relying on them some more. I'm pretty sure we had sufficient usage of collaborative documents and uploaded files to warrant my approach but I should have examined the possibility of relying on users to manually migrate this content.

I looked back at my logs this morning and it looks like it actually took about 16 hours for the initial migration of about 600MB which admittedly is still long. I was able to run multiple instances of my process similar to what you're planning to do so that helped me some. Once the initial migration of an account was done the subsequent syncing migrations were much faster since they wouldn't remigrate content.

I'm glad to learn more about getmail. I think I tried it before but I didn't think to make it work the way you're using it. My goal was more for a true syncing process but as you said that requires a lot more time given the amount of data needing to be migrated.

Your plan sounds very well thought out. In my opinion, it sounds like you've covered your bases well.

I only have two remaining thoughts about this.

1. You said imapsync might be tunable to have similar success as getmail. Is that worth pursuing since it means you could skip the temporary storage of content and just migrate directly to Zimbra?

2. Even if you stick with getmail for getting content out of FirstClass I wonder if you should consider using imapsync (or other tool) to migrate the content from your temporary MailDir storage to Zimbra.
This might avoid some of the slowness with zmmailbox commands that you've reported. I've also seen slowness of some commands on our production Zimbra server.
I'm not familiar with the "zmmailbox addmessage" command but I wonder if doing the final migration to Zimbra via imap would avoid any unnecessary complexities since imap transfer is pretty standard whereas "zmmailbox addmessage" is not.

[sglewis]

When I migrated I only required the user to move everything under their Mailbox folder in FirstClass. I didn't want to rely on them to do much else but it sounds like I could have saved myself a lot of work by relying on them some more. I'm pretty sure we had sufficient usage of collaborative documents and uploaded files to warrant my approach but I should have examined the possibility of relying on users to manually migrate this content.

I suppose our approaches are quite similar, I just think the added step of moving existing folders under Mailbox helps ensure there are no duplicate names, and also helps encourage the cleanup we wish to do.

I looked back at my logs this morning and it looks like it actually took about 16 hours for the initial migration of about 600MB which admittedly is still long. I was able to run multiple instances of my process similar to what you're planning to do so that helped me some. Once the initial migration of an account was done the subsequent syncing migrations were much faster since they wouldn't remigrate content.

I'm glad to learn more about getmail. I think I tried it before but I didn't think to make it work the way you're using it. My goal was more for a true syncing process but as you said that requires a lot more time given the amount of data needing to be migrated.

I would LOVE to do a sync, but I fear our data sets are just too much larger, and the FC bottleneck too much to overcome. I have some multi-gigabyte boxes, and frankly, this is a HUGE concern with any method. But having timed the sync process, it just seems that it will take so long just to query the FC side, even sizing up messages in a folder seemed absolutely horrendously slow.

I only have two remaining thoughts about this.

1. You said imapsync might be tunable to have similar success as getmail. Is that worth pursuing since it means you could skip the temporary storage of content and just migrate directly to Zimbra?

2. Even if you stick with getmail for getting content out of FirstClass I wonder if you should consider using imapsync (or other tool) to migrate the content from your temporary MailDir storage to Zimbra.
This might avoid some of the slowness with zmmailbox commands that you've reported. I've also seen slowness of some commands on our production Zimbra server.
I'm not familiar with the "zmmailbox addmessage" command but I wonder if doing the final migration to Zimbra via imap would avoid any unnecessary complexities since imap transfer is pretty standard whereas "zmmailbox addmessage" is not.

This is an excellent suggestion... once I'm done, since i have valid Maildir, I would just need to install a local IMAP server on an alternate port. I will def. check this out, thanks!

I can honestly say that you've been invaluable, in saving my countless hours of discovery while formulating a plan here. Your employer should be happy to have you.

[jsnapp]

Thanks for your words of encouragement. I want my experiences (good and bad) to directly benefit others. I thank God for giving me the opportunity here.

As I mentioned before, your work has taught me a lot too and I'm sure it will benefit many others.

Good luck with the rest of your migration and don't forget the substeps you mentioned before.

[sglewis]

bla1 - OpenText has recently released a tool that will export mailboxes into an mbox format, which is easily importable. It has the same problems their own "batch export" process has for us, which is to say if batch export can't export a particular message, neither will this tool be able to, but it's world's better than the above methods, and much closer to 100% complete.

It does not to well with unaddressed saved drafts, certain types of attachments (specifically, FC messages attached to another FC message), ICS appointments, and doesn't work with any non-email format, such as contacts, calendars, or document types.

Sounds like you're small enough to make it work, however, and contacts and calendars are relatively easy to move manually anyway.

[sglewis]

I will have to admit, I have no knowledge of that era's FirstClass, but wish you luck!

[jsnapp]

I just mailed with an employee of opentext. He told me that the server version 5, which i have is too old to use the export tool.

I had a look at the firstclass2imap export tool, but i don't know exactly how to use it. It seems it cannot connect to my sql-database-server. I'll figure that out soon.

Henning

I'm glad OpenText has finally worked to provide a decent export tool. I'm reluctant to lead anyone down the path of using the "script" that I wrote only because it's a pretty involved setup. I used it about 3 years ago and it worked pretty well. I think we were running FirstClass version 5 but it's been so long I don't remember off hand. If you're still entitled to upgrades you may want to go through the trouble just so you can use their new export tool. Otherwise you might be able to use the many options that the imapsync tool has to work around the FirstClass limitations. I'd even suggest modifying the code of imapsync if it helped migration before suggesting using my tool.

Good luck. Let me know if I can be anymore help.

[truezjz]

Hi Guys,

I got lots of inspiration from this thread;

Currently we are also dealing with FirstClass. Can someone point out In FC, is it possible to use Export command to export a individual message if I know the FC unique-id?

Thanks.