Help needed migrating from FirstCLass

[tenpagyatso]

We're in the process of migrating from FirstClass (FC) 5 to Zimbra. Our interim solution is to have both our FC server and Zimbra server running concurrently.

As mail comes to our network it enters our spam catcher, which is configured to pass all email to our Zimbra box. We're a small college and have 200+ new students entering this fall. So on the Zimbra box we've created full user accounts for all new students and created aliases in Zimbra to pass email for current FC users on to the FC server.

The problems lies with our current FC users trying to send email to any of the new students who have full accounts on Zimbra. Short of manually creating either limited user accounts on FC to forward all new student email back to the Zimbra box or hack some sort of FC multiple public list/conference we're stuck on how to make an interim solution.

So if you've successfully migrated to Zimbra from FirstClass please let me know if your situation was at all like what I've described. And as usual, if any of this is unclear please ask and I can provide more detail.

Thanks,
Steven

[gnyce]

Sorry, can't help you with the First Class side of things, but what we did to co-exist Zimbra and our other/legacy mail server was to use the spamcatcher (linux box with postfix) to route between the two. If an address/account did not exist on our legacy server, it would forward to the spam catcher (same with Zimbra), and with postfix on that server, we used the Transports file to tell postfix which server to forward mail two.

[jsnapp]

We're at the tail end of migrating from FirstClass 8 to Zimbra.

For a short period of time we ran both systems sided by side with our IT staff using Zimbra while the rest of our students, faculty, staff (~2500 users) continued to use FC.

I setup postfix transport files in our anti-spam email gateways and in zimbra the same as the person who posted before me.

As for the First Class system I used a batch admin script to create redirect rules for each of the 20 or so IT staff accounts so that no one still using First Class would notice any difference that they were now sending their email to IT staff who were using Zimbra.

So this method would require you to create new student accounts in FC and craete redirect rules for each of their accounts.

With all that said I can also say it was nice to make the final cutover for ALL users to Zimbra this past Monday. I'm thrilled with how Zimbra is turning out for us.

[sglewis]

Hey, I saw you moved from fc to Zimbra. We are preparing to do something similar, any tips? I keep seeing warnings that imap doesn't expose all messages, especially older mail flagged as read and sent items. Did you have trouble, and did you use imapsync or another tool? Also did you move conferences to Zimbra, we are assuming we'll use resource accounts to replicate these, how did you handle the mapping of permissions?

[jsnapp]

First let me say that my experience with FirstClass migration was back in 2008. Things may be different with the version of FirstClass you are running now.

At the time we had about 2660 accounts. I migrated a little more than 2630 accounts with 100% success and the remaining 20+ accounts were satisfied with less than 100% migration. What was really amazing is that our migration process included all FirstClass email, uploaded file, documents, forms etc from user accounts. As far as I can tell we are still the only ones who have had that kind of success migrating away from FirstClass.

I'll start with our easy solutions.

We made users migrate calendars if they had them. Support staff were available for this process if needed.

We used regular Zimbra accounts for departmental use and shared these departmental accounts to appropriate users. I didn't have an automated way to capture permissions from FirstClass. I gathered required permissions from the people who knew what they needed and then used scripts or manual entry to create departmental accounts, shares and permissions on the Zimbra server.

Ok, here's the hard stuff.

Back in 2008 I researched the FirstClass support forums and the web in general for what others had done for migration and it all pointed to the same conclusion. Migrating all FirstClass content to any other system is almost impossible.

The most significant cause is that FirstClass's imap protocol implementation is not standards compliant. As you said, imap would not even include any sent email. I can't remember if it didn't include read email. During my research of other FirstClass user experiences I found some people used imapsync and told their users to simply live with the fact that so many email would not get migrated. I tried a number of ways to get things to work with imap and nothing worked. Even when things seemed to work a little it was very inconsistent which made it completely unreliable.

Most email systems (gmail, zimbra, exchange server, etc...) expect to use imap to sync folders when migrating so this posed a significant problem.

At this point we're only talking about email migration and we can already see how disappointing the results will be.
When considering how you will migrate other items like uploaded files, documents, forms, calendars, etc you will find that there is no simple way to get access to them.

Basically I concluded there was no simple/standard way to migrate from FirstClass so I came up with a complicated way.

I discovered that FirstClass's pop3 protocol was consistent in what it did but it still did not include sent email, uploaded files, etc. Of course, pop3 also limits you only to the user's inbox. But, at lease it was a start.

I created a temporary FirstClass account for migration purposes. Let's call that account "mig".
I used batch admin scripts to copy a folder from userA's account to the mig account.
Then used a perl script to connect to mig's account using pop3 and connect to userA's account on the zimbra server using imap and sync the email.
I inserted a unique id into each email as I copied it to the Zimbra server so the next time this folder was sync'd I could first compare the contents of the folder on FirstClass with the contents of the folder on Zimbra thereby establishing a true synchronization process.
Then I'd empty the mig account and copy userA's next folder to it and migrate it to Zimbra and so on.

The really big breakthrough came when a coworker suggested that I could modify the results of the batch admin folder copy script which described each item in the FirstClass folder that was being migrated. This meant that I could change the behind the scenes description of sent items, documents, forms, uploaded files, etc to look like regular email that would be visible via pop3.

So, I setup a separate linux server (let's call it "migserver") to handle the migration process between FirstClass and Zimbra. This server made batch admin requests to FirstClass. The results of those batch admin scripts were emailed to the migserver where they would be manipulated by the migration script and resubmitted to the FirstClass server as a new batch admin script.

FirstClass still turned out to have so many complicating issues so I just tweaked my migration script to solve each one if possible.
There's even more to the process than this but this should at least capture the idea of what I did.

The migration of some of our larger accounts (greater than 200MB) took quite a while with this process on the first pass but with subsequent passes the duration would shorten since it was a true sync.

I hope that information is helpful. Let me know if I can be of more help.

[sglewis]

This is awesome, thanks. I see how (relatively) simple this might be to start putting together, however, I've noticed that doing a dir +b on user mailboxes returns ONE LEVEL of folders only (it's not recursive). Had you solved that problem, or was that the bulk of your migration server's scripts, dealing with traversing deeper and deeper through folders to see just how much there was to copy?

[jsnapp]

Your script will have to do the traversing.

As with most of the components of the migration script you'll find that conceptually things won't seem too difficult. But, once you start dealing with real results you'll realize that there are a lot of gotcha's to deal with.
Some of them are problems inherent with the way FirstClass does things.
Others have to deal with differences between the way FirstClass and Zimbra do things. For example, each system allows/disallows for certain characters in a folder name. Your script will have to take that into account and make conversions as necessary.

Another thing to consider is that the method I've described requires a batch admin script to email to the migration server the actual contents of a folder as a single email so it can be manipulated by the migration server and sent back to the FirstClass server. Those contents include all the attachments as well. Obviously that could make for a very large email. Once I decided what maximum size I wanted that single email to be I wrote my script so it would send a "DIR" batch admin request to get a date sorted list of each item in the folder being migrated. Then, as best I could, I would pick date ranges where the items within that range would not exceed the maximum size I chose. Then I would run the batch admin script that would export that specific date range of the folder, manipulate it, migrate it and then move onto the next date range until the folder is migrated and then do the same thing with the next folder until all folders are migrated. Of course if a folder is small enough then it could be migrated in one pass.

This certainly adds to the complexity but it was necessary. The rest of the complexity comes from correctly manipulating that single email so it can go back to the FirstClass server and transform as many items as possible into items that can be retrieved via pop3 and thereby migrated to Zimbra via imap.

[sglewis]

Ok... many, many false starts in (I have like 5 partially working scripts that do various small pieces here), I have two questions for you...

Given the following snippet from export:

// Reference: 5001143:128828372
New Relative "" "Administrator" "test subject" Message 23032 0 0 0 128828372 -U+R
Put Previous 8120 7 1252 8140 0 8141 0 8126 -925743380 9 "test subject"
Put Previous 8 "Administrator" -V
Put Previous 4 0 "Scott Lewis" -V
Put Previous 4 1 "Hartley Hinds" -V
Put Previous 5 0 "Administrator" -V
Put Previous 6 attach3 +FS

Put Properties Previous 1018 14 -925743380

I've deduced the following:

From: Administrator
Subject: test subject
Two recipients: Scott Lewis and Hartley Hinds
One CC: Administrator
Plus an attachment to the message "attach3" which contains the body of the message.

1) where on earth is the date hidden? I know I can match up to a "dir" command to get date/time, but it has to be in here, or it couldn't be recreated in a FirstClass to FirstClass migration, which 'export' is designed for in the first place.

2) The attachments have some funny characters in the beginning and end. Did you encounter that? Did you solve that, or ignore it?

My current plan, so you know:

1) Create a HUGE list of export messages, with the subject containing the USERNAME and PATH, so I know where to put the mail on the other side.
2) Email those exports (breaking down large folders by size, constrained by date, like you suggested) and push them into a migration SMTP server.
3) Grab the raw files out of Maildir, and reproduce each message one by one, creating an RFC-822 message by faking some headers, adding the to: from: subject: date etc, and then copying in the text of the messages.
4) Using zmmailbox's addmessage facility to inject them right into the proper place in Zimbra.

this avoids imapsync (never grabs all mail) and pop2sync which also ignores sent, and occasionally chokes on a message.

How am I doing? Close?

Calendars seem fairly easy, since I can ICS export them. We'll take the hit to export group calendars manually, and we'll have users export their own, either emailing them to us (noobs) or importing themselves in Zimbra (relatively proficient users). Contacts I haven't looked at, but am not worried.

We rely on no other main features, other than conferences, which should export/dir just the same as regular mail.

[jsnapp]

1) I'm almost certain the datetimestamp is encoded in the value -925743380.
FirstClass items don't always have that value in the "Put Previous 8120..." line but for migration purposes it has to be put in there. Just another of the gotchas I've been mentioning. You shouldn't need to actually use these values for anything else though. Creating your date range of items to export is easier done with the results from Batch Admin "Dir" command. At least that's how I did it.

2) Off hand I'm not sure what you mean by strange characters in the attachments. You might need to show me an example. There are issues you'll have to deal with when it comes to attachments but I don't remember having these particular problems.

Sounds like you're handling calendars the same way we did. I think contacts can be migrated into zimbra in a similar way without too much trouble as well.

I like the sound of "zmmailbox addmessage" I wonder if it was there 2 years ago but I don't think batch admin export results will give you RFC-822 compliant content. Which is why I went the route I did.

I also wanted to clarify that I agree that pop3 access alone won't see the "sent" email. However pop3 access in FirstClass is at least far more reliable when pulling email from the inbox in contrast to FirstClass's imap protocol which to put it simply should just be considered broken. FirstClass employees even admit to this. I don't think batch admin export results will give you RFC-822 compliant content which is why I chose to modify those export results and send them back to FirstClass to populate a migration account. By modifying I was able to get "sent" email as well as other FirstClass items to be visible by pop3.

I've submitted my code to github and am in the process of cleaning it up to make it more usable by others. Feel free to take a look at firstclass2imap.

[sglewis]

I'll be looking at your code, that's both very gracious of you to post, and very exciting for us to get our hands on. Yes, zmmailbox addmessage is a wonderful method, although the similar command line method to create a folder seems abnormally slow in our test environment (2.5 to 3.5 seconds per folder create on an idle, yet powerful server).

I deduced yesterday that the -925743380 value is a date stamp, but was unable to decipher it. Comparing the message to the results of a DIR helped me get the information, but it seemed like a needless, wasted step. Sure would be nice to see a epoch time or something that just stands out as obvious.

No, batch exports doesn't give me RFC-822 content, but I have been able to CREATE an RFC-822 message based on the results, although it seems pretty easy to overwhelm the system with large exports, which is fine for my test bed, but not fine when we do it in production. One thought was creating "X" export commands, with one for the inbox, plus one for each subfolder + it's recursive subfolders (ie: INBOX:Test and INBOX:Test:Test2), with a subject line to help identify the user and folder contained within the export's returned message.

From there, POPing those out, then splitting them into RFC-822 messages adding enough fake headers to make it "kosher" so to speak, and then creating destination folders in Zimbra, then adding them in, one at a time (well, several at a time through threading, but you get my drift).

I had hoped to decode the date/time stamp in the export, as this would make the script much easier. Right now, I use an export sans messages to get the list of folders (better than DIR, since I can exclude linked conferences, which we don't want to import over and over again). Then I do the individual exports, but I'm having to run a DIR and compare the message ID... but just to get the stupid time stamp!

I've kind of put that aside, though, as I am finding that (and perhaps they've improved it slightly) IMAP access is "fairly" reliable in a controlled fashion. So the script I'm playing with now:

1) Telnets to the IMAP server and uses expect to dump a list of folders. I then parse out just folders whose path begins "INBOX/Zimbra Migration/*" as we're encouraging users to cleanup, and move only folders absolutely necessary to speed the migration.

2) Grabs each folder, one at a time into local Maildir folders. I'm ignoring any message larger than 15mb, as that seems to be the point in which imapsync, mailsync and ten thousand other sync scripts fail when talking to FirstClass.

3) Imports the mail into Zimbra.

For whatever reason, the script that works best so far in pulling data from FirstClass is not meant to work in the fashion that IMAPsync does, so the only choice is to stick it in mbox, MailDir, or "submit it" to an MTA, which of course would spawn actual message deliveries. Hence the extra step.

And no, this doesn't mitigate the requirement for users to COPY sent mail into a folder, and then UNSEND the copies. Without that step, it's hidden from IMAP. Without the prior COPY, the actual unsend will pull the message out from the recipient, if it's a local FirstClass user who still has the message laying around.

My immediate higher ups have resigned them to the facts that end users will have to take steps here, but if the bulk of the mail transfer is automated, this will help.

This seems much quicker, since I'm no longer doing ANY exports and imports, it makes the SAN guys happier since I'm no longer asking for 1tb of scratch space and it seems "fairly" quick, which is a extra benefit, since I'd actually given up on the prospects of doing it "quick" and was resigned to it taking something closer to forever and a day.