occasional imapsync invalid date format error

[oayfer]

Hello,

I am testing migration from a Solaris 9/dovecot system to Zimbra 5.0 on RHEL5. imapsync seems to work very well except for some occasional errors like:

Use of uninitialized value in concatenation (.) or string at /usr/bin/imapsync line 1340.
10791 BAD parse error: invalid date format
 at /usr/bin/imapsync line 1358
Couldn't append msg #28587 (Subject:[something]) to folder INBOX: 10791 BAD parse error: invalid date format

By occasional, I mean a few hundred such erroneous e-mails for the entire migration of 300000+ e-mails.

I tried finding the bad date headers, but can't see anyhting wrong with them. Here are two examples:

Date: Sun, 31 Dec 2006 17:18:22 -0800

Date: Mon, 31 Dec 2007 16:04:21 -0800

This led me to check the dates associated with more of the errors, and they are all December 31st!

Any ideas what is going on here?

Thanks.

[oayfer]

Running with the --debug flag produces these two interesting lines:

internal date from 1: [31-Dec-2007 16:04:27 +4000]
internal date from 1: [""] (fixed)

compared to a good e-mail that produces:

internal date from 1: [18-Mar-2008 18:34:21 -0700]
internal date from 1: ["18-Mar-2008 18:34:21 -0700"] (fixed)

+4000 doesn't make sense as a time offset, and I still don't know where that comes from.

In my understanding, the internal dates as derived from an mbox file come from the very first line of an e-mail that begins with "From ". In this example, those lines are as follows:

broken:

From user@domain.com  Mon Dec 31 16:04:27 2007

not broken:

From user@domain.com  Tue Mar 18 18:34:21 2008

On the same system where I'm running imapsync, I have tried this tiny perl program:

#!/usr/bin/perl

require Date::Manip;
Date::Manip->import(qw(ParseDate Date_Cmp UnixDate));

$a = 'Mon Dec 31 16:04:27 2007';

$d = ParseDate($a);

$dd = UnixDate($d, "%d-%b-%Y %H:%M:%S %z");

print $dd;
print "\n\n";

which returns:

31-Dec-2007 16:04:27 -0700

Which is the correct answer (note -0700 vs +4000). So the date manipulations involved in imapsync seem to be working correctly.

This is the last roadblock I have come across in the process of migrating to Zimbra. Any help would be greatly appreciated.

Thanks.

[bthom73]

Hi Oayfer,

Oddly enough, I just ran into the same thing. Any luck figuring out what was causing it?

The user in question here is migrating from Dovecot as well, so maybe it's somehow related to Dovecot? Or maybe a bug in imapsync?

Thanks,
Brian

[oayfer]

Over 3 years later I'm having trouble remembering the details but I think we ended up simply isolating the handful of messages among hundreds of thousands and dealt with them individually. I will look through my old notes when I get a chance and will post again if I find any further clues.

[bthom73]

Thanks for the quick reply. I also noticed that the troublesome messages are within hour range of the new year, accounting for timezone offset. We're in the Eastern timezone (UTC-5 in December) and all of the troublesome messages are within 5 hours of midnight on Dec 31st. So, I'm assuming it's some sort of new year rollover issue.

Although our Zimbra server is running Ubuntu 8.04LTS (and that's where I was running imapsync from), as a test I just did a fresh install of Ubuntu 10.04.1LTS on a separate machine and tried to do the imapsync again from there. Same issue. So, if the issue is a bug in imapsync or a bug in PERL's Date::Manip (called by imapsync), it still seems to be broken.

If you run into any notes, please let me know.

Thanks again,
Brian

[oayfer]

Found some notes. It seems we thought this was a Solaris problem at the time, but you have proven this to be wrong. Perhaps the bug is in the Perl libraries. In any case, since this affected so few messages, we fixed this using the following addition (in Perl) to our migration scripts:

$d = $f_idate;
$debug and print "internal date from 1: [$d]\n";

# this is a horrible TERRIBLE workaround for dovecot/solaris9/mbox
# returning +4000 as the tz offset for dates near a year boundary
# like dec31 and jan1. don't do this. this really is terrible.
# it may make some internaldates wrong by 1hr or something.
#
if ($d =~ /\+4000$/) {
$d =~ s/\+4000$/\-0700/;
$debug and print "internal date from 1: [$d] FIXED solaris9 bug\n";
}

require Date::Manip;
Date::Manip->import(qw(ParseDate Date_Cmp UnixDate));

Hope this helps. Feel free to reach out with additional questions, I try to help the community as much as I can.