Discussion:
Repairing imported mbox files (was Re: Imported mbox files have botched Received date in WebUser)
Charles Koleczek
2006-02-21 20:02:17 UTC
Permalink
Gavin, thanks for the reply. It is too late for an IMAP import of my
old mail store. We're cutover and now I'm faced with a cleanup of the
old mail.

CommuniGate's documentation about copying mbox files states that I am
going to lose the state, but that these legacy mail files will be
swallowed...
http://www.stalker.com/CommuniGatePro/Migration.html#Mailboxes

What CommuniGate failed to do was create an accurate "From<>(S......."
message delimiter.
All my message delimiters have an "empty" date...
From <>(S_______-000000000029) Sun Jan 01 00:00:00 1900
newly received messages, post import look good...
From <>(________-000000007659) Tue Feb 21 19:41:35 2006

I am resigned to living with lost message state (read, replied,
forwarded, etc). That I can sell my users on.
However losing that date confuses CGP WebUser and all versions of
Outlook into thinking the date is 12/31/1899.

Scrambling that received date in Outlook and WebUser is unacceptable.
Searches for over the CGP mailing list don't find any previous mentions
of this situation. But I'm sure it has happened often enough that
somebody has posted a perl script to parse these new mbox folders and
correct that delimiter. That is what I need to do and I'm hoping not to
have to do it all from scratch.

Regards,

Chuck Koleczek
I'm in the midst of a migration from a UW IMAPD.
I am copying over the mbox format files into users' CGP directories,
renaming them with *.mbox. So far the import is fine as far as
Thunderbird IMAP clients are concerned. All the mail looks fine and in
order.
HOWEVER, via the WebUser interface all the Received dates are the same.
I've tried cleaning that up by copying all the messages from one IMAP
folder to another with Thunderbird. No luck, all the Received dates are
still stuck on the same date of two weeks ago (which seems to have no
special significance).
Searches on this haven't turned up any like situations...but this can't
be that uncommon?!
The problem is probably due to you copying the mbox files rather than
"moving" them. You get similar problems if you migrate mail from one
server to another simply by moving messages within a mail client - the
messages on arrival have their date set to the date they were moved
rather than the date they were created.
There are some threads on this topic from way back - you might want to
search the archive etc. There is also a CGP utility called IMAPMove
that works very well - migrates messages between two imap servers and
preserves date information. Find more about it
here: http://www.stalker.com/CommuniGatePro/Migration.html#IMAPMove
Hope this helps.
Best regards
Gavin Lawrie
#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>
Stefan Seiz
2006-02-21 20:58:18 UTC
Permalink
Post by Charles Koleczek
All my message delimiters have an "empty" date...
From <>(S_______-000000000029) Sun Jan 01 00:00:00 1900
Wouldn't a simple search and replace do?

I mean search for "Sun Jan 01 00:00:00 1900" and replace it with an actual
date like for instance the date of your start with CGP.

If that'll do, i'd shut down CGP, do the search and replace on all the .mbox
files and then restart CGP. This might be needed as CGP does some caching
and the changes might be lost if CGP flushes it's cache after you did the
search and replace...

--
Stefan Seiz <http://www.stefanseiz.com>
Spamto: <***@imd.net>




#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>
Charles Koleczek
2006-02-22 17:53:40 UTC
Permalink
I got a call from CGP support in California just now. I appreciate
that. But we didn't map out a solution to the problem. I am hoping the
guys in Russia will help me out while I have a window to put in a fix
tonight!

I am going to modify all my .mbox files that have the "Sun Jan 01
00:00:00 1900" date.
If I use some of the Perl mbox utilities out there I think I can,
relatively easily:
1)parse out individual messages
2)look for a problem date in the delimiter "From <>..." line
3)modify that line with the date from the Date header in the email.
4)Write a the rest of the "From <>..." delimiter line in a way that
makes sense to CGP.
5)Put this back into a .mbox that will show the correct date in
WebUser and Outlook.

Question on the meaning of the delimiter line...
Post by Charles Koleczek
From <>(________-000000007659) Tue Feb 21 19:41:35 2006
Inside the parentheses we have some state bits before the hyphen, then
a simple sequence after the hyphen.
If there some documentation on this header information?
Can I just start with (________-000000000001) then
(________-000000000002) in each mbox?

-Chuck Koleczek
Post by Charles Koleczek
All my message delimiters have an "empty" date...
From <>(S_______-000000000029) Sun Jan 01 00:00:00 1900
Wouldn't a simple search and replace do?
I mean search for "Sun Jan 01 00:00:00 1900" and replace it with an
actual
Post by Charles Koleczek
date like for instance the date of your start with CGP.
If that'll do, i'd shut down CGP, do the search and replace on all the
.mbox
files and then restart CGP. This might be needed as CGP does some
caching
Post by Charles Koleczek
and the changes might be lost if CGP flushes it's cache after you
did the
Post by Charles Koleczek
search and replace...
--
Stefan Seiz <http://www.stefanseiz.com>
Gavin, thanks for the reply. It is too late for an IMAP import of my
old mail store. We're cutover and now I'm faced with a cleanup of the
old mail.
CommuniGate's documentation about copying mbox files states that I am
going to lose the state, but that these legacy mail files will be
swallowed...
http://www.stalker.com/CommuniGatePro/Migration.html#Mailboxes
What CommuniGate failed to do was create an accurate "From<>(S......."
message delimiter.
All my message delimiters have an "empty" date...
From <>(S_______-000000000029) Sun Jan 01 00:00:00 1900
newly received messages, post import look good...
From <>(________-000000007659) Tue Feb 21 19:41:35 2006
I am resigned to living with lost message state (read, replied,
forwarded, etc). That I can sell my users on.
However losing that date confuses CGP WebUser and all versions of
Outlook into thinking the date is 12/31/1899.
Scrambling that received date in Outlook and WebUser is unacceptable.
Searches for over the CGP mailing list don't find any previous mentions
of this situation. But I'm sure it has happened often enough that
somebody has posted a perl script to parse these new mbox folders and
correct that delimiter. That is what I need to do and I'm hoping not to
have to do it all from scratch.
Regards,
Chuck Koleczek
I'm in the midst of a migration from a UW IMAPD.
I am copying over the mbox format files into users' CGP
directories,
Post by Charles Koleczek
renaming them with *.mbox. So far the import is fine as far as
Thunderbird IMAP clients are concerned. All the mail looks fine
and in
Post by Charles Koleczek
order.
HOWEVER, via the WebUser interface all the Received dates are
the same.
Post by Charles Koleczek
I've tried cleaning that up by copying all the messages from
one IMAP
Post by Charles Koleczek
folder to another with Thunderbird. No luck, all the Received
dates are
Post by Charles Koleczek
still stuck on the same date of two weeks ago (which seems to
have no
Post by Charles Koleczek
special significance).
Searches on this haven't turned up any like situations...but
this can't
Post by Charles Koleczek
be that uncommon?!
The problem is probably due to you copying the mbox files rather than
"moving" them. You get similar problems if you migrate mail from one
server to another simply by moving messages within a mail client
- the
Post by Charles Koleczek
messages on arrival have their date set to the date they were moved
rather than the date they were created.
There are some threads on this topic from way back - you might
want to
Post by Charles Koleczek
search the archive etc. There is also a CGP utility called IMAPMove
that works very well - migrates messages between two imap servers and
preserves date information. Find more about it
here: http://www.stalker.com/CommuniGatePro/Migration.html#IMAPMove
Hope this helps.
Best regards
Gavin Lawrie
#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>
Technical Support
2006-02-22 18:00:25 UTC
Permalink
Hello,
Post by Charles Koleczek
I got a call from CGP support in California just now. I appreciate
that. But we didn't map out a solution to the problem. I am hoping the
guys in Russia will help me out while I have a window to put in a fix
tonight!
I am going to modify all my .mbox files that have the "Sun Jan 01
00:00:00 1900" date.
If I use some of the Perl mbox utilities out there I think I can,
1)parse out individual messages
2)look for a problem date in the delimiter "From <>..." line
3)modify that line with the date from the Date header in the email.
4)Write a the rest of the "From <>..." delimiter line in a way that
makes sense to CGP.
5)Put this back into a .mbox that will show the correct date in
WebUser and Outlook.
Question on the meaning of the delimiter line...
Post by Charles Koleczek
From <>(________-000000007659) Tue Feb 21 19:41:35 2006
Inside the parentheses we have some state bits before the hyphen, then
a simple sequence after the hyphen.
If there some documentation on this header information?
Can I just start with (________-000000000001) then
(________-000000000002) in each mbox?
Yes. The only requirement is that these numbers increase within a
mailbox file (not necessary sequential, but always ascending.)

Underscores are placeholders for message flags. Leave them as they are -
that would mean no flags set.
Post by Charles Koleczek
-Chuck Koleczek
Post by Charles Koleczek
All my message delimiters have an "empty" date...
From <>(S_______-000000000029) Sun Jan 01 00:00:00 1900
Wouldn't a simple search and replace do?
I mean search for "Sun Jan 01 00:00:00 1900" and replace it with an
actual
Post by Charles Koleczek
date like for instance the date of your start with CGP.
If that'll do, i'd shut down CGP, do the search and replace on all the
.mbox
files and then restart CGP. This might be needed as CGP does some
caching
Post by Charles Koleczek
and the changes might be lost if CGP flushes it's cache after you
did the
Post by Charles Koleczek
search and replace...
--
Stefan Seiz <http://www.stefanseiz.com>
Gavin, thanks for the reply. It is too late for an IMAP import of my
old mail store. We're cutover and now I'm faced with a cleanup of the
old mail.
CommuniGate's documentation about copying mbox files states that I am
going to lose the state, but that these legacy mail files will be
swallowed...
http://www.stalker.com/CommuniGatePro/Migration.html#Mailboxes
What CommuniGate failed to do was create an accurate "From<>(S......."
message delimiter.
All my message delimiters have an "empty" date...
From <>(S_______-000000000029) Sun Jan 01 00:00:00 1900
newly received messages, post import look good...
From <>(________-000000007659) Tue Feb 21 19:41:35 2006
I am resigned to living with lost message state (read, replied,
forwarded, etc). That I can sell my users on.
However losing that date confuses CGP WebUser and all versions of
Outlook into thinking the date is 12/31/1899.
Scrambling that received date in Outlook and WebUser is unacceptable.
Searches for over the CGP mailing list don't find any previous mentions
of this situation. But I'm sure it has happened often enough that
somebody has posted a perl script to parse these new mbox folders and
correct that delimiter. That is what I need to do and I'm hoping not to
have to do it all from scratch.
Regards,
Chuck Koleczek
I'm in the midst of a migration from a UW IMAPD.
I am copying over the mbox format files into users' CGP
directories,
Post by Charles Koleczek
renaming them with *.mbox. So far the import is fine as far as
Thunderbird IMAP clients are concerned. All the mail looks fine
and in
Post by Charles Koleczek
order.
HOWEVER, via the WebUser interface all the Received dates are
the same.
Post by Charles Koleczek
I've tried cleaning that up by copying all the messages from
one IMAP
Post by Charles Koleczek
folder to another with Thunderbird. No luck, all the Received
dates are
Post by Charles Koleczek
still stuck on the same date of two weeks ago (which seems to
have no
Post by Charles Koleczek
special significance).
Searches on this haven't turned up any like situations...but
this can't
Post by Charles Koleczek
be that uncommon?!
The problem is probably due to you copying the mbox files rather than
"moving" them. You get similar problems if you migrate mail from one
server to another simply by moving messages within a mail client
- the
Post by Charles Koleczek
messages on arrival have their date set to the date they were moved
rather than the date they were created.
There are some threads on this topic from way back - you might
want to
Post by Charles Koleczek
search the archive etc. There is also a CGP utility called IMAPMove
that works very well - migrates messages between two imap servers and
preserves date information. Find more about it
here: http://www.stalker.com/CommuniGatePro/Migration.html#IMAPMove
Hope this helps.
Best regards
Gavin Lawrie
--
Best regards,
Dmitry Akindinov

=======================================================================
When answering to letters sent to you by the tech.support staff, make
sure the original message you have received is included into your reply.

#############################################################
This message is sent to you because you are subscribed to
the mailing list <***@mail.stalker.com>.
To unsubscribe, E-mail to: <CGatePro-***@mail.stalker.com>
To switch to the DIGEST mode, E-mail to <CGatePro-***@mail.stalker.com>
To switch to the INDEX mode, E-mail to <CGatePro-***@mail.stalker.com>
Send administrative queries to <CGatePro-***@mail.stalker.com>
Loading...