Results 1 to 7 of 7

Thread: [SOLVED] Vorratsdatenspeicherung - longer log storage

  1. #1
    Offermann is offline Intermediate Member
    Join Date
    Jan 2008
    Location
    Berlin, Germany
    Posts
    24
    Rep Power
    7

    Default [SOLVED] Vorratsdatenspeicherung - longer log storage

    Hello

    since 2008-01-01 in Germany, according to "Vorratsdatenspeicherung", extensive information about mail traffic has to be stored by mail providers for 6 month. Then the information has to be deleted. How does Zimbra have to be configured to comply with Vorratsdatenspeicherung?

    Thanks, regards,
    Philipp

  2. #2
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    http://www.zimbra.com/forums/announc...r-profile.html, here's several solutions based on how much modification you want to do:

    Just so you're aware, the NE version has an excellent add-on that does envelope forking called Archiving & Discovery w/ cross mailbox search
    often referred to as ZAD for short - more on it:
    Zimbra Archiving and Discovery Whitepaper.pdf
    Zimbra Archiving and Discovery Release Notes.pdf
    Zimbra_Archiving_Discovery_Webinar_Dec_2007.pdf

    Compare ZCS Editions

    For FOSS, many use this method of archiving: Enhancement Hacks - power of filters & always_bcc
    In postfix it's essentially: always_bcc = catchallArchiveMonkey@domain.com

    Individually you can use 'sender_bcc_maps' and 'recipient_bcc_maps'
    (search Postfix.org)
    Here's an example: [SOLVED] Automatically CC an account? Is this possible?

    If you want to just get incoming mails, visit the account > forwarding tab > forwarding addresses hidden from the user (this would kinda be the same as doing recipient_bcc_maps).
    Last edited by mmorse; 01-04-2008 at 11:59 AM.

  3. #3
    Offermann is offline Intermediate Member
    Join Date
    Jan 2008
    Location
    Berlin, Germany
    Posts
    24
    Rep Power
    7

    Default Vorratsdatenspeicherung

    Hello mmorse,

    thank you for your answer. What I understood of your answer is that you propose to send a bcc of all incoming mails for archiving. That is not what is meant by German Vorratsdatenspeicherung. I even think that bccing would be illegal.
    I assume (though I'm not an expert on this matter) that the following information has to be stored for 6 month and then has to be deleted within 1 month:

    SMTP outbound:
    - Date, time and timezone
    - Sender e-mail
    - IP address of sender
    - All recipients e-mail

    SMTP inbound:
    - Date, time and timezone
    - Sender e-mail
    - Recipient e-mail
    - IP address of sender

    IMAP fetch:
    - Date, time and timezone
    - IMAP identification
    - IP address

    POP3 fetch:
    - Date, time and timezone
    - POP identification
    - IP address

    I think some of these informations are covered by current log files. Which information is missing? And how to configure logfile rotation to comply to the law?

    Thanks, regards, Philipp
    Last edited by Offermann; 01-13-2008 at 01:46 PM.

  4. #4
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Ah, so you need just the logs - unlike most US requirements which require you to store the entire email itself which is why ZAD was developed.

    Examine your /etc/logrotate.d/zimbra and increase the rotate #'s.

    Their set on daily - so you have one log/day.
    (Weekly or monthly would be other options - but that would make for some large logs, as well as making this more difficult to plan out/examine when needed.)

    Simply change rotate numbers to 180 (6 x 30 or so days in a month) and they should be deleted exactly after 6 months.

    Before doing any zimbra upgrades, you should make a backup of those logs to a safe place so you don't accidentally loose any, then re-correct the logrotate file as needed after upgrade.

    For other pruning values, as the zimbra user type: contab -e

    Relevant logs being /var/log/zimbra.log & contents of /opt/zimbra/log/

    There's also the logger db - to adjust storage:
    su - zimbra
    zmprov mcf zimbraLogRawLifetime xd
    (31d is default)

    If you're curious how this process works examine /opt/zimbra/bin/zmmsgtrace & /opt/zimbra/libexec/zmlogprocess (which is fired off every 10 min from crontab against the zimbra_logger DB.)

    (There's also zimbraLogSummaryLifetime which is set to 730d by default.)
    Last edited by mmorse; 12-01-2008 at 01:56 PM.

  5. #5
    Offermann is offline Intermediate Member
    Join Date
    Jan 2008
    Location
    Berlin, Germany
    Posts
    24
    Rep Power
    7

    Default Missing Information

    Hello mmorse,

    thanks for your quick answer. I tried to figure out where the information is stored. The only place where I found relevant information is /opt/zimbra/log/mailbox.log. I found all information necessary for IMAP access and POP access. What I didn't find is:
    - IP address of sender machine for SMTP inbound (it only stores my own IP address, which I know anyway)
    - SMTP outbound (couldn't find any information whatsoever in any log file)

    Do you have any hint where this information is stored and/or how to make Zimbra log this information?

    Additionally, the rotation for mailbox.log doesn't seem to be configured in /etc/logrotate.d/zimbra. It seems to have a similar configuration scheme to audit.log, which is not in logrotate neither. Any hint where the rotation for mailbox.log is configured?

    Thanks, regards,
    Philipp

    PS: Reading the law I discovered that there is a grace period till 1.1.2009. Still I would like to know if Zimbra can be configured to comply to legal regulations before that date.

  6. #6
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Quote Originally Posted by Offermann View Post
    Do you have any hint where this information is stored and/or how to make Zimbra log this information?
    Quote Originally Posted by mmorse View Post
    Relevant logs being /var/log/zimbra.log & contents of /opt/zimbra/log/
    There's also the zmmsgtrace command
    Please read: /docs/ne/latest/administration_guide/9_Monitoring.12.1.html
    Quote Originally Posted by Offermann View Post
    Additionally, the rotation for mailbox.log doesn't seem to be configured in /etc/logrotate.d/zimbra. It seems to have a similar configuration scheme to audit.log, which is not in logrotate neither. Any hint where the rotation for mailbox.log is configured?
    The log pruning deletes logs from that are over eight days old. The job runs at 2:30 a.m You can configure it via crontab (as the zimbra user type crontab -e).
    Contents of /opt/zimbra/log/:
    # ZIMBRASTART -- DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRAEND
    #
    # Log pruning
    #
    30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
    Jetty (aka mailboxd) access logs:
    # Log pruning
    #
    30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/n$
    /var/log/zimbra.log:
    Configured via /opt/zimbra/conf/zmlogrotate or/and /etc/logrotate.d/zimbra

  7. #7
    Offermann is offline Intermediate Member
    Join Date
    Jan 2008
    Location
    Berlin, Germany
    Posts
    24
    Rep Power
    7

    Default How I configured it

    I think I finally got it. I've documented how I configured Zimbra as a reference for other: I'm open for comments.

    Configuration:

    Two files are needed: /var/log/zimbra.log and /opt/zimbra/log/mailbox.log. The log rotation for these two files has to be set between 6 and 7 month.
    For /var/log/zimbra.log:
    In /etc/logrotate.d/zimbra in the block for /var/log/zimbra.log, add a line "rotate 190" and comment out the line "notifempty".

    For /opt/zimbra/log/mailbox.log:
    mkdir /opt/zimbra/log/mailbox
    In /opt/zimbra/conf/log4j.properties.in change line starting log4j.appender.LOGFILE.File to "log4j.appender.LOGFILE.File=/opt/zimbra/log/mailbox/mailbox.log". Restart Zimbra.
    As user zimbra, run "crontab -e". Add line "30 2 * * * find /opt/zimbra/log/mailbox/ -type f -name \*.log\* -mtime +190 -exec rm {} \; > /dev/null 2>&1".


    Where to find the information:
    IMAP and POP access is in /opt/zimbra/log/mailbox/mailbox.log.
    IMAP access looks like
    Code:
    2008-01-16 07:26:20,166 INFO  [ImapServer-1] [name=me@here.com;ip=xxx.xxx.xxx.xxx;] imap - user me@here.com authenticated,
    mechanism=login
    POP access looks accordingly.

    SMTP is in /var/log/zimbra.log.
    SMTP outbound is a bit more complicated:
    Code:
    Jan 16 21:15:43 here postfix/smtpd[20392]: 8068B1BD0105: client=domainname[xxx.xxx.xxx.xxx], sasl_method=PLAIN, sasl_use
    rname=me@here.com
    Jan 16 21:15:43 here postfix/cleanup[20395]: 8068B1BD0105: message-id=<200801162115.37576.me@here.com>
    Jan 16 21:15:43 here postfix/qmgr[383]: 8068B1BD0105: from=<me@here.com>, size=2966, nrcpt=1 (queue active)
    Jan 16 21:15:43 here amavis[12460]: (12460-04) ESMTP::10024 /opt/zimbra/amavisd/tmp/amavis-20080116T210132-12460: <me@here.com
    > -> <you@here.com> SIZE=2966 Received: from here.com ([127.0.0.1]) by localhost (here.com [127.0.0.1]) (amavisd-new, port
     10024) with ESMTP for <you@there.com>; Wed, 16 Jan 2008 21:15:43 +0100 (CET)
    Finally, SMTP inbound:
    Code:
    Jan 16 13:45:05 here postfix/smtpd[1446]: B8C0A1BD0105: client=domainname[xxx.xxx.xxx.xxx]
    Jan 16 13:45:05 here postfix/cleanup[1449]: B8C0A1BD0105: message-id=<111630.71635.qm@domainname>
    Jan 16 13:45:05 here postfix/qmgr[383]: B8C0A1BD0105: from=<you@there.com>, size=2806, nrcpt=1 (queue active)
    Jan 16 13:45:05 here amavis[32670]: (32670-11) ESMTP::10024 /opt/zimbra/amavisd/tmp/amavis-20080116T043730-32670: <you@there.com
    > -> <me@here.com> SIZE=2806 BODY=8BITMIME Received: from here.com ([127.0.0.1]) by localhost (here.com [127.0.0.1]) (ama
    visd-new, port 10024) with ESMTP for <me@here.com>; Wed, 16 Jan 2008 13:45:05 +0100 (CET)

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Errors installing Outlook Connector
    By Tim G in forum Zimbra Connector for Outlook
    Replies: 57
    Last Post: 05-05-2011, 02:27 PM
  2. Trying to understand audit log
    By rfoster in forum Administrators
    Replies: 2
    Last Post: 02-13-2008, 02:55 PM
  3. Connector Fails
    By ILLCOMM in forum Zimbra Connector for Outlook
    Replies: 4
    Last Post: 09-28-2007, 12:08 PM
  4. Error Installing Outlook Connector
    By DanO in forum Zimbra Connector for Outlook
    Replies: 17
    Last Post: 08-28-2007, 09:35 AM
  5. Replies: 2
    Last Post: 10-02-2006, 08:44 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •