Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: LDAP MDB back end file huge

  1. #1
    halfgaar's Avatar
    halfgaar is offline Active Member
    Join Date
    Dec 2009
    Location
    Netherlands
    Posts
    47
    Rep Power
    5

    Default LDAP MDB back end file huge

    Hi,

    I just upgraded to zcs-8.0.5_GA_5839.UBUNTU10_64.20130910123904 (from 8.0). Ever since the upgrade, the file /opt/zimbra/data/ldap/mdb/db/data.mdb is a 50 GB sparse file, taking 800 kB of space. My backup tool, rdiff-backup, doesn't understand sparse files, and thus backs it up as a 50 GB normal file, making my increments huge and unmanagable.

    What can I do about this?

    I do make slapcat backups daily, so could I just exclude /opt/zimbra/data/ldap/mdb*? My slapcat backup script is:

    Code:
    #!/bin/bash
    
    if [ "`/usr/bin/whoami`" != "zimbra" ]; then
      echo "Not user zimbra, exiting"
      exit 1
    fi
    
    backup_dir="/var/backups/ldap"
    
    /opt/zimbra/libexec/zmslapcat $backup_dir
    /usr/bin/find $backup_dir -type f -mtime +10 -delete
    But I never had to re-load this data. What would happen my server crashes, I restore the backup and use "/opt/zimbra/libexec/zmslapadd < backupfile" to re-load the ldap data? Will that suffice?

    edit: I expanded the script above to also use the mdb_copy command. So, the following script makes slapcat and mdb_copy backups:

    Code:
    #!/bin/bash
    #
    # Author: Wiebe Cazemier <wiebe@halfgaar.net>
    #
    # Since Zimbra 8.0 or 8.0.1 or thereabouts, the LDAP data backend is a 50-80 GB
    # sparse file. This needs to be excluded from the main backup (rdiff-backup in
    # my case), and the LDAP data needs to be backed up another way, which is what
    # this script does.
    #
    # This script backs up LDAP with an ldiff dump (slapcat) and mdb_copy. The
    # ldiff dump should be enough, but better to be safe than sorry.
    #
    # mdb_copy is included since Zimbra 8.0.2.
    
    backup_dir="/var/backups/ldap"
    mailto="admin@example.com"
    
    #############
    
    slapcat="/opt/zimbra/libexec/zmslapcat"
    mdb_copy="/opt/zimbra/openldap/bin/mdb_copy"
    mdb_src="/opt/zimbra/data/ldap/mdb/db"
    
    echo_err()
    {
      echo "$1" 1>&2
      echo "$1" | mail -s "Error in LDAP backup" "$mailto"
    }
    
    if [ "`/usr/bin/whoami`" != "zimbra" ]; then
      echo_err "Not user zimbra, exiting"
      exit 1
    fi
    
    if [ ! -d "$backup_dir" ]; then
      echo_err "$backup_dir doesn't exist"
      exit 1
    fi
    
    if [ ! -w "$backup_dir" ]; then
      echo_err "$backup_dir is not writable."
      exit 1
    fi
    
    if [ ! -x "$slapcat" ]; then
      echo_err "$slapcat not found."
      exit 1
    fi
    
    if [ ! -x "$mdb_copy" ]; then
      echo_err "$mdb_copy not found"
      exit 1
    fi 
    
    $slapcat "$backup_dir"
    
    if [ $? -ne 0 ]; then
      echo_err "slapcat failed"
      exit 1
    fi
    
    mdb_backup_file="$backup_dir/data.mdb"
    datestamp="$(date +%Y.%m.%d-%H.%M.%S)"
    mdb_backup_file_target="${mdb_backup_file}_$datestamp"
    
    $mdb_copy "$mdb_src" "$backup_dir"
    
    if [ $? -ne 0 ] || [ ! -f "$mdb_backup_file" ]; then
      echo_err "mdb_copy failed"
      exit 1
    fi
    
    mv "$mdb_backup_file" "$mdb_backup_file_target"
    
    /usr/bin/find "$backup_dir" -type f -mtime +10 -delete
    Last edited by halfgaar; 12-01-2013 at 04:14 AM. Reason: I pasted in the script that is now my backup of LDAP

  2. #2
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,566
    Rep Power
    57

    Default

    Have you looked at the other thread(s) on this topic and the description on how to backup this file correctly?
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  3. #3
    halfgaar's Avatar
    halfgaar is offline Active Member
    Join Date
    Dec 2009
    Location
    Netherlands
    Posts
    47
    Rep Power
    5

    Default

    Well, I searched for "MDB", but that yielded 0 results... I guess I will search differently

  4. #4
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,566
    Rep Power
    57

  5. #5
    halfgaar's Avatar
    halfgaar is offline Active Member
    Join Date
    Dec 2009
    Location
    Netherlands
    Posts
    47
    Rep Power
    5

    Default

    Apparently, the slapcat method works, but I can't really find much info about it in the wiki, as there apparently should be.

    I'm just excluding /opt/zimbra/data/ldap/mdb* and hope for the best, I guess.

  6. #6
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,566
    Rep Power
    57

    Default

    Quote Originally Posted by halfgaar View Post
    Apparently, the slapcat method works, but I can't really find much info about it in the wiki, as there apparently should be.
    The thread link that you've posted gives you full details of how to backup (or reduce the size of) the mdb file, did you read all of the thread and the wiki article?

    Quote Originally Posted by halfgaar View Post
    I'm just excluding /opt/zimbra/data/ldap/mdb* and hope for the best, I guess.
    That's not a good idea and there's no such thing as 'hope for the best' when you've a file to backup - you need to take a valid backup of that file and not leave it to chance.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  7. #7
    halfgaar's Avatar
    halfgaar is offline Active Member
    Join Date
    Dec 2009
    Location
    Netherlands
    Posts
    47
    Rep Power
    5

    Default

    The thread link that you've posted gives you full details of how to backup (or reduce the size of) the mdb file, did you read all of the thread and the wiki article?
    I read about mdb_copy, but I can't really find wiki pages about using slapcat in a backup procedure. If it's really true that slapcat suffices, then I don't really see the need to make a mdb_copy script as well, especially because I have to manually code what rdiff-backup's increment and clean-up does for me.

    Plus, tools that have a dump mechanism are usually better backed up using that, than copying the back-end files. I also have a MySQL dump running nightly, for example.

    That's not a good idea and there's no such thing as 'hope for the best' when you've a file to backup - you need to take a valid backup of that file and not leave it to chance.
    'Hope for the best' was kind of a wrong choice of words.

    Normally, a slapcat is all the backup you need for LDAP, and I just figured it should do here as well, also because others are saying it.

    But, to be safe, I guess I have to backup that mdb file too... Too bad rdiff-backup can't handle spare files. It wouldn't have been a problem if it did.

  8. #8
    ccelis5215 is offline Elite Member
    Join Date
    Jun 2011
    Location
    Caracas Venezuela
    Posts
    470
    Rep Power
    4

    Default

    Quote Originally Posted by halfgaar View Post
    I read about mdb_copy, but I can't really find wiki pages about using slapcat in a backup procedure. If it's really true that slapcat suffices, then I don't really see the need to make a mdb_copy script as well, especially because I have to manually code what rdiff-backup's increment and clean-up does for me.

    Plus, tools that have a dump mechanism are usually better backed up using that, than copying the back-end files. I also have a MySQL dump running nightly, for example.



    'Hope for the best' was kind of a wrong choice of words.

    Normally, a slapcat is all the backup you need for LDAP, and I just figured it should do here as well, also because others are saying it.

    But, to be safe, I guess I have to backup that mdb file too... Too bad rdiff-backup can't handle spare files. It wouldn't have been a problem if it did.

    guess you tested you backup procedure...

    as always, if you mind, let us know about your advances.

    ccelis.

  9. #9
    halfgaar's Avatar
    halfgaar is offline Active Member
    Join Date
    Dec 2009
    Location
    Netherlands
    Posts
    47
    Rep Power
    5

    Default

    guess you tested you backup procedure...
    Haha, you're right, I should. A full test is tricky, though. For it to be a full test, I'd have to restore it with the same network settings in the same subnet, to see if everything works. That will conflict with the existing server.

    I do two backups BTW: one with rdiff-backup to another disk, and one with the RestAPI account export feature, also to another disk. I'm pretty confident that I'll be able to restore from my backups.

    as always, if you mind, let us know about your advances.
    I will. I will probably update the ldap script above. I guess I should also put it on the wiki somewhere, but I wouldn't really know where. It's already a jungle of different backup procedures.

  10. #10
    halfgaar's Avatar
    halfgaar is offline Active Member
    Join Date
    Dec 2009
    Location
    Netherlands
    Posts
    47
    Rep Power
    5

    Default

    Hmm, mdb_copy doesn't copy the file as sparse. Restoring it to a new server will then fail, because when there is no more space in the mdb data file, you have dataloss.

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 3
    Last Post: 10-19-2011, 01:21 PM
  2. Log File Sizes = HUGE
    By samgreco in forum Administrators
    Replies: 12
    Last Post: 06-01-2010, 03:29 PM
  3. ZCB Log File Directory - HUGE
    By derek.rose in forum Zimbra Connector for BlackBerry
    Replies: 0
    Last Post: 04-21-2010, 04:56 AM
  4. Importing tgz file back into Zimbra
    By mrballz in forum General Questions
    Replies: 8
    Last Post: 02-11-2009, 10:37 AM
  5. zmmtaconfig.log file huge
    By msf004 in forum Administrators
    Replies: 1
    Last Post: 06-25-2008, 01:09 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •