Page 6 of 12 FirstFirst ... 45678 ... LastLast
Results 51 to 60 of 115

Thread: ldap database went from 97meg to 86gig

  1. #51
    winepress is offline Member
    Join Date
    Nov 2012
    Location
    Seattle, WA, US
    Posts
    14
    Rep Power
    2

    Default

    Hi Quanah,

    Thanks for the pointers. I'm glad to know I'm not totally locked into an 80GB situation.

    Regards,
    Kevin

  2. #52
    winepress is offline Member
    Join Date
    Nov 2012
    Location
    Seattle, WA, US
    Posts
    14
    Rep Power
    2

    Default

    Greetings All,

    Using Quanah's suggestions, here are step-by-step instructions of what I did on my system, which worked quite well. Feel free to improve as necessary:

    This is for a 64MB datasize. Please change ldap_db_maxsize and ldap_accesslog_maxsize as necessary to suit your environment.

    As Zimbra User:
    Code:
    # update database size
    zmlocalconfig -e ldap_db_maxsize=67108864
    
    # update log size
    zmlocalconfig -e ldap_accesslog_maxsize=536870912
    
    # allow time for zmconfigd to apply new setting (as suggested by Quanah)
    sleep 90
    
    # stop slapd
    ldap stop
    
    # change to database directory
    cd /opt/zimbra/data/ldap/mdb/db
    
    # backup database
    /opt/zimbra/libexec/zmslapcat /opt/zimbra/data/ldap/mdb/db
    
    # move to .old
    mv data.mdb data.mdb.old
    
    # recreate database with new size and restore from backup
    /opt/zimbra/openldap/sbin/slapadd -q -b "" -F /opt/zimbra/data/ldap/config -l /opt/zimbra/data/ldap/mdb/db/ldap.bak
    
    # start slapd
    ldap start
    
    # archive original
    gzip data.mdb.old
    I hope this helps!

    Thanks,
    Kevin
    Last edited by winepress; 11-26-2012 at 04:26 PM. Reason: change to code formatting

  3. #53
    quanah is offline Zimbra Employee
    Join Date
    May 2007
    Location
    Zimbra
    Posts
    1,185
    Rep Power
    9

    Default

    My one note would be that after running the zmlocalconfig commands, you should add a sleep 90, to allow time for zmconfigd to update the OpenLDAP database configuration. Otherwise, the change may not have taken effect yet when you run ldap stop
    Quanah Gibson-Mount
    Server Architect
    Zimbra, Inc
    --------------------
    Zimbra :: the leader in open source messaging and collaboration

  4. #54
    quanah is offline Zimbra Employee
    Join Date
    May 2007
    Location
    Zimbra
    Posts
    1,185
    Rep Power
    9

    Default

    Actually one other note -- I wouldn't make the accesslog DB setting so low. Accesslog growth can be significant on a master.
    Last edited by quanah; 11-26-2012 at 05:26 PM.
    Quanah Gibson-Mount
    Server Architect
    Zimbra, Inc
    --------------------
    Zimbra :: the leader in open source messaging and collaboration

  5. #55
    winepress is offline Member
    Join Date
    Nov 2012
    Location
    Seattle, WA, US
    Posts
    14
    Rep Power
    2

    Default

    Thanks Quanah,

    I've edited the code snippet above to include the "sleep 90".

    I also changed the log file size to 512MB, although, as you said, that may need to be higher for systems with heavy traffic. Ours is quite low, so I'm not too concerned with having it at that level.

    Regards,
    Kevin

  6. #56
    quanah is offline Zimbra Employee
    Join Date
    May 2007
    Location
    Zimbra
    Posts
    1,185
    Rep Power
    9

    Default

    Excellent, thanks.

    It is on my to-do list to write a script to monitor database size vs maxsize, so that alerts can be integrated into the admin console. There will also be a tool named mdb_copy in future OpenLDAP releases that allows copying an MDB file directly (if slapd is stopped). That would be the correct solution instead of rsync for doing a backup of the /opt/zimbra/data/ldap/{mdb,accesslog}/db directories, and it also only copies the actual file size vs the allocated file size.
    Quanah Gibson-Mount
    Server Architect
    Zimbra, Inc
    --------------------
    Zimbra :: the leader in open source messaging and collaboration

  7. #57
    quanah is offline Zimbra Employee
    Join Date
    May 2007
    Location
    Zimbra
    Posts
    1,185
    Rep Power
    9

    Default

    A couple of follow up bits:

    cp does correctly copy the file as sparse -- The issue is that cp has to scan the entire 80GB file while copying. But the end result is correct:

    Code:
    zimbra@zre-ldap002:~/data/ldap/mdb/db$ ls -l
    -rw------- 1 zimbra zimbra 85899345920 Nov 26 17:02 data.mdb
    
    zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb
    2.7M    data.mdb
    
    zimbra@zre-ldap002:~/data/ldap/mdb/db$ time cp data.mdb /tmp
    
    real    4m40.298s
    user    0m8.520s
    sys     0m47.840s
    
    zimbra@zre-ldap002:~/data/ldap/mdb/db$ ls -l /tmp/data.mdb
    -rw------- 1 zimbra zimbra 85899345920 Nov 26 17:07 /tmp/data.mdb
    zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h /tmp/data.mdb
    2.7M    /tmp/data.mdb
    2.7M    total
    Using the -S flag to rsync also does the correct thing:
    Code:
    zimbra@zre-ldap002:/tmp/q2$ time /opt/zimbra/rsync/bin/rsync -S /opt/zimbra/data/ldap/mdb/db/data.mdb .
    ls -ls
    
    real    17m49.909s
    user    9m15.850s
    sys     3m32.470s
    
    zimbra@zre-ldap002:/tmp/q2$ du -c -h data.mdb
    2.7M    data.mdb
    2.7M    total
    So, if people want to use rsync or cp to back up their database, this is acceptable for ZCS 8 *as long as ldap is stopped*. It remains unacceptable for ZCS7 and prior, due to the use of BDB in those releases.

    --Quanah
    Quanah Gibson-Mount
    Server Architect
    Zimbra, Inc
    --------------------
    Zimbra :: the leader in open source messaging and collaboration

  8. #58
    quanah is offline Zimbra Employee
    Join Date
    May 2007
    Location
    Zimbra
    Posts
    1,185
    Rep Power
    9

    Default

    Quote Originally Posted by winepress View Post
    Greetings All,

    Using Quanah's suggestions, here are step-by-step instructions of what I did on my system, which worked quite well. Feel free to improve as necessary:

    Another note -- in ZCS 8.0.2 and later, when you resize the DB via localconfig, that is all you have to do. As soon as the modifictaion is made to slapd, the data.mdb file will automatically resize to whatever the new maxsize value is. No need to slapcat & slapadd to change the size of the file.
    Quanah Gibson-Mount
    Server Architect
    Zimbra, Inc
    --------------------
    Zimbra :: the leader in open source messaging and collaboration

  9. #59
    winepress is offline Member
    Join Date
    Nov 2012
    Location
    Seattle, WA, US
    Posts
    14
    Rep Power
    2

    Default

    Hi Quanah,

    Thanks for the updates on this. That's good info to know about cp -S and rsync -S. I think we now have some viable solutions listed here for FOSS community. I appreciate your help.

    Regards,
    Kevin

  10. #60
    mowglie is offline Intermediate Member
    Join Date
    Oct 2010
    Posts
    24
    Rep Power
    4

    Default

    not to beat a dead bush (or whatever saying is) but just so i am clear:

    running ubuntu 12.04, zimbra ose 8.01, data.mdb allocated size 80GB & actual size 5.7MB.

    if i issue 'services zimbra stop' (which stops ldap), then 'tar -pj -cvf backup.tar.bz2 data.mdb', will it take a lot less time than if i do not stop ldap first? i know this is not proper backup method. i know the resultant bz2 file will be same size whether i stop ldap or not. i just want to know if it will take less time to make the bz2 if i stop ldap first (it takes forever due to data.mdb if i dont).

    this is only for curiosity; i am not advocating any particular backup methods or anything else. pls do not tear my head off for asking.

Page 6 of 12 FirstFirst ... 45678 ... LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Extending LDAP database
    By frost983 in forum Developers
    Replies: 3
    Last Post: 11-02-2010, 06:46 AM
  2. [SOLVED] Zmsetservername choking on ldap database
    By Emmanuel Kasper in forum Administrators
    Replies: 7
    Last Post: 10-21-2009, 09:50 AM
  3. How do I browse Zimbra LDAP database?
    By williamn in forum Administrators
    Replies: 4
    Last Post: 04-16-2008, 01:47 AM
  4. Replies: 0
    Last Post: 01-14-2008, 11:41 AM
  5. change ldap database
    By Grejao in forum Administrators
    Replies: 1
    Last Post: 12-07-2007, 08:39 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •