Page 5 of 12 FirstFirst ... 34567 ... LastLast
Results 41 to 50 of 115

Thread: ldap database went from 97meg to 86gig

  1. #41
    samgann is offline Intermediate Member
    Join Date
    Feb 2012
    Posts
    17
    Rep Power
    3

    Default

    ok I guess I understand now.
    The 86 gig is allocated. Why does the data base need that much if the database is less that 200 meg? I now you said that rsync is not a great way to backup the database. Why does a rsync backup that file as a 86 gig file. Because the space is 86 gigs allocated ? I need to find another way to backup my servers. I have a main server and a backup server. Can you suggest another way? I think this whole thread is about this issue of backing up and not understanding the allocation of disk space from the 8.01 upgrade. The upgrade took alot of people off gaurd when they tried to backup there servers.

    Thanks for your support
    looking forward to a reply on a new way to backup the servers
    Sam
    Last edited by samgann; 11-23-2012 at 10:20 PM. Reason: spelling

  2. #42
    quanah is offline Zimbra Employee
    Join Date
    May 2007
    Location
    Zimbra
    Posts
    1,285
    Rep Power
    10

    Default

    Quanah Gibson-Mount
    Server Architect
    Zimbra, Inc
    --------------------
    Zimbra :: the leader in open source messaging and collaboration

  3. #43
    kevindods is offline Advanced Member
    Join Date
    Oct 2005
    Posts
    181
    Rep Power
    9

    Default

    I think if you use virtual instances of Zimbra then taking a snapshot of the server before updating anything is a nice simple way of doing it.

    Working on server metal directly these things are not so simple. Generally the process was to rsync the /opt/zimbra directory once, then stop Zimbra services do another rsync with LDAP and all other services stopped to produce consistent files that updated the initial rsync.

    For many FOSS users this approach works well especially in smaller businesses and is relatively simple to script and cron job and is very reliable

    I think that Quanah has explained clearly and helpfully how to backup the LDAP database in a reliable way, thanks Quanah - that is good and useful.

    What none of us seem to quite understand why the DB has 80GB provisioned when most DBs are a few hundred MB at most. If the DB was closer to expected maximums the issues we experienced would not arise at all. Perhaps if the focus is on server instances in virtualized cloud environments with cloud storage, block devices and simple snapshots then these concerns many smaller users experience do not become apparent.

    RSYNC is a perfectly reasonable way to backup a system that has no open DBs or files and that is how it has been used. Things change and that is often good, so long as the change continues to help. The idea of dealing better with the LDAP DB is a great idea, but why provision 80GB for all DBs and adversely affect other supporting processes that work in a wider application when there is no need.

    That last point is the one I am trying to understand, is there a clear need to provision 80GB of DB space for something that is usually no where near that size. Is it not better to fit the purpose as needed and provision more appropriate sizes for the DB and avoid the hassle. I can't answer that which is why I wanted Quanah to explain the drive to provision the 80GB not say 1GB or 500MB. Can you explain that Quanah?

    The file data.mdb reports its size as 80GB and that is not the actual used space, but it means to all FS based utilities it will be seen as 80GB. There is no checking for available space so that capacity may not be available even at creation and that will cause problems if it ever did get to 80GB or whenever the space available is consumed.

    Can you explain Zimbra thinking here? I really just want to get this understood and move on but the logic is not yet working through to my headspace ;-)

    EDIT
    Quanah - quick thought we may be at crossed purpose - if you, yourself, stop all ldap services and cp the data.mdb file are you actually saying you think it will only copy say, 7MB data and not copy 80GB? From my tests it really does spend ages and copy 80GB of nothing and create a genuine 80GB file.
    Last edited by kevindods; 11-24-2012 at 08:18 AM. Reason: after though!

  4. #44
    moren is offline Trained Alumni
    Join Date
    Jun 2007
    Location
    Halmstad, Sweden
    Posts
    58
    Rep Power
    8

    Default

    The correct method of backing up is explained in https://bugzilla.zimbra.com/show_bug.cgi?id=78781#c4.

    Could this rsync option help to keep the size of the file not to be expanded ?

    -S, --sparse
    Try to handle sparse files efficiently so they take up less
    space on the destination. Conflicts with --inplace because it’s
    not possible to overwrite data in a sparse fashion.
    Last edited by moren; 11-24-2012 at 10:50 AM.
    Regards
    Morén
    8.0.5_P1 / NE / RHEL6_64

  5. #45
    kevindods is offline Advanced Member
    Join Date
    Oct 2005
    Posts
    181
    Rep Power
    9

    Default

    I wonder if this is just a sparse file? normally a util like cp will detect that and work with it but this one seems a little different since cp sees it as a normal 80GB file.

  6. #46
    winepress is offline Member
    Join Date
    Nov 2012
    Location
    Seattle, WA, US
    Posts
    14
    Rep Power
    2

    Default

    Dear Zimbra,

    I think you're completely missing the point. Why on earth is a 5.5MB / 3.3MB / whatever size file being reported as 80GB? This thread feels like some sort of police interrogation where we're being made to believe that it's okay to cover up an apparent error, and if we're made to feel stupid enough, we'll believe it. This is not a professional or logical way to approach a problem.

    Look, if the file size is 3.3MB, then it should be reported to the OS as the same. That's just common sense. Otherwise, if it really is an 80GB file, then just say so and make sure it's consistent among all other file utilities. 3.3MB does not equal 80GB, and our bosses and customers will not accept explanations like the one you're giving.

    Secondly, best practice doesn't always equal real world. For instance, we have a mixed environment of FOSS and Commercial software. Our backup software is in fact commercial and will back up an entire linux box. We use this software to backup all of our machines in a best-practice method (disk-to-disk-to-tape-to-offsite-to-archive). However, our software doesn't have native Zimbra support, so it sees 80GB, not 3.3MB.

    Our request here is that you work with OpenLDAP to come up with a viable solution. Again, consistency is key. For my business, this is a critical problem, and if I must have a static file size, I need some way to adjust it down to something more reasonable. My database is 4 years old and only 3.3MB. So to have it inflated to 80GB is unreasonable.

    Regards,
    Kevin

  7. #47
    phoenix is offline Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,581
    Rep Power
    57

    Default

    Quote Originally Posted by winepress View Post
    Our request here is that you work with OpenLDAP to come up with a viable solution.
    I think you'll find that Zimbra is 'in touch' with OpenLDAP, check out the names of the Contributors to the OpenLDAP project. The solution you've been given for backing-up the mdb is a workable solution, it just needs incorporating into your current backup procedure for Zimbra.

    Quote Originally Posted by winepress View Post
    So to have it inflated to 80GB is unreasonable.
    Your database is not inflated to 80GB, as has been mentioned before that is allocated space not used space.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  8. #48
    winepress is offline Member
    Join Date
    Nov 2012
    Location
    Seattle, WA, US
    Posts
    14
    Rep Power
    2

    Default

    Greetings Bill,

    Quote Originally Posted by phoenix View Post
    I think you'll find that Zimbra is 'in touch' with OpenLDAP, check out the names of the Contributors to the OpenLDAP project. The solution you've been given for backing-up the mdb is a workable solution, it just needs incorporating into your current backup procedure for Zimbra.
    Agreed, it's a workable solution, but a breaking one for FOSS users. It should have been introduced in 8.0.0, the major release, not 8.0.1, a very minor release meant for bug fixes only. Revisioning systems like this were put in place so sysadmins can decide when to implement updates vs. upgrades.

    Quote Originally Posted by phoenix View Post
    Your database is not inflated to 80GB, as has been mentioned before that is allocated space not used space.
    With all due respect, you're not saying anything of substance here. If I own 5 acres of land, all of that land belongs to me, even though my house may only occupy 3000 sq ft. The rest of the world has no rights to my 5 acres, not just my house. So here, in real-world scenarios, I have to treat "allocated space" as used space. If I don't, I'd be extremely irresponsible and could bring harm to the data I'm supposed to protect.

    This wouldn't be a problem if the default size wasn't so huge. And it appears to be one-way only. I'd be satisfied if I could change my "allocated space" down to 64 MB, but I couldn't find any way to do that. Do you know of a way?

    Regards,
    Kevin
    Last edited by winepress; 11-24-2012 at 02:38 PM. Reason: spelling correction

  9. #49
    quanah is offline Zimbra Employee
    Join Date
    May 2007
    Location
    Zimbra
    Posts
    1,285
    Rep Power
    10

    Default

    Quote Originally Posted by winepress View Post
    This wouldn't be a problem if the default size wasn't so huge. And it appears to be one-way only. I'd be satisfied if I could change my "allocated space" down to 64 MB, but I couldn't find any way to do that. Do you know of a way?

    Regards,
    Kevin
    It is trivial.

    Do the following as the zimbra user:

    Change the LC key for the max size of the database. Let zmconfigd push that to the OpenLDAP config DB (just watch zimbra.log).
    After that is done, stop ldap, zmslapcat your DB, move /opt/zimbra/data/ldap/mdb aside, recreate /opt/zimbra/data/ldap/mdb/db
    Re-run slapadd to reload your DB. It will now only show an allocation of MAXSIZE
    Make sure maxsize is large enough to hold your DB + plenty of room for growth.
    Quanah Gibson-Mount
    Server Architect
    Zimbra, Inc
    --------------------
    Zimbra :: the leader in open source messaging and collaboration

  10. #50
    quanah is offline Zimbra Employee
    Join Date
    May 2007
    Location
    Zimbra
    Posts
    1,285
    Rep Power
    10

    Default

    Quote Originally Posted by kevindods View Post
    What none of us seem to quite understand why the DB has 80GB provisioned when most DBs are a few hundred MB at most. If the DB was closer to expected maximums the issues we experienced would not arise at all. Perhaps if the focus is on server instances in virtualized cloud environments with cloud storage, block devices and simple snapshots then these concerns many smaller users experience do not become apparent.
    It is provisioned at 80GB because MDB has a maxsize limit. To allow upgrading from an unknown size, I had to choose some number that I was sure would allow anyone to upgrade without issue. I know there is no one who uses zimbra with an 80GB database.

    RSYNC is a perfectly reasonable way to backup a system that has no open DBs or files and that is how it has been used. Things change and that is often good, so long as the change continues to help. The idea of dealing better with the LDAP DB is a great idea, but why provision 80GB for all DBs and adversely affect other supporting processes that work in a wider application when there is no need.
    Rsync is not, and never has been, a valid way of backing up the LDAP server. The fact that this is referenced in numerous location is a mistake that should be corrected. Using rsync is the wrong thing to do in any circumstance, regardless of the 80GB copying issue. In any case, there is nothing holding you to using the 80GB maximum data size. This is why there is a localconfig key in ZCS 8 that allows you to adjust the maxsize, and this is why that key is documented on the Zimbra Wiki for OpenLDAP tuning in ZCS 8.

    That last point is the one I am trying to understand, is there a clear need to provision 80GB of DB space for something that is usually no where near that size. Is it not better to fit the purpose as needed and provision more appropriate sizes for the DB and avoid the hassle. I can't answer that which is why I wanted Quanah to explain the drive to provision the 80GB not say 1GB or 500MB. Can you explain that Quanah?
    As noted above -- We needed some maximum that we could ensure any migration would allow for. There are certainly installations with multi-gigabyte LDAP databases. I have a 16GB one from a customer I use for testing internally. I'm fairly positive no one has an 80GB size database.

    Again, no one is forcing you to maintain the 80GB maximum, and the tuning key was explicitly added so that people who had smaller DBs could adjust for it accordingly. However, you must *always* keep your maxsize larger than your total database size, or data loss will occur.

    --Quanah
    Quanah Gibson-Mount
    Server Architect
    Zimbra, Inc
    --------------------
    Zimbra :: the leader in open source messaging and collaboration

Page 5 of 12 FirstFirst ... 34567 ... LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Extending LDAP database
    By frost983 in forum Developers
    Replies: 3
    Last Post: 11-02-2010, 06:46 AM
  2. [SOLVED] Zmsetservername choking on ldap database
    By Emmanuel Kasper in forum Administrators
    Replies: 7
    Last Post: 10-21-2009, 09:50 AM
  3. How do I browse Zimbra LDAP database?
    By williamn in forum Administrators
    Replies: 4
    Last Post: 04-16-2008, 01:47 AM
  4. Replies: 0
    Last Post: 01-14-2008, 11:41 AM
  5. change ldap database
    By Grejao in forum Administrators
    Replies: 1
    Last Post: 12-07-2007, 08:39 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •