Results 1 to 7 of 7

Thread: Upgrading with a clear backout path

  1. #1
    Baylink is offline Elite Member
    Join Date
    Aug 2008
    Location
    St Pete FL USA
    Posts
    392
    Rep Power
    6

    Default Upgrading with a clear backout path

    So... I'm *finally* licensed (CFOs can be such a pain), and running 5.0.9, and pretty clearly, it's time to upgrade now to .15.

    The question is: how?

    More specifically: how do I do an upgrade without a) setting up a whole new machine, or b) overwriting the current program directory, not to mention restructuring any databases that I won't be able to restructure back -- and let's not forget that if I have to back-out, I need to be able to *reprocess* any incoming mail (either from the outside world or from users) that arrives while I'm on the new system.

    I'm *really* big on back-out plans, partially from reading Tom Limoncelli's excellent The Practice Of System And Network Administration, which you should run right out and buy, and partially because it's *email* -- the bosses get cranky when email breaks; that's why I'm moving to Zimbra in the first place.

    Anyone else out there who's as paranoid as me, and has a good answer here?
    Jay R. Ashworth - ZCS 6.0.9CE/CentOS5 - St Pete FL US - Music - Blog - Photography - IANAL - IAAMA
    Try to Ask Questions The Smart Way -- you'll get better answers.

    Put your product and version in your profile/signature - All opinions strictly my own, even though I have an employer these days.
    If you [SOLVE] something, please tell everyone how for the archives
    And, please... read what people write, and answer the questions they asked, not the ones they didn't.

  2. #2
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default

    Quote Originally Posted by Baylink View Post
    So... I'm *finally* licensed (CFOs can be such a pain), and running 5.0.9, and pretty clearly, it's time to upgrade now to .15.

    The question is: how?

    More specifically: how do I do an upgrade without a) setting up a whole new machine, or b) overwriting the current program directory, not to mention restructuring any databases that I won't be able to restructure back -- and let's not forget that if I have to back-out, I need to be able to *reprocess* any incoming mail (either from the outside world or from users) that arrives while I'm on the new system.

    I'm *really* big on back-out plans, partially from reading Tom Limoncelli's excellent The Practice Of System And Network Administration, which you should run right out and buy, and partially because it's *email* -- the bosses get cranky when email breaks; that's why I'm moving to Zimbra in the first place.

    Anyone else out there who's as paranoid as me, and has a good answer here?
    The ultimate backout/rollback plan with a failed upgrade of Zimbra involves reinstalling the old rpms and copying back the entire /opt/zimbra tree as Zimbra upgrades are done in-place on the same server.

    The Zimbra installer that does the upgrade, depending upon the version, tends to update MySQL databases, LDAP schemas, and other core items that just won't work if you simply reinstall the old rpms.

    Migrating to a new server can be a pain, and the in-place upgrades are generally reliable, so that's what we have always done.

    So, at the risk of oversimplifying, here's what we do:
    1. Have a backup MX somewhere else to handle inbound emails while your Zimbra server is down. There's a script on the wiki to extract all the valid email addresses on your Zimbra system. We have a plain-Jane Postfix box at a second data center as a backup MX for all of our hosted domains. Not having to worry about bouncing inbound email during the upgrade is a good thing.
    2. Depending upon how big the /opt/zimbra tree is, either a few days or a few hours before your planned upgrade, with Zimbra still running, rsync the entire /opt/zimbra tree somewhere safe. You can run this rsync again say, 30 minutes before your planned maintenance window begins. The command we use is: rsync -avzH --delete /opt/zimbra root@destination_server:/opt_zimbra_backup.
    3. Once you hit the maintenance window and folks expect Zimbra to be down, block all inbound ports at the firewall to prevent Zimbra from accepting any new emails and users from accessing Zimbra during the upgrade process.
    4. Run zmbackup -f -a all to get a full Zimbra backup.
    5. Run zmcontrol shutdown (stops the Zimbra management processes), and check there are no Zimbra processes running by using top, ps, or whatever else you like.
    6. Run that rsync job one last time, now that zimbra is stopped. This gives you the clean copy of the /opt/zimbra you'll need to rsync back if the upgrade truly fails and a restore from the Zimbra backup doesn't work either.
    7. Run the installer from the new version, and if no errors, take a new full Zimbra backup and then open up the ports on the firewall.
    8. Your are done!


    FWIW, our experience has been that if you:
    • shutdown Zimbra manually, instead of letting the installer do it;
    • read the Release Notes carefully for any potential "gotchas" that require manual work before the actual upgrade;
    • let the installer take whatever time it needs to run whatever checks it wants during the upgrade process, and;
    • let the installer complete even when it looks like it's doing nothing;

    then your upgrades will be pretty seamless.

    Last tip: since we do almost all of our upgrades remotely via ssh, we typically like to have a second ssh window open running top to keep our blood pressure down during the upgrade. The Zimbra installer scripts have a lot of sleep commands in there, and it can be very nerve-wracking watching the installer do nothing for a while. There have been several posts here where users aborted the installer because they thought it had hung. We have never seen that, but, it helps to be patient (and maybe scarf a Valium beforehand if you are the nervous type...)!

    We have also never had an upgrade fail, and we have been running NE since 4.0.3. I expect someday we will; these things eventually happen despite everyone's best intentions. But the plan above gives us two bites at the apple for rolling back, and we think that's a good amount of insurance.

    Hope that helps,
    Mark

  3. #3
    Baylink is offline Elite Member
    Join Date
    Aug 2008
    Location
    St Pete FL USA
    Posts
    392
    Rep Power
    6

    Default

    That covers me for failures *during* the upgrade ... but it doesn't let me back-out afterwards, if I find some nassty bug that eats me precioussesss. :-)

    I guess I'll have to cook up some way to tee off incoming mail on the way in so I have it to replay if I need to.

    Outgoing mail sent during a posited interregnum between upgrade and backout is even harder to deal with... and may be impossible.

    Ok; well, I feel a little bit better now; thanks.
    Jay R. Ashworth - ZCS 6.0.9CE/CentOS5 - St Pete FL US - Music - Blog - Photography - IANAL - IAAMA
    Try to Ask Questions The Smart Way -- you'll get better answers.

    Put your product and version in your profile/signature - All opinions strictly my own, even though I have an employer these days.
    If you [SOLVE] something, please tell everyone how for the archives
    And, please... read what people write, and answer the questions they asked, not the ones they didn't.

  4. #4
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default

    Quote Originally Posted by Baylink View Post
    That covers me for failures *during* the upgrade ... but it doesn't let me back-out afterwards, if I find some nassty bug that eats me precioussesss. :-)

    I guess I'll have to cook up some way to tee off incoming mail on the way in so I have it to replay if I need to.

    Outgoing mail sent during a posited interregnum between upgrade and backout is even harder to deal with... and may be impossible.

    Ok; well, I feel a little bit better now; thanks.
    Jay,

    You are right that our process gives you rollback protection during the upgrade but not afterwards.

    The way we protect against having to do a rollback hours or days after an upgrade is to avoid that possibility. We don't have a plan to be able to do that, and I'm not sure you can, really since I don't know how you would record and later play back things like users moving emails from one folder to another or changing their passwords--especially when many upgrades make fundamental changes to the LDAP schema and the MySQL databases (as well as sometimes updating OpenLDAP and MySQL themselves)..

    When we are nervous, we do upgrades on some test virtual machines we have, with restored copies of our production stores.

    We also watch the forums here, to see if others who have a need to upgrade sooner experience any issues.

    If you look at the history of 5.0.12 > 5.0.15, .12 was a highly QA'd release with one nasty "gotcha". .13 through .15 were released in quick succession to address single issues.

    FWIW, we held pat on 5.0.8, avoiding all upgrades until 5.0.15, and we have had zero issues as I said with the upgrade process and with the post-upgrade running of 5.0.15.

    I'm not going to say you may not have problems with 5.0.15, but if you search the forums for "5.0.15" you will quickly see the challenges others have faced with this version, and then you can evaluate whether your installation will likely face those challenges--or not.

    But the bottom line is that I have never heard of anyone doing a post-upgrade rollback several days later without the loss of all inbound emails and changes since the upgrade was performed.

    All the best,
    Mark

  5. #5
    gmsmith is offline Moderator
    Join Date
    Apr 2006
    Location
    Williamsburg, VA
    Posts
    451
    Rep Power
    8

    Default

    Yeah, let me echo Mark's comments...don't jump on an upgrade the first day it is out...watch the forums and do your own off production system testing. We usually upgrade about 30 days after Zimbra releases. This gives us internal QA time and lets others flush out some issues that may arise.

    Mark's backup process is outstanding and should be followed, this will protect you during the upgrade. I might also suggest if you are doing remote ssh connections to use screen, so in case you lose local connectivity you don't run the risk of any issues and always monitor /tmp/zmsetup.log with a tail -f, keeps the blood pressure down (for the most part).

  6. #6
    Baylink is offline Elite Member
    Join Date
    Aug 2008
    Location
    St Pete FL USA
    Posts
    392
    Rep Power
    6

    Default

    Good points, all. I'm a professional paranoid, but I guess at some point... :-)

    Thanks, guys.
    Jay R. Ashworth - ZCS 6.0.9CE/CentOS5 - St Pete FL US - Music - Blog - Photography - IANAL - IAAMA
    Try to Ask Questions The Smart Way -- you'll get better answers.

    Put your product and version in your profile/signature - All opinions strictly my own, even though I have an employer these days.
    If you [SOLVE] something, please tell everyone how for the archives
    And, please... read what people write, and answer the questions they asked, not the ones they didn't.

  7. #7
    bdial's Avatar
    bdial is offline Moderator
    Join Date
    Jul 2007
    Location
    Baltimore
    Posts
    1,649
    Rep Power
    10

    Default

    this is one of the reasons i am absolutely in love with vmware. by snapshotting a server, you can do whatever you want to it and then if you really hose things up just revert to the snapshot wtihin seconds. it will even snapshot the running memory so when you revert your server returns to the exact running point it was at.

    currently our zimbra mta & ldap servers are virtual. i always snapshot them both before upgrade in case something goes wrong with either upgrade. of course my mailbox servers are still physical and they're usually the most problematic so i still sweat a bit there! one of my goals this year is to get them virtualized.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 29
    Last Post: 10-31-2012, 11:11 PM
  2. upgrading from 5.0.4 to 5.0.5 opensource
    By smoke in forum Installation
    Replies: 4
    Last Post: 10-19-2008, 10:38 AM
  3. Replies: 4
    Last Post: 04-26-2006, 10:11 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •