Patching Strategy - poll

[Carolanne Fougerat]

We have been hotfix averse in our 5.3 environment and endeavor to do better in 6. While realizing the burden that testing and regression testing incurs on every one - not to mention the 'Oops, we didn't test for that' PRD incidents that happens from time to time, I was wondering if others could share their patching strategy for Tidal.

I would be interested to know:

• Do you actively look at the monthly releases and do you ALWAYS apply the monthly released hotfixes?
  
• If not always, how often do you routinely patch (not including critical hotfixes)
  
• Does Tidal fall under some compliance guidelines that require you to be updated on patches?
  
• How long does the new hotfix sit in DEV before you bless it for PRD? Do you have a formal regression...or any testing that users sign off on for each hotfix before it can be applied to PRD or do you just let it sit in DEV for a month or two and apply to PRD if no issues are reported? We have 15 or so teams that use Tidal so its a challenge since they use the product differently.
  
• If things go awry in PRD, do you perform a backout, or do you engage support but not back out hotfix?
  
• Do you automate any of your testing (ha!) - really wanting to get some basic testing criteria for patches - in addition to the info from the readme that you care about testing.
  
• Do you bundle your hotfix with the DB and OS patches in one go, or do you deploy them separately?
  
• We hadn't planned on patching our agents when we upgrade to 6 (too many moving parts). Any tips on how you handle agent patching - do you do it in one go or rollout a few at a time? Is it autmated?
  

Thanks again - this forum is really useful to to learn how other companies use, manage and maintain this tool. It is one of our most critical applications so we really thread lightly on any changes to PRD.

[Marc Clasby]

• Do you actively look at the monthly releases and do you ALWAYS apply the monthly released hotfixes?
  
  • 5.3.1 Yes we review the readme and decide on a case by case basis
    
    • We were bit by the change in quoted behavior
      
  • 6.1 We will be applying patches more frequently but apply each one – 1 month lag
    
• If not always, how often do you routinely patch (not including critical hotfixes)
  
  • 5.3.1 We do not routinely patch (current prod rev 5.3.1.318)
    
  • 6.1 – 1 month lag ( honestly I feel more comfortable now that they are publishing monthly)
    
• Does Tidal fall under some compliance guidance that require you to be updated on patches?
  
  • Not for our organization we have business and change control but no software patching requirement
    
• How long does the new hotfix sit in DEV before you bless it for PRD? Do you have a formal regression...or any testing that users sign off on for each hotfix before it can be applied to PRD or do you just let it sit in DEV for a month or two and apply to PRD if no issues are reported? We have 15 or so teams that use Tidal so its a challenge since they use the product differently.
  
  • No formal timetable for testing in DEV but we also then apply to UAT then to PRD. In general we run for several months in DEV 1 month in UAT. We time prod with a windows patch weekend. We also leverage a Vcloud/Wmware Lab environment and look at the install via a DEMO license... which we have found very useful
    
  • Generally our internal customers are either application support or developers who can pick up on root cause easier that regular users
    
• If things go awry in PRD, do you perform a back out, or do you engage support but not back out hotfix?
  
  • It all depends on severity of impact, ability to reach support, and time to apply fix.
    
  • We wouldn’t be afraid to either back out or fix and move forward but would likely back out to “working” version
    
• Do you automate any of your testing (ha!) - really wanting to get some basic testing criteria for patches - in addition to the info from the readme that you care about testing.
  
  • We have a series of jobs that test some basic functionality but it could always be expanding and improved. We insert them ad-hoc to test but plan on automating a "test suite" of jobs with more frequent patching
    
• Do you bundle your hotfix with the DB and OS patches in on go, or do you deploy them separately?
  
  • We try to limit the changes where possible. From an O/S perspective we patch Prd every other month and dev every month… so we just time properly
    
  • Depending on Cisco patch we might run on a non PRD O/S patch weekend
    
• We hadn't planned on patching our agents when we upgrade to 6 (too many moving parts). Any tips on how you handle agent patching - do you do it in one go or rollout a few at a time? Is it autmated?
  
  • I like you thinking.. As long as Agents are at a 6x compatible level I would recommend that as a follow up change after you are confirmed stable.
    
  • Typically we roll out all simultaneously, currently not automated but we are looking to automate for 6.1.x
    

[Carolanne Fougerat]

Thanks for your response. Just to give you an idea of what were are thinking about on our end:

Frequency of patching:

We want to patch a minimum of twice a year. The frequency may increase once we get the confidence. Currrently being in the middle of upgrade we patch 6.1 a lot as we try to get to that stable (to us) version. There is really not a lot of new features we want to implement and we tend err on the thinking that a known set of bugs in a mostly stable environment is easier to live with compared to applying a new hotfix with unknown bugs.

One snag I already see though is browser version support since IE and FF patch a lot and they are mostly security related.

Testing hotfix:

A stripped down version of the upgrade testing scripts + testing readme items that impact us = what we hope users will validate after hotfix is applied to DEV. Then after 2 - 3 months, we apply to PRD once all signoffs are complete.
Per automating the testing portion, I guess its easy enough to create a jobgroup that has different types of job and schedule scenarios. The user interaction with the interface is the issue since we don't have tools to simulate that yet. I have used a tool called Badboy in the past - dunno if that is overkill compared to just testing manually. If we patch more frequently than twice a year it may make more sense to.

Strategy for Patching:

• I have been conflicted about the value of applying a hotfix version behind. That was my original plan, that is also the strategy of our Oracle RAC admins, but I am still on the fence on whether to do this or apply the latest anyway. I guess this will depend on what the latest says it has addressed since the prior hotfix and see if we can live with not having those.
  
• Regarding back out - we have had to do this in the past due to an item that was not tested that turned out to have a bug and I can't remember if they had to restore the DB as well since I wasn't the Tidal admin then OR if it was all the agents that had to be reverted to prior version. I guess needing to restore the DB would be more of a possibility with upgrades rather than hotfixes - can't really remember since we haven't patched PRD in so long. So I have no concrete plan on this yet.
  
• Applying hotfix during Linux OS patching may not work for us. Our 5.3 is in Windows (quick downtime)- but 6.1 is Linux and that downtime makes server unavailable to us for an hour. Oracle cluster patching is a few hours long. So we are thinking we can't bundle our patching with theirs either. We are however planning to incorporate our DB DR architecture for 6.1 so that we don't incur a 4 hour downtime like we currently do each quarter for Oracle RAC patching. Once we failover the live DB to the standby DB (on the other data center that is not being patched), we should only have a 15 minutes downtime on tidal.