Doing the Rolling Upgrade from 6120 to 6248 on a production environment

[BobWhitcombe]

Is there a detailed runbook for this process?  Or material that postdates the August 2015 material in the "Upgrading Cisco UCS Hardware" Chapter on moving between UCS 2.0 releases.

The project is to first upgrade the client to a stable FW across all the 6120 fabrics, IOM's, and servers (B230-M2, B420-M3, B200-M4) of FW2.2.(8f) across all elements.  Then decom the support 6120 and add in the 6248 and proceed.  Ultimate objective is to upgrade the fabrics to 6248's, Upgrade two Chassis IOM's from 2104 to 2204 and all the M81KR's on 16 B230-M2's to VIC-1280 - then upgrade the entire environment to FW 3.1.

We have strongly advised against performing this on a production environment without taking an outage, the client says they have no choice. Whilst indemnifying ourselves - we are performing due diligence to acquire the latest material.  Concerned about details and gotchas given we are using a FW set several times removed from what was available 18 months ago when this was written, and moving to a completely new model with the 3.1. 

[cweinhold]

I'd be worried a bit about behavior changes with jumping FW releases. E.g., blindly bringing forward a 4-year old BIOS policy is asking for serious trouble. Similarly, there are UCS Manager changes that are nominally backward compatible, but relying on that might not be in your client's best interest. (firmware management, iSCSI IQNs, pinning versus disjoint L2, etc -- there's a lot of room to take advantage of new features and ways of configuring old things).

You could use the chassis as an excuse to steer the upgrade path. They're certainly using N20-6508, which are now end-of-sale. The replacement UCSB-5108-AC2 has been out for a few years now, and has added IOM connectivity that Cisco has started to use with UCSM 3.x.  I would encourage your client to build all new FI/chassis/IOM pairs, and then bring servers over one-at-a-time. You can reuse service profiles, or you could build new ones. It's hard to preserve identities (MAC/WWPN) with such a move, but it _can_ be done.

If they do insist on moving forward with an in-place upgrade, you should insist on this before embarking:

-Craig

[BobWhitcombe]

Thanks Craig for your reply.

Completely agree. Just did a walk-through today – see the environment below. Not only that, but neither L1 nor L2 were connected between the two fabrics – indicating they had the two fabrics but they are not in a cluster. When I tried to gently move the orange fiber cables up to confirm device I was warned the client would prefer it not be touched as they had lost connectivity in the past and one of their projects was to “clean up the mess and shorten the cables”.

☺[cid:image001.png@01D29DA9.3590C680]

[cweinhold · Accepted solution]

No L1/L2 ?!  That means the secondary FI/IOM's are essentially offline and haven't seen config changes or software upgrades in years. If the primary failed, it'd take many hours for a UCS expert to get the system back up on the secondary. Needless to say, a fully operational cluster is a prerequisite for a non-disruptive 6248 migration...

With those physical warning signs, I'll bet you can't wait to see the logical configuration!

When I encounter a new UCS system, I first run its config through my UCS Config Parser tool (you can find it at http://beeline.org/ucs/).  It's like an uglier, offline version of Cisco's UCS Health Check script. The one thing it does well is group service profiles by similarities, which makes it easy to spot inconsistencies ("why does server 7 have a different host firmware pack from the rest?").  It also sniffs up some common config problems, like missing objects.

[BobWhitcombe]

Thanks for this – I have installed lots of greenfield but not done any upgrades and was aware of the Health Check – but it did not come to mind on this until you mentioned it. Will have them run this and your Config Parser tool as we will be generating at least 20 added SP’s to accommodate the platform upgrade changes pedning.

Thanks again!