Seamless upgrade of 6500 from SUP 720-3B to Sup VS-S2T-10G

[jtgenterprise]

Hello,

I have a requirement to perform a supervisor upgrade for a client. The setup is as following:

2x 6509 switches with 2 active-standby SUP 720-3B supervisors.

My client initally stated that the switches were acting as a primary/secondary HSRP pair, but apparently it is not enrirely the case. This means that although HSRP is configured for a lot of their SVIs, some servers are only physically patched to one of the 2 switches.

Presales have scoped 4x VS-S2T-10G cards to perform a seamless upgrade on the secondary unit first and then the primary. The full BoM is as below:

VS-S2T-10G=              Cat 6500 Sup 2T with 2x 10GbE and 3x1GbE with MSFC5 PFC4
S2TIBK9-15201SY       IOS IP Base
MEM-C6k-INTFL          1GB Internal 1GB Compact Flash
VS-SUP2T-10G           Catalyst 6500 Supervisor Engine 2T Baseboard
VS-F6K-PFC4              Cat 6k 80G Sys Daughter Board Sup2T PFC4
MEM-SUP2T-2GB       Catalyst 6500 2GB Memory for Sup2T and Sup2TXL

The current switches have the same setup as per below:

MUK_Core1#sh module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1     4     CEF720 4 port 10-Gigabit Ethernet                      WS-X6704-10GE
2    24    CEF720 24 port 1000mb SFP                              WS-X6724-SFP
3     8     Network Analysis Module                                     WS-SVC-NAM-2
4    48    CEF720 48 port 10/100/1000mb Ethernet           WS-X6748-GE-TX
5    2      Supervisor Engine 720 (Active)                             WS-SUP720-3B
6    2      Supervisor Engine 720 (Hot)                                 WS-SUP720-3B
7    48    CEF720 48 port 10/100/1000mb Ethernet            WS-X6748-GE-TX
8   48     CEF720 48 port 10/100/1000mb Ethernet            WS-X6748-GE-TX
9   48     CEF720 48 port 10/100/1000mb Ethernet            WS-X6748-GE-TX

My questions are as following:

1) Is it possible to perform a seamless upgrade between on the switches?

I will need to remove the SUP 720-3B hot standby, install the VS-S2T-10G, save the config (and hopefully VLAN.dat) and force a failover.Do some verification tests and also do the same on the (previously active) card.

2) Is there an official guide on how to perform a seamless hardware upgrade on those devices? I could only found an upgrade guide for IOS images but no guide into actually upgrading a supervisor while the chassis is running.

Edit: I have clarified some things and added more information.

[Giuseppe Larosa]

Hello jtenteprise,

first of all, SUP 720 3B does not support VSS, or the current setup is not using a VSS, or the current supervisors are of different type that supports VSS.

Second point: in order to properly support linecards with DFC, you would need to buy and install DFC4 for them.

See the following datasheet for VS-2T-10G

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/catalyst-6500-series-supervisor-engine-2t/data_sheet_c78-648214.html

If you have linecards using DFC, there is no way to perform a seamless migration as you need to power off the chassis, extract each linecard and change every DFC daughther card with DFC4 new one.

Hope to help

Giuseppe

[jtgenterprise]

Hello,

The switches are only running HSRP between them and No VSS.

What I was asking is whether I could perform a line card upgrade without powering the entire switch off. I cannot seem to find any documentation whether this is possible or not.

Each switch has 10 years of uptime. I am too scared to power them off, unplug the old sups and install the new ones. I am pretty sure it will never come back up.

Is a sup upgrade even feasible or do I need a complete outage?

[Giuseppe Larosa · Accepted solution]

Hello,

I understand your concerns.

Normal linecards support online insertion and removal and so you could extract one and then do all necessary operations on it like changing the DFC as  I have explained before.

However, when it comes to supervisors you can remove the standby supervisor and insert the new one, but I am afraid that the system can be unstable or even it can crashes with two supervisors with different models and generation.

Notice also that each supervisor either Sup720 and Sup 2T has also on board its own switching fabric and they have different speeds.

The only  tuning that might help is reviewing the redundancy mode:

stateful switchover SSO requires supervisor models to match for sure.

A possible approach could be that of changing redudancy mode from SSO (likely configured) to RPR that has looser requirements.

However, my guess is that new Sup 2T will stuck in ROMMON state until the first Sup720 is present and active.

Besides this linecards, with new DFC4 will likely be put offline by Sup720 if it finds out a non matching DFC on the linecard.

If all linecards have CFC is easier for the linecards as they do not need any change.

Plan a very long time window for maintenance.

Have all HSRP groups active on chassis1. Disable all access switch links to chassis2 for safety.

And make the migration on chassis2.

From lessions learned on this first migration you will know how long it takes to perform the process.

In the third migration phase you can build the VSS between the two chassis with all new Sup 2T installed and running.

Hope to help

Giuseppe

[jtgenterprise]

Wow,

This was the most detailed explanation I got regarding a sup upgrade!

Thanks.

And by the way, I forgot to post the daughtercard output of the switch. You were right, we do need to install new daughtercards.

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1      Centralized Forwarding Card     WS-F6700-CFC         2.1 Ok
2     Distributed Forwarding Card       WS-F6700-DFC3B     4.6 Ok
4     Centralized Forwarding Card       WS-F6700-CFC        4.1 Ok
5     Policy Feature Card 3                WS-F6K-PFC3B          2.3 Ok
5     MSFC3 Daughterboard              WS-SUP720               3.1 Ok
6     Policy Feature Card 3                WS-F6K-PFC3B          2.3 Ok
6 MSFC3 Daughterboard WS-SUP720 3.1 Ok
7 Centralized Forwarding Card WS-F6700-CFC 4.0 Ok
8 Centralized Forwarding Card WS-F6700-CFC 4.0 Ok
9 Centralized Forwarding Card WS-F6700-CFC 4.0 Ok

[Mark Malone]

Hi
You should really reboot at least once a year by best practice to clear process memory etc , restart fresh switch , processes get hung over time , memory can leak in software etc theres a few reasons but i have worked in hospitals where they had same issues could never get an outage , its a risk running like that though

If your using same sups , a sup upgrade is fine and achievable if using the sups already in the chassis yes , but if you use the new sups which are VSS capable then no ,you need a hit , you can fail between sups in same chassis and limit outages , see below


heres a doc on dual sup 65s upgrade in same chassis minimal downtime , read it fully

Catalyst 6000/6500 Series Switches with Redundant Supervisor Engines Software Image Upgrade Configuration Example

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/71585-cat6k-red-supeng-swimg-upg.html

is that what your trying to do

[Mark Malone]

Hi
there VSS sups , you cant do seamless going to VSS from standard your going to take hits no matter what way , reason being theres a conversion section where it turns the switches into VSS capable , this causes a reboot , no way around it part of the official roll out . But when its done then you can do ISSU on VSS for future upgrades when its complete but even thats not bullet proof and can fail but it has worked for us before without downtime

switch convert mode virtual

! You’ll be asked if it is OK to save the
! running config and reboot the switch.
! Answer yes and then be patient.
! It takes a while for the switch to reboot.

............................
only way it could be seamless if the sups were not being removed , you could upgrade one and failover to the second all in the chassis and upgrade etc without downtime but you have to physically swap sups and they cant sync together 3b and a VS sup

my opinion would be take the hit in a full window and convert both chassis to VSS and get them working as 1 logical unit , also depending on your image your using quad sup per VSS the second sups could be in rommon mode , wasnt supported until newer releases in VSS

https://www.cisco.com/c/dam/en/us/products/collateral/interfaces-modules/network-modules/white_paper_c11_429338.pdf

https://www.netcraftsmen.com/vss-configuration/