topic Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade in Wireless

9800-L-C in HA Rommon and Ethernet Phy Upgrade

Gehrig_W — Mon, 13 Mar 2023 14:16:44 GMT

Hello Cisco WLAN Experts,

I was asked to upgrade the ROMMON and Ethernet Phy Firmware of our malworking 9800-L-C-HA für WLAN Guests.

Background: Our 9800-L-C gets stuck from time to time and stops packet forwarding on 2.5Gbps-Interfaces

Ticket is open and we have carried out several remediations on this box running with around 4000 users.

The latest Release Notes on Upgrading Field Programmable on 9800 show a solved caveat regarding packet drop on the 2.5Gbps-Interface, which we would like to solve with this upgrade.

9800L: MAC congested @Low PPS, drops/TX_XOFF in Bay 0-Tune settings and pause frames in mgig ports
CSCvq48018

Therefore I would like to know the right procedure to upgrade two of 9800-L-C in HA-Mode.

The guide for this is rather confusing:

https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_upgrade_fpga_c9800.html#id_132283

My questions:

In which order and which steps do I need to upgrade the two boxes ?

1. ROMMON

upgrade rom-monitor filename bootflash:C9800-L-rommon.1612-3r.pkg chassis active r0

Shall I follow this command with a "Reload" or a "redundancy force-switchover" to carry out same command on the second box ?

2. Ethernet-Phy Firmware Upgrade

upgrade hw-programmable phy filename bootflash:C9800-L-hw-programmables.17.11.01.pkg

Shall I follow this command with a "Reload" or a "redundancy force-switchover" to carry out same command on the second box ?

The Guide is only describing a ROMMON-Upgrade of a HA-Setup on the last page, but not the Ethernet-Phy-Upgrade

for a HA-setup !

Who can give some good advise how to carry out the upgrade in correct order and sequence for both acitve and standby unit ?

Thank You in advance

Wini

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Mark Elsen — Mon, 13 Mar 2023 16:55:25 GMT

- To be honest when it comes to cases like this and or TAC case is already open , you should have a valid TAC procedure (request) too in the context of business importance (pointing to TAC as the 'authoritative responder') ; some initial remark from https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_upgrade_fpga_c9800.html#id_132227
>...If your setup is in High Availability mode, break the High Availability

If you talk about the difference between "reload" and "redundancy force-switchover" in your post then the first will reload the complete HA SSO pair and "redundancy force-switchover" will reboot the current (active controller ) only , e.g.

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Leo Laohoo — Tue, 14 Mar 2023 05:15:01 GMT

Post the complete output to the following commands:

sh version
sh platform

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

UKW-NK-Cisco — Tue, 14 Mar 2023 07:29:21 GMT

Hello Marce1000 and Leo Laohoo,

thank You for Your feedback.

That's what I meant with confusing information from the Upgrade guide:

From https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_upgrade_fpga_c9800.html#id_132227
>...If your setup is in High Availability mode, break the High Availability

But at the end of the document, an action plan is shown to update a HA-configuration without breaking the HA:

Upgrading ROMMON in a High-Availability Topology

This procedure helps to upgrade ROMMON of the controller in an High Availability (HA) topology.

Step 1	Run the upgrade command on the active device. Device# upgrade rom-monitor filename bootflash:rommonfile.pkg chassis active r0 The ROMMON gets upgraded and requires a reboot.
Step 2	Run the following command on the current active to complete the upgrade. Device# redundancy force-switch over After running this command, wait for the controller to join the HA pair and proceed to the next step.
Step 3	Run the upgrade command on the standby device. Device# upgrade rom-monitor filename bootflash:rommonfile.pkg chassis active r0 The ROMMON gets upgraded and requires a reboot.
Step 4	Run the following command on the standby device to complete the upgrade. Device# redundancy force-switch over After running this command, wait for the controller to join the HA pair.
Step 5	Use the show command to verify the updates on the HA setup. Device# show rom-monitor chassis active r0

So my question is: Is it necessary to break the HA on beforehand or not ?

Can the Ethernet-PHy-upgrade be done in same manner without breaking the HA ?

And here is the output of "show version" and "show platform"

WLC-9800-Guest#show version
Cisco IOS XE Software, Version 17.03.06 - SMU-PATCHED
Cisco IOS Software [Amsterdam], C9800 Software (C9800_IOSXE-K9), Version 17.3.6, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2022 by Cisco Systems, Inc.
Compiled Fri 16-Sep-22 02:09 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2022 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: 16.12(1r)

WLC-9800-Guest uptime is 7 weeks, 4 days, 23 hours, 37 minutes
Uptime for this control processor is 5 days, 18 hours, 3 minutes
System returned to ROM by SSO Switchover at 14:20:54 MEZ Wed Mar 8 2023
System restarted at 14:25:56 MEZ Wed Mar 8 2023
System image file is "bootflash:packages.conf"
Last reload reason: redundancy force-switchover

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Type: Smart License is permanent
License Level: adventerprise
Next reload license Level: adventerprise
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage

The current crypto throughput level is 0 kbps

Active SMU Information:
State (St): C - Committed, U - Uncommitted
--------------------------------------------------------------------------------
Type Defect_ID Version St Filename
--------------------------------------------------------------------------------
SMU CSCwd03847 17.03.06. C C9800-L-universalk9_wlc.17.03.06.CSCwd03847.S
SMU CSCwc05366 17.03.06. C C9800-L-universalk9_wlc.17.03.06.CSCwc05366.S
--------------------------------------------------------------------------------

Smart Licensing Status: Registration Not Applicable/Not Applicable

cisco C9800-L-C-K9 (KATAR) processor (revision KATAR) with 1702611K/6147K bytes of memory.
Processor board ID FCL2418007M
Router operating mode: Autonomous
2 Virtual Ethernet interfaces
4 2.5 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
16777216K bytes of physical memory.
26251263K bytes of eUSB flash at bootflash:.
26251263K bytes of eUSB flash at bootflash-2:.

Base Ethernet MAC Address : F4:BD:9E:58:44:80

Installation mode is INSTALL

Configuration register is 0x102

WLC-9800-Guest#show platform
Chassis type: C9800-L-C-K9

Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 C9800-L-C-K9 ok 5d18h
0/0 BUILT-IN-4x2_5GE ok 5d18h
0/1 BUILT-IN-2x10GE-C ok 5d18h
R0 C9800-L-C-K9 ok, active 5d18h
F0 C9800-L-C-K9 ok, active 5d18h
P0 PWR-12V ok 5d18h
P1 C9800-L-C-K9-FAN ok 5d18h

Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 19050918 16.12(1r)
R0 19050918 16.12(1r)
F0 19050918 16.12(1r)

WLC-9800-Guest#

WLC-9800-Guest#show chassis rmi
Chassis/Stack Mac Address : f4bd.9e58.4480 - Local Mac Address
Mac persistency wait time: Indefinite
Local Redundancy Port Type: Twisted Pair
H/W Current
Chassis# Role Mac Address Priority Version State IP RMI-IP
--------------------------------------------------------------------------------------------------------
*1 Active f4bd.9e58.4480 2 V02 Ready 203.0.113.2 NA
2 Standby f4bd.9e58.4840 1 V02 Ready 203.0.113.1 NA

Please advise

Kind regards

Wini

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Leo Laohoo — Tue, 14 Mar 2023 07:45:22 GMT

@UKW-NK-Cisco wrote:
ROM: 16.12(1r)
0 19050918 16.12(1r)
R0 19050918 16.12(1r)
F0 19050918 16.12(1r)

The WLC is already on ROMMON version 16.12(1r).

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Mark Elsen — Tue, 14 Mar 2023 07:50:56 GMT

>... an action plan is shown to update a HA-configuration without breaking the HA:
I know , the document is confusing , therefore and as I stated : use and or get an authoritative procedure from Cisco TAC ,

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

UKW-NK-Cisco — Tue, 14 Mar 2023 07:56:46 GMT

Yes, it is.

I was asked to upgrade to version 16.12(3R) and 17.3.2 (newer is 17.11.1)

And Yes, I have asked TAC for a clear procedure.

Kind regards

Wini

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Mark Elsen — Tue, 14 Mar 2023 08:08:07 GMT

>...And Yes, I have asked TAC for a clear procedure.
Good!

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Leo Laohoo — Tue, 14 Mar 2023 08:36:12 GMT

I have a solution to make this simpler but it will mean rebooting both units at the same time.

Here are the steps:

Copy the ROMMON file into the Active unit.
Copy the ROMMON file into the Secondary unit.
Initiate the ROMMON upgrade of the Active unit using the command "upgrade rom-monitor filename bootflash:rommonfile.pkg chassis ACTIVE r0"
Initiate the ROMMON upgrade of the Secondary unit using the command "upgrade rom-monitor filename bootflash:rommonfile.pkg chassis STANDBY r0".
Save and rebot both units.

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Mark Elsen — Tue, 14 Mar 2023 08:53:06 GMT

- I advise against this because it may lead to 'unmastered events' and or having the risk of both controllers being unavailable when something goes wrong. Also wireless services (production) will get interrupted when both controllers are unavailable. The documentation also states that the controller may be rebooted several times during rommon upgrade(s) , it may confuse HA too. Use TAC procedure 'when it comes...' ,

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Leo Laohoo — Tue, 14 Mar 2023 08:56:26 GMT

@Mark Elsen wrote:
The documentation also states that the controller may be rebooted several times during rommon upgrade(s) , it may confuse HA too.

I have upgraded the ROMMONs of 4 x 9800-80 and two 9800-L-C.

None of them had an unexpected rebooted.

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Rich R — Tue, 14 Mar 2023 12:31:20 GMT

Same process will apply to ROMMON and PHY and I would do it the way the upgrade guide suggests, although Leo's process could work too, but like Marce I'd feel safer doing them one at a time - so:
- Upgrade active
- switchover
- Upgrade standby (now active)
- switchover
At this point you're back on the primary active and both have been reloaded with new firmware. You'll need to do this for ROMMON and then do it again for the PHY. I wouldn't risk doing them at the same time as that can have unpredictable effects due to the upgrade scripts saving and restoring the environment variables before and after the upgrade so doing them both at the same time could end up restoring the wrong environment variables.

And before you start make sure you have working console access to both chassis in case something goes wrong - I've seen the standby go into maintenance mode unexpectedly for example.

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

UKW-NK-Cisco — Tue, 14 Mar 2023 13:52:14 GMT

Hello Rich R,

thank You for precise description of upgrading ROMMON and Ethernet Phy on 9800-L-C in HA-constellation.

I have planned the upgrade tomorrow after green light from hospital and IT change board and will keep You informed about the outcome. Maybe it can help to improve the according Upgrading manual for 9800-WLC-customers.

By the way, Cisco also needs to upgrade the document "Recommended Cisco IOS XE Releases for Catalyst 9800 WLCs" regarding the Version of Ethernet Phy. ( Old: 17.3.2 New: 17.11.1)

Kind regards

Wini

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Rich R — Tue, 14 Mar 2023 13:40:27 GMT

You can use the feedback option to report the update needed. TAC might be sticking with the old version intentionally though for the same reason as us. It took a TAC case and months of escalation and follow up to get the upgrade guide updated previously. The team responsible seem to be really lazy about updating that guide, and clearly have not learned their lesson!

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Leo Laohoo — Tue, 14 Mar 2023 22:06:29 GMT

@UKW-NK-Cisco wrote:
Cisco also needs to upgrade the document

Any English-writing instructor will have a massive stroke if they happen to see the quality of the documents that are out there.

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Gehrig_W — Wed, 15 Mar 2023 10:17:54 GMT

Hello

I would like to inform You that the ROMMON and Ethernet PHY upgrade went through without any disturbance on our 9800-L-C Guest WLC according to proposal from Rich R. Here is the full cooking recipe without breaking HA and WLAN client disconnects:

Thank You Rich R, Leo Laohoo and marce1000 for Your good hints and assistance.

I asked TAC also to update the Install guide regarding "Breaking HA first" which is not necessary

Kind regards

Wini

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Rich R — Wed, 15 Mar 2023 11:05:11 GMT

Excellent - thanks for sharing.
Hopefully this resolves your original problem.

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

NickyJKernow — Fri, 12 Jan 2024 17:21:08 GMT

Hi,

Did this ever help resolve the original issue you had?

Re: 9800-L-C in HA Rommon and Ethernet Phy Upgrade

Gehrig_W — Mon, 15 Jan 2024 10:43:11 GMT

Hello Nick,

no, unfortunately this didn't help to resolve our main issue. The bug solved with this is CSCvq48018 instead.

one major problem on the 9800-L was not working autonegotaion on the physical Twin-Gig-Ports.

In our case the box is connected to an HP-switch and autonegotion did not work using LLDP.

The 9800-L worked with 2,5Gbps, while the HP-Switch communicated on 1Gbps using autonegotiation. This caused packet loss and DHCP-problems durign peak times. We changed to "speed 1000" and "duplex full" on both sides and "no autonegotation auto" on the 9800-L to get rid of the DHCP-problems.

Kind regards

Wini