9800L-C in HA Service-Port Problem after SW-Upgrade

[Gehrig_W]

Hello Cisco WLAN-experts,

after upgrading a HA-pair of 9800-L-C from 17.3.5b to SW-version 17.3.6 we noticed that the ServicePort-Interface Gi0

is no longer working. I was connected via its IP-adress during the ISSU-upgrade.

As soon as the former Standby unit became active with the new image, I was logged out.

I did a "redundany force-failover" after the successful ISSU-upgrade to bring the inital Active unit again into its correct role.

All ports came up with the exception of the Service Port interface GigabitEthernet0:

In the GUi its show a red arrow for Status, a green arrow for Admin status. Here is the output of "show int gi0"

LC-9800-Guest#show int gi0
GigabitEthernet0 is down, line protocol is down
Hardware is RP management port, address is f4bd.9e58.4481 (bia f4bd.9e58.4481)
Description: Out-of-Band via Sw_144_36_OOB_vMotion_HA Ten1/0/30
Internet address is 10.208.36.52/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto Duplex, Auto Speed, link type is auto, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 1d23h, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec

What might cause this issue ?

Unfortunately the following good old Cisco-commands are not available on the brandnew 9800-platform:

show int counter error

show interface link

test cable tdr

The Out-of-Band-switch on the opposite side of Gi0 is an HP-switch by the way:

<Sw_144_36_OOB_vMotion_HA>display current int Te1/0/30
#
interface Ten-GigabitEthernet1/0/30
port link-mode bridge
description OOB_GUEST_WLC_9800_KNE
undo enable snmp trap updown
port access vlan 1193
stp root-protection
stp edged-port

<Sw_144_36_OOB_vMotion_HA>display int Te1/0/30
Ten-GigabitEthernet1/0/30
Current state: DOWN
Line protocol state: DOWN
IP packet frame type: Ethernet II, hardware address: 7848-5941-70d3
Description: OOB_GUEST_WLC_9800_KNE
Bandwidth: 1000000 kbps
Loopback is not set
Media type is twisted pair, port hardware type is 1000_BASE_T_AN_SFP
Unknown-speed mode, unknown-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
Flow-control is not enabled
Maximum frame length: 10000
Allow jumbo frames to pass
Broadcast max-ratio: 100%
Multicast max-ratio: 100%
Unicast max-ratio: 100%
PVID: 1193
MDI type: Automdix
Port link-type: Access
Tagged VLANs: None
Untagged VLANs: 1193
Port priority: 0
Last link flapping: 2 days 2 hours 37 minutes

Any ideas how I can dig deeper into this problem on Cisco 9800-L-C or HP-switch and get the line protocol up and running again ?

Shall I reload the 9800C-L-HA to bring the Interface up and running again ?

Any good tipps welcome

Have a nice weekend

Greetings from Frankonia

Wini

[marce1000]

  - Normally this should kind of never happen , start by a  test of pulling the port connection at the controller and or the hp switch and connect it again , then review controller logs  and the logs on the hp switch too. Make sure port configuration(s) on the hp switch are correct of an end node connection for this connection (such as the equivalent of mode access and port-fast on a Cisco switch).
For the time being this is  probably unrelated , but it is a good idea to review the 9800-L-C configuration anyway  with the CLI command : show  tech   wireless , have the output analyzed by  https://cway.cisco.com/tools/WirelessAnalyzer/  , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer.               Checkout all advisories!

 M.

[Rich R]

Have you checked the PHY firmware is up to date as per the TAC recommended link below?

And next stop TAC case - assuming you have checked and re-seated the cable

[Gehrig_W]

Hello marce1000,

thank You for Your hint regarding the WLC config Analyzer.

may I remind You on Murphy´s law ! The Wireles Analyzer shows nothin in relation to the non-working Gi0-Interface on the 9800-L-C after the SW-Upgrade.

Hello Rich R,

thank You for Your hint upgrading the PHY  firmware.

HEre we are running on an obsolete version as far as I can see:

WLC-9800-Guest#show rom-monitor chassis standby R0
==========================================================

System Bootstrap, Version 16.12(1r), RELEASE SOFTWARE
Copyright (c) 1994-2019 by cisco Systems, Inc.

WLC-9800-Guest#show platform hardware chassis active qfp datapath pmd ifdev | i FW

FW Version : 0x80000757
FW MDIO : 9.1.2 ID: 43503 vers: 1376
FW Version : 0x80000757
FW MDIO : 9.1.2 ID: 43503 vers: 1376
FW Version : 0x80000756
FW MDIO : 9.1.2 ID: 43503 vers: 1376
FW Version : 0x80000756
FW MDIO : 9.1.2 ID: 43503 vers: 1376
FW Version : 3.1.76
FW Version : 3.1.76

I will plan for an upgrade next week and keep You informed.

Thanks a lot.

Wini

[marce1000]

 - If possible , also try to check if the problem is related to the switch (model) and or by using  a Cisco switch (e.g.) , 

 M.

[Rich R]

Yes update ROMMON: https://software.cisco.com/download/home/286323158/type/282046486/release/16.12(3r)

and latest PHY https://software.cisco.com/download/home/286323158/type/283425232/release/17.11.1

although yet again they've not documented the changes in that release (no release notes and not mentioned in the upgrade doc).  It took a TAC case to get them to document the previous changes but they clearly have not learned the lesson.  If you want to be cautious stick to 17.3.2 which is minimum required.

[UKW-NK-Cisco]

Hello Rich R

thank You very much for Your advise. I will go forward to upgrade to the advised 17.3.2-version.

Let's see whether this cures the Gi0-Interface.

Kind regards

Wini