9600 quad supervisor, stackwise, SVL link err-disabled

[balzsnacsa]

Hello guys. I have the following problem. In a Virtual StackWise environment, there are four SVL links and two DAD links. One of the SVL link err-disabled. I changed QSFP. How can I turn this one back on? There are no "shutdown"/"no shutdown" commands at the SVL port.

sh stackwise-virtual dual-active-detection
In dual-active recovery mode: No
Recovery Reload: Enabled

Dual-Active-Detection Configuration:
-------------------------------------
Switch Dad port               Status
------ ------------           ---------
1      TwentyFiveGigE1/1/0/47 up
       TwentyFiveGigE1/1/0/48 up
2      TwentyFiveGigE2/1/0/47 up
       TwentyFiveGigE2/1/0/48 up

sh stackwise-virtual link
Stackwise Virtual Link(SVL) Information:
----------------------------------------
Flags:
------
Link Status
-----------
U-Up D-Down
Protocol Status
---------------
S-Suspended P-Pending E-Error T-Timeout R-Ready
-----------------------------------------------
Switch SVL Ports                        Link-Status Protocol-Status
------ --- -----                        ----------- ---------------
1      1   FortyGigabitEthernet1/5/0/23 U           R
           FortyGigabitEthernet1/5/0/24 U           R
           FortyGigabitEthernet1/6/0/23 U           R
           FortyGigabitEthernet1/6/0/24 D           T
2      1   FortyGigabitEthernet2/5/0/23 U           R
           FortyGigabitEthernet2/5/0/24 U           R
           FortyGigabitEthernet2/6/0/23 U           R
           FortyGigabitEthernet2/6/0/24 D           S


sh stackwise-virtual neighbors
Stackwise Virtual Link(SVL) Neighbors Information:
--------------------------------------------------
Switch SVL Local Port                   Remote Port
------ --- ----------                   -----------
1      1   FortyGigabitEthernet1/5/0/23 FortyGigabitEthernet2/5/0/23
           FortyGigabitEthernet1/5/0/24 FortyGigabitEthernet2/5/0/24
           FortyGigabitEthernet1/6/0/23 FortyGigabitEthernet2/6/0/23
           FortyGigabitEthernet1/6/0/24
2      1   FortyGigabitEthernet2/5/0/23 FortyGigabitEthernet1/5/0/23
           FortyGigabitEthernet2/5/0/24 FortyGigabitEthernet1/5/0/24
           FortyGigabitEthernet2/6/0/23 FortyGigabitEthernet1/6/0/23
           FortyGigabitEthernet2/6/0/24

sh int status | inc StackWise
Twe1/1/0/47 #### StackWise DAD connected 4094 full 25G SFP-10/25GBase-CSR
Twe1/1/0/48 #### StackWise DAD connected 4094 full 25G SFP-10/25GBase-CSR
Fo1/5/0/23 #### StackWise SVL connected 4094 full 40G QSFP 40GE CSR-S
Fo1/5/0/24 #### StackWise SVL connected 4094 full 40G QSFP 40GE CSR-S
Fo1/6/0/23 #### StackWise SVL connected 4094 full 40G QSFP 40GE CSR-S
Fo1/6/0/24 #### StackWise SVL err-disabled 4094 full 40G QSFP 40GE CSR-S
Twe2/1/0/47 #### StackWise DAD connected 4094 full 25G SFP-10/25GBase-CSR
Twe2/1/0/48 #### StackWise DAD connected 4094 full 25G SFP-10/25GBase-CSR
Fo2/5/0/23 #### StackWise SVL connected 4094 full 40G QSFP 40GE CSR-S
Fo2/5/0/24 #### StackWise SVL connected 4094 full 40G QSFP 40GE CSR-S
Fo2/6/0/23 #### StackWise SVL connected 4094 full 40G QSFP 40GE CSR-S
Fo2/6/0/24 #### StackWise SVL notconnect 4094 full 40G QSFP 40GE CSR-S



int Fo1/6/0/24
(config-if)#?
Distributed stack interface configuration commands:
  default            Set a command to its defaults
  description        Interface specific description
  exit               Exit from interface configuration mode
  no                 Negate a command or set its defaults
  stackwise-virtual  Configure Stackwise Virtual link values


(config-if)#no ?
  description        Interface specific description
  stackwise-virtual  Configure Stackwise Virtual link values
(config-if)#

[pman]

Hi,

Error-disable of the SVL/DAD Links

Possible Cause

The transceiver/SFP/QSFP used on the SVL/DAD links might be faulty at the hardware layer and the xcvrs might itself generate frequent link flaps and thereby error-disable the SVL/DAD links.

Tips to troubleshoot

1. Verify show idprom interface <SVL/DAD link>

2. Verify if any of the interfaces are indeed err-disabled on the switch, use show interfaces status err-disabled.

If you are unable to find the root cause please provide these command outputs

1. show errdisable flap-values

2. show errdisable recovery

3. show errdisable detect

4. show tech-support stackwise-virtual

5. Collect trace archive file

Possible Solution

1. Shut/ No Shut SVL LINKS: test platform software nif_mgr port disable / enable LPN

2. Physical OIR of the transceiver or try to change the transceiver or cable on SVL/DAD links.

about your question: "How can I turn this one back on? There are no "shutdown"/"no shutdown" commands at the SVL port."

How to Shut/Unshut SVL Port?

Shut and no shut comands are disabled on SVL and DAD ports when the system configured in StackWise-virtual mode. For test purposes, if there is a need to shut/unshut SVL ports, try as shown here:  

Switch#test platform software nif_mgr port ?
  disable  shutdown port
  enable   unshut port

Alternatively, use the below approach which can simulate SFP/QSFP OIR from software (e.g. on Cat9400/Cat9500H/Cat9600). These are hidden commands and requires 'service internal' to be configured:

Switch#test idprom interface <...> ?
  fake-insert  Fake insert
  fake-remove  Fake remove

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/216537-troubleshoot-svl-on-catalyst-9000-switch.html 

[balzsnacsa]

Hi,

the err-disabled port is Fo1/6/0/24. 

Is the command?

switch#test platform software nif_mgr port disable 6 24 ?

switch#test platform software nif_mgr port enable 6 24 ?

But, switch 1, Fo1/6/0/24  and switch 2, Fo2/6/0/24 are the same. How does the switch know which port to handle?

Explanation:

switch 1 slot 6 port 24 LPN=24:

switch#sh platform software fed sw 1 ifm mappings
Interface                    IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
FortyGigabitEthernet1/6/0/24 0x173  5    2    1    19   0      16   12   24 504  NIF  Y

switch#test platform software nif_mgr port enable ?

  <0-10>  Slot

switch#test platform software nif_mgr port enable 6 ?

  <1-96>  LPN to toggle

switch#test platform software nif_mgr port enable 6 24


but switch 2 slot 6 port 24 LPN=24:

switch#sh platform software fed sw 1 ifm mappings
Interface                    IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
FortyGigabitEthernet2/6/0/24 0xb0   5    2    1    19   0      16   12   24 1080 NIF  Y


switch#test platform software nif_mgr port enable 6 24

[marce1000]

 - Check logs for this particular condition and or event too. Particular log entries may point to the cause and possible error-recovery (post log entries here).

 M.

[balzsnacsa]

Yep, I saw it: the link has flapped. I changed QSFP but the err-disabled status has not vanished.

091254: Oct 10 20:24:21.171 CET: %NIF_MGR-6-PORT_LINK_DOWN: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 is DOWN.
091255: Oct 10 20:24:21.171 CET: %NIF_MGR-6-PORT_CONN_DISCONNECTED: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 connection has DISCONNECTED: CONN_ERR_PORT_LINK_DOWN_EVENT
091256: Oct 10 20:24:21.322 CET: %NIF_MGR-6-PORT_LINK_UP: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 is UP.
091257: Oct 10 20:24:21.322 CET: %NIF_MGR-6-PORT_CONN_PENDING: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 connection is PENDING.
091258: Oct 10 20:24:22.455 CET: %NIF_MGR-6-PORT_LINK_DOWN: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 is DOWN.
091259: Oct 10 20:24:22.455 CET: %NIF_MGR-6-PORT_CONN_DISCONNECTED: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 connection has DISCONNECTED: CONN_ERR_PORT_LINK_DOWN_EVENT
091260: Oct 10 20:24:22.606 CET: %NIF_MGR-6-PORT_LINK_UP: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 is UP.
091261: Oct 10 20:24:22.606 CET: %NIF_MGR-6-PORT_CONN_PENDING: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 connection is PENDING.
091262: Oct 10 20:24:26.366 CET: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Chassis 2 R0/0: nif_mgr: PORT:FortyGigabitEthernet Port 24 on Slot 6 changed state from Pending to Ready.
091263: Oct 10 20:24:26.366 CET: %NIF_MGR-6-PORT_CONN_READY: Chassis 2 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 connection is READY.
091264: Oct 10 20:24:26.488 CET: %NIF_MGR-6-PORT_LINK_DOWN: Chassis 2 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 is DOWN.
091265: Oct 10 20:24:26.488 CET: %NIF_MGR-6-PORT_CONN_DISCONNECTED: Chassis 2 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 connection has DISCONNECTED: CONN_ERR_PORT_LINK_DOWN_EVENT
091266: Oct 10 20:24:26.614 CET: %NIF_MGR-6-PORT_LINK_UP: Chassis 2 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 is UP.
091267: Oct 10 20:24:26.614 CET: %NIF_MGR-6-PORT_CONN_PENDING: Chassis 2 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 connection is PENDING.
091268: Oct 10 20:24:27.483 CET: %NIF_MGR-6-PORT_LINK_DOWN: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 is DOWN.
091269: Oct 10 20:24:27.483 CET: %NIF_MGR-6-PORT_CONN_DISCONNECTED: Chassis 1 R0/0: nif_mgr: FortyGigabitEthernet Port 24 on Slot 6 on front side stack link 0 connection has DISCONNECTED: CONN_ERR_PORT_LINK_DOWN_EVENT

091289: Oct 10 20:24:32.025 CET: %PM-4-ERR_DISABLE: link-flap error detected on Fo1/6/0/24, putting Fo1/6/0/24 in err-disable state

FortyGigabitEthernet1/6/0/24 is down, line protocol is down (err-disabled)
Hardware is Forty Gigabit Ethernet, address is 8c94.1f87.9017 (bia 8c94.1f87.9017)
Description: #### StackWise SVL link ####
MTU 1500 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
reliability 253/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 40Gb/s, link type is force-up, media type is QSFP 40GE CSR-S
input flow-control is on, output flow-control is off
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
31836011220 packets input, 21455291690725 bytes, 0 no buffer
Received 6383895 broadcasts (6383895 multicasts)
0 runts, 0 giants, 0 throttles
4564 input errors, 4069 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 6383895 multicast, 0 pause input
0 input packets with dribble condition detected
12329999830 packets output, 13685129763646 bytes, 0 underruns
Output 0 broadcasts (0 multicasts)
0 output errors, 0 collisions, 5 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[marce1000]

      - FYI :  https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx23125

 M.