C9300 UDLD Err-Disable After Stack Replacement

[Guardian452]

I recently replaced a stack of switches with C9300s and encountered an interesting UDLD issue. After reviewing with seniors on my team, it seems it may have had something to do with the order of operations.

Switches were uplinked via two 10G SR optics and OM3 MM fiber to another stack of C9300s. Both stacks running IOS-XE 17.12.06. The stack had the following configuration: 

#show switch

Switch/Stack Mac Address : f8a7.3aba.fb80 - Local Mac Address

Mac persistency wait time: Indefinite

                                             H/W   Current

Switch#   Role    Mac Address     Priority Version  State

-------------------------------------------------------------------------------------

*1       Active   f8a7.3aba.fb80     15     V04     Ready

 2       Standby  6887.c697.3f80     13     V05     Ready

 3       Member   14a2.a034.cd00     11     V05     Ready

Uplinks were on Te1/1/1 and Te2/1/1

interface TenGigabitEthernet1/1/1

 description po1

 switchport trunk native vlan 999

 switchport trunk allowed vlan 4-6,11,14,18,63,64,220,263,457,611,725,862

 switchport mode trunk

 switchport nonegotiate

 ip arp inspection trust

 ip dhcp snooping trust

interface TenGigabitEthernet2/1/1

 description po1

 switchport trunk native vlan 999

 switchport trunk allowed vlan 4-6,11,14,18,63,64,220,263,457,611,725,862

 switchport mode trunk

 switchport nonegotiate

 ip arp inspection trust

 ip dhcp snooping trust

These links were in a port-channel on the old switch and the new switch, though they were removed from the port-channel for troubleshooting when we noticed the err-disabled state.

I started by stacking all of the switches, powering on switch 1, and moving client ports over from the old stack. After 10-15 minutes, I powered up the second switch. At some point during this procedure, I moved the 10G uplink optic over to switch 1, and switch 2, but I can’t remember exactly when. I see that the uplinks are good, so I keep moving client ports over and eventually power up switch 3.

I notice that UDLD err-disabled alarms on the upstream links begin appearing. We have err-disable recovery for UDLD on, so this was alarming every 5 minutes. 

I swapped optics to 10G LR and moved to different strands to rule out any L1 issues. At this point, we were still getting UDLD err-disabled on the uplinks every 5 minutes.

We begin to suspect something in the configuration, so my colleague wipes the switch and rebuilds from our template. After the reload, the uplinks come up and stay up.

During review, there were only a couple of console and vty line deviations from our template.

UDLD configuration on the replaced switch stack and upstream match in both the b

show run | i udld
udld aggressive

errdisable recovery cause udld

What could have caused the uplinks to throw UDLD? Was this an order of operations issue? One of the seniors on my team said it can cause control plane issues if all switches in the stack are powered up at separate times. Is this true? I’ve checked the bug tracker and didn’t see anything similar for this release/hardware combination.

[Mark Elsen · Accepted solution]

  - @Guardian452           Review these bug reports : https://bst.cisco.com/bugsearch/bug/CSCvo46352?rfs=qvred
                                                                                https://bst.cisco.com/bugsearch/bug/CSCwd64709?rfs=qvred
                                                                                https://bst.cisco.com/bugsearch/bug/CSCwj27430?rfs=qvred
                                                                                https://bst.cisco.com/bugsearch/bug/CSCwd01674

  M.

[Guardian452]

Thanks Mark. 

I did review CSCvo46352, but had some doubts if this was the cause.

Conditions for CSCwd01674 were not met; no switch was removed from the new (replacement) stack. Restarting the whole stack resolved the issue, so this tracks. 

Bingo! Conditions are an exact match for CSCwd64709, though the release is not an exact match. 

I'll cast my net a little wider next time when searching bug reports. 

[balaji.bandi]

I notice that UDLD err-disabled alarms on the upstream links begin appearing. We have err-disable recovery for UDLD on, so this was alarming every 5 minutes. 

This can also occur when the ports are looped, when you console the switch, what other logs you see - do you see any Ports going up and down due to BPDU ?

These links were in a port-channel on the old switch and the new switch, though they were removed from the port-channel for troubleshooting when we noticed the err-disabled state.

both the interface shows trunks config, how they connected to uplink to same switch on other side ?

what happens when you remove one of the uplink is the UDLD occurs ?

you need to write some diagram, i am sure this is either Loops in the STP or some BPDU causing the issue based on the information.

We begin to suspect something in the configuration, so my colleague wipes the switch and rebuilds from our template. After the reload, the uplinks come up and stay up.

is this ok now ? or you still have issue with UDLD, since its resolve hard to say what may have caused the issue, my assumption as above. 

[Guardian452]

Reviewing logs it was only UDLD errors. No BPDUs observed. 

Currently only Te1/1/1 is connected to the other side. This wasn't clear from my output above. They were previously port-channeled. 

This is OK now. No issue with UDLD as  CSCwd64709 shows, the issue did not occur when we restarted the whole stack.