C9500 Stackwise virtual : Down !!

[Jaki74]

Hi Folks,

We have a Pair of C9500-16X on version 16.9.1.

They are located in different rooms connected through multimode fibre with SFP-10GBase-SR.

They are configured with stackwise-virtual on ports Te1/0/15, Te1/0/16 and  Te2/0/15, Te2/0/16.

We had a known cabling issue on port Te1/0/16 what caused this port to get err-disabled.

Now after several weeks the other Port Te1/0/15 got err-disabled as well due to link-flaps

Mar 15 04:25:45.828 CET: %NIF_MGR-6-PORT_CONN_DISCONNECTED: Switch 1 R0/0: nif_mgr: Port 15 on front side stack link 0 connection has DISCONNECTED: CONN_ERR_PORT_LINK_DOWN_EVENT
Mar 15 04:25:45.828 CET: %NIF_MGR-6-STACK_LINK_DOWN: Switch 1 R0/0: nif_mgr: Front side stack link 0 is DOWN.
Mar 15 04:25:45.828 CET: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 1 on Switch 1 is down
Mar 15 04:25:46.830 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/15, changed state to down
Mar 15 04:25:47.830 CET: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/15, changed state to down
Mar 15 04:30:24.764 CET: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/15, changed state to up
Mar 15 04:30:25.480 CET: %PM-4-ERR_DISABLE: link-flap error detected on Te1/0/15, putting Te1/0/15 in err-disable state
Mar 15 04:30:27.136 CET: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/15, changed state to down

Thus only one core switch is part of the Stack.

I have issues getting the interface re-enabled!

Shut/no shut commands are not available on stackwise-virtual ports.

So I tried to remove the port from the stackwise virtual:

DKHOM1A01#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
DKHOM1A01(config)#interface TenGigabitEthernet1/0/15
DKHOM1A01(config-if)#no  stackwise-virtual link 1
WARNING: Stackwise Virtual Configuration for TenGigabitEthernet1/0/15 will be removed on reboot.
DKHOM1A01(config-if)#

But I cannot rebooting the only remaining core switch, this will bring the whole plant down ....

Any ideas how to solve the situation?

[Leo Laohoo]

Reboot both units will restore the VS.

[Jaki74]

Thanks Leo,
I opened this discussion to see if there is any other option to rebooting both units.

This is a manufacturing location and reboot of cores will bring the whole network down, which has a very high negative user and financial impact.

[Leo Laohoo]

The reason why I'm saying this is because I believed I've read this behaviour before in a bug notice.

[Jaki74]

We tried removing and re-inserting the uplink SFTP's but interfaces stay in err-disabled 

Mar 15 12:01:37.910 CET: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/0/16 removed
Mar 15 12:01:42.485 CET: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/0/16
Mar 15 12:01:51.350 CET: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/0/15 removed
Mar 15 12:01:56.823 CET: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/0/15

Te1/0/15  VSS Link           err-disabled 4094         full  10000 SFP-10GBase-SR
Te1/0/16  VSS Link           err-disabled 4094         full  10000 SFP-10GBase-SR

[balaji.bandi]

As suggested @Leo Laohoo reboot to restore the service back.

If the Services already running, Still you are at risk, service may have interuption, so do this in maintenance window.

Error-disable could be diffrent reasons, do you see any Physcal Cable issue, after checking cables and confirmed no issued.

shut the interface and no shut and see if the problem resolves.

[Ihshaz Deen]

Hi @Jaki74 ,

You cannot directly clear the err-disabled in the Stackwise Virtual Link Port with just shut/no shut command. It is disabled by default. Below is the way to clear the errdisable:

Initially will have to check whether the Link-flap is enabled in errdisable function by using "show errdisable recovery"

If its disabled, you will have to enable it using the "XX(config)#errdisable recovery cause link-flap"

(optional) by default the timer is set to 300s, you can set it to lesser number to recover quickly by "XX(config)#errdisable recovery interval 30"

Thats it.

As per the above config, after 30s the error disabled will be removed.

Note: Make sure you rectify the Cabling infrastructure, else you will face the issue again