4500x VSS Stack SFP's don't link ports

Jim Matuska · ‎08-03-2018

We have 2 WS-C4500X-32 all SPF+ port switches in a VSS stack. This stack is running version cat4500e-universalk9.SPA.03.10.01.E.152-6.E1 code with an Enterprise Services license.

I have been having issues with getting various random ports on both switches in the stack. I can install an SFP module (doesn't matter whether it is a 1GbitBaseT SFP, 1Gbit LH SMD, 1Gbit MMD module, 10 Gbit LR or even older non DOM enabled SFP modules). The switch picks up the SFP module just fine, but when I try to link into another switch the port just stays in a down/down state.

If I switch to another port (using the Same SFP module) it works just fine with the same configuration.

I am only using genuine Cisco SFP and SFP+ modules. I first thought it might be some bad ports on one switch in the stack but I recently started seeing the same issue on the other switch too.

The other strange thing is that if I do a show interface transceiver command I can see adequate light levels from the SFP modules but it does not link.

Has anyone else run into this problem?

andresfr · ‎08-03-2018

Hi npt,

I hope you're doing great.

* When connecting the 4500X-32 VSS to another switch, are you bundling the interfaces in a port-channel? Or you're just connecting one port on one of the VSS chassis to the other switch?

* Have you check if the other switch is showing any errors in the logs?

Could you recreate the failure and provide the output of the following commands? Collect from both the VSS and the other switch:

show run interface <interface_type/id>

show interface status

show interface status err-disable

show spanning-tree interface <interface_type/id>

show logg

---

!If using etherchannel

show etherchannel summary

Regards

Jim Matuska · ‎08-03-2018

* When connecting the 4500X-32 VSS to another switch, are you bundling the interfaces in a port-channel? Or you're just connecting one port on one of the VSS chassis to the other switch?

The interfaces are in a port channel for VSS and it is showing both switches operational with switch 1 as the active Virtual Switch and Switch 2 as the standby virtual switch. They are connected with 2 Direct Attached 10 Gbit Cables.

* Have you check if the other switch is showing any errors in the logs?

How would I check the other switch for errors? In VSS I only have one configuration and VTY console for both switches and when I look at the log it doesn't show any errors. I unplugged one of my fiber cables in an etherchannel bundle from port tengig 1/1/8 that was fully functional and plugged into port tengig 1/1/7 and the only thing in the log is that the interface 1/1/8 went down and left the port channel. Nothing is listed in the logs about the port 1/1/7 that I have the fiber cable plugged into. It doesn't matter if I don't use etherchannel for another link, the same thing happens. I just don't get any link. Right now I have found about 4 ports on the second switch and one port on the first switch with this issue, but there could be more.

Here is the other strange thing. The output of show int transceiver shows good light on the SFP module but no link.

4500x-CORE#show int ten 1/1/7 transceiver
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
If device is externally calibrated, only calibrated values are printed.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
NA or N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).

                                 Optical   Optical
           Temperature Voltage Tx Power Rx Power
Port       (Celsius)    (Volts) (dBm)     (dBm)
--------- ----------- ------- -------- --------
Te1/1/7      29.0       3.32      -5.5      -8.4

show run interface <interface_type/id>

interface TenGigabitEthernet1/1/7
description tengig 1/1/7 Possible Bad Port
switchport trunk allowed vlan (Removed)
switchport trunk native vlan (Removed)
switchport mode trunk
switchport nonegotiate
channel-group 201 mode on
end

show interface status

4500x-CORE#show int status (SENSATIVE Data Removed)

Port      Name               Status       Vlan       Duplex Speed Type
Te1/1/1   TenGig 1/1/1    connected    trunk        full a-1000 1000BaseT
Te1/1/2   TenGig-1/1/2    connected    trunk        full a-1000 1000BaseLH
Te1/1/3   TenGig 1/1/3    connected ****    full a-10G 10GBase-LR
Te1/1/4   TenGig 1/1/4 connected ****    full a-1000 1000BaseLH
Te1/1/5   TenGig 1/1/5    connected    ****    full a-100 100BaseFX
Te1/1/6   TenGig 1/1/6    connected    trunk        full a-100 100BaseFX
Te1/1/7   tengig 1/1/7 Possi notconnect   trunk        full   auto 1000BaseLH
Te1/1/8   TenGig 1/1/8 to La notconnect   trunk        full   auto 1000BaseLH
Te1/1/9                      notconnect   1            full   auto 1000BaseSX
Te1/1/10                     notconnect   1            full   auto No XCVR
Te1/1/11                     notconnect   1            full   auto No XCVR
Te1/1/12                     notconnect   1            full   auto No XCVR
Te1/1/13                     notconnect   1            full   auto No XCVR
Te1/1/14                     notconnect   1            full   auto No XCVR
Te1/1/15                     notconnect   1            full   auto No XCVR
Te1/1/16                     notconnect   1            full   auto No XCVR
Te1/1/17                     notconnect   1            full   auto No XCVR
Te1/1/18                     notconnect   1            full   auto No XCVR
Te1/1/19                     notconnect   1            full   auto No XCVR
Te1/1/20                     notconnect   1            full   auto No XCVR
Te1/1/21                     notconnect   1            full   auto No XCVR
Te1/1/22                     notconnect   1            full   auto No XCVR

Port      Name               Status       Vlan       Duplex Speed Type
Te1/1/23                     notconnect   1            full   auto No XCVR
Te1/1/24                     notconnect   1            full   auto No XCVR
Te1/1/25 TenGig 1/1/25    connected    trunk        full a-10G 10GBase-CU 7M
Te1/1/26 TenGig 1/1/26 connected    trunk        full a-10G 10GBase-LR
Te1/1/27 TenGig 1/1/27    connected    trunk        full a-10G 10GBase-CU 7M
Te1/1/28 TenGig 1/1/28 connected    routed       full a-1000 1000BaseT
Te1/1/29 TenGig 1/1/29 connected    trunk        full a-10G 10GBase-LR
Te1/1/30 TenGig 1/1/30 connected    trunk        full a-10G 10GBase-CU 3M
Te1/1/31 TenGig 1/1/31 VSS connected    trunk        full a-10G 10GBase-CU 1M
Te1/1/32 TenGig 1/1/32 VSS connected    trunk        full a-10G 10GBase-CU 1M
Te2/1/1   possible bad port notconnect   **** full   auto No XCVR
Te2/1/2   TenGig 2/1/2    connected ****    full a-100 100BaseFX
Te2/1/3   possible bad port notconnect **** full   auto No XCVR
Te2/1/4   possible bad port notconnect **** full   auto No XCVR
Te2/1/5   TenGig 2/1/5 connected    trunk        full a-1000 1000BaseLH
Te2/1/6   temp    connected ****    full a-1000 1000BaseT
Te2/1/7    notconnect   1            full   auto No XCVR
Te2/1/8   TenGig 2/1/8 connected    trunk        full a-1000 1000BaseLH
Te2/1/9                      notconnect   1            full   auto 1000BaseLH
Te2/1/10                     notconnect   1            full   auto No XCVR
Te2/1/11                     notconnect   1            full   auto No XCVR
Te2/1/12                     notconnect   1            full   auto No XCVR
Te2/1/13                     notconnect   1            full   auto No XCVR
Te2/1/14                     notconnect   1            full   auto No XCVR

Port      Name               Status       Vlan       Duplex Speed Type
Te2/1/15                     notconnect   1            full   auto No XCVR
Te2/1/16                     notconnect   1            full   auto No XCVR
Te2/1/17                     notconnect   1            full   auto No XCVR
Te2/1/18                     notconnect   1            full   auto No XCVR
Te2/1/19                     notconnect   1            full   auto No XCVR
Te2/1/20                     notconnect   1            full   auto No XCVR
Te2/1/21                     notconnect   1            full   auto No XCVR
Te2/1/22                     notconnect   1            full   auto No XCVR
Te2/1/23                     notconnect   1            full   auto No XCVR
Te2/1/24                     notconnect   1            full   auto No XCVR
Te2/1/25 TenGig 1/1/25 connected    trunk        full a-10G 10GBase-CU 7M
Te2/1/26 TenGig 2/1/26    notconnect   trunk        full   auto 10GBase-LR
Te2/1/27 TenGig 2/1/27    connected    trunk        full a-10G 10GBase-CU 7M
Te2/1/28                     notconnect   1            full   auto No XCVR
Te2/1/29 TenGig 2/1/29 connected    trunk        full a-10G 10GBase-LR
Te2/1/30 TenGig 2/1/30    connected    trunk        full a-10G 10GBase-CU 3M
Te2/1/31 TenGig 2/1/31 VSS connected    trunk        full a-10G 10GBase-CU 1M
Te2/1/32 TenGig 2/1/32 VSS connected    trunk        full a-10G 10GBase-CU 1M
Po11      Port-Channel 11 VS connected    trunk      a-full a-10G
Po12      Port-Channel 12 VS connected    trunk      a-full a-10G
Po13      Port-Channel 13 connected    trunk      a-full a-10G
Po200     Port-Channel 200 connected    trunk      a-full a-10G
Po201     Port-Channel 201 connected    trunk      a-full a-1000

show interface status err-disable

No output, no ports error disabled

show spanning-tree interface <interface_type/id>

4500x-CORE#show spanning-tree int tengig 1/1/7
no spanning tree info available for TenGigabitEthernet1/1/7

show log (Only showing log from when switching ports (not seeing any errors in log file)

Aug 3 14:34:22: %EC-5-UNBUNDLE: Interface TenGigabitEthernet1/1/8 left the port
-channel Port-channel201
Aug 3 14:34:22: %EC-5-UNBUNDLE: STANDBY:Interface TenGigabitEthernet1/1/8 left
the port-channel Port-channel201
Aug 3 14:34:38: %EC-5-BUNDLE: Interface TenGigabitEthernet1/1/8 joined port-cha
nnel Port-channel201
Aug 3 14:34:38: %EC-5-BUNDLE: STANDBY:Interface TenGigabitEthernet1/1/8 joined
port-channel Port-channel201
Aug 3 16:00:29: %EC-5-UNBUNDLE: Interface TenGigabitEthernet1/1/8 left the port
-channel Port-channel201
Aug 3 16:00:29: %EC-5-UNBUNDLE: STANDBY:Interface TenGigabitEthernet1/1/8 left
the port-channel Port-channel201

---

!If using etherchannel

show etherchannel summary

Number of channel-groups in use: 5
Number of aggregators: 5

Group Port-channel Protocol    Ports
------+-------------+-----------+-----------------------------------------------
11     Po11(SU)         -        Te1/1/31(P) Te1/1/32(P)
12     Po12(SU)         -        Te2/1/31(P) Te2/1/32(P)
13     Po13(SU)         -        Te1/1/30(P) Te2/1/30(P)
200    Po200(SU)        -        Te1/1/26(P) Te2/1/26(D)
201    Po201(SU)        -        Te1/1/7(D) Te1/1/8(D) Te2/1/8(P)

4500x-CORE#

Let me know what you think.

Leo Laohoo · ‎08-03-2018

Have you tried to shut/no shut the port?

Jim Matuska · ‎08-03-2018

Yes, I did try a shut then no shut on the interface. Tried that multiple times with no change. The strange thing though is even though I see the physical port turn orange when disabled, no interface changes show in the log on the switch stack.

Leo Laohoo · ‎08-03-2018

@Jim Matuska wrote:

no interface changes show in the log on the switch stack.

That's because 4K & 6K have logging of port changes is disabled by default.

@Jim Matuska wrote:

If I switch to another port (using the Same SFP module) it works just fine with the same configuration.

Are the ports in the same line card or different slots?

Jim Matuska · ‎08-03-2018

2 Separate 4500x 1U Chassis stacked together in VSS with dual 10 Gbit Direct Attach cables. Port issues are on both switches.

What is the configuration to enable logging on port changes? Maybe that will give me some errors when I try to link into one of those ports.

Jim

Leo Laohoo · ‎08-03-2018

@Jim Matuska wrote:

What is the configuration to enable logging on port changes?

Command: logging event link-status

@Jim Matuska wrote:

201 Po201(SU) - Te1/1/7(D) Te1/1/8(D) Te2/1/8(P)

Can you try port Te 1/1/13 and/or Ten 1/1/24?

pieterh · ‎08-06-2018

you mention
- various random ports on both switches in the stack
- If I switch to another port (using the Same SFP module) it works just fine with the same configuration.

can you relate the port to a specific ASIC?

frpm: https://ciscointerworking.wordpress.com/2015/06/14/asic-redundancy/

Command: “show platform mapping ports“
Front panel ports: “Interface“
ASIC mapping: “PortSet“

Jim Matuska · ‎08-06-2018

No, it doesn't look like it is ASIC specific as the 4500x has only 2 ports per ASIC and port TenGig 1/1/8 links in fine, but 1/1/7 doesn't and they are both on ASIC 19.

Also, on the second switch ports TenGig 2/1/1 won't link but 2/1/2 will (both on ASIC 15) and also ports TenGig 2/1/3 and 2/1/4 won't link but are on the same ASIC 17.

Jim Matuska · ‎08-06-2018

@Leo Laohoo wrote:

Can you try port Te 1/1/13 and/or Ten 1/1/24?

Ports Ten 1/1/13 and Ten 1/1/24 link in just fine.

Leo Laohoo · ‎08-06-2018

Ports Ten 1/1/7 to Ten 1/1/12 is not working but ports Ten 1/1/13 is working.
I suspect this is an IOS bug. I'd recommend you get a TAC case running.

Jim Matuska · ‎08-06-2018

Actually so far I have only encountered 4 ports that won't link in, port TenGig 1/1/7 (on the first switch in the stack), and TenGig 2/1/1, 2/1/3, and 2/1/4 (on the second switch in the vss stack). So far I haven't run into any issues with the other ports on the switches. I am thinking possible software bug too at this point.

pieterh · ‎08-07-2018

Sorry entered by mistake, cannot remove only edit

fabricejuvet · ‎01-31-2020

Hello,

I currently have two stacks of 4500-X switches configured in VSS mode. When I try to connect them together via a port-channel, the interface of one of them "blocking" ! For information, the spanning-tree and MST mode. Do you have any idea what the problem is ? T

hank you in advance.