Cisco 2960-X 15.2(7)E2 Connectivity Issue

[SK41570]

Hello,

Following an upgrade to IOS 15.2(7)E2 we have experienced connectivity issues for some devices connected to one of our VLANs. We have a port channel configured with 2 x 10Gb ports using twinax cables. Connectivity is only affected on 1 interface in the port channel. I have tested taking down 1 interface at a time and it appears that only 1 interface works correctly. What's peculiar is that we have a few other VLANs which are trunked down on this port channel and they work on both interfaces with no connectivity issues. The configuration on the interfaces are identical and both get bundled into the port channel when up. Spanning tree is showing that all VLANs, Ports are forwarding correctly. I haven't seen any error messages when debugged spanning tree or the ether-channel. 

We have a simple network topology - access switch stack 2960-x with a port channel to a core 3850 switch. There are no loops in the network. This issue only started after the upgrade to IOS 15.2(7)E2. No other configuration changes had been made. 

Any suggestions or ideas on what could be causing this issue would be appreciated. 

Thank you.

[balaji.bandi]

Not that we aware of any bugs, but as you mentioned only occured after upgrade.

have you tried changing to different port in different Asic. (both the sides ? see that improves any )

[SK41570]

Hi Balaji, I have been troubleshooting remotely at the moment so I have not been able to test changing the ports / ASIC. I might need to schedule a site visit if I'm not able to resolve the issue remotely.  

[Georg Pauwen]

Hello,

what is the output of:

show etherchannel load-balance

?

[SK41570]

Hi Georg,

Same output for both sides:

EtherChannel Load-Balancing Configuration:
src-mac

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
IPv4: Source MAC address
IPv6: Source MAC address

[Georg Pauwen]

Hello,

try src-dst-ip:

Switch(config)#port-channel load-balance { src-mac | dst-mac | src-dst-mac | src-ip | dst-ip | src-dst-ip | src-port | dst-port | src-dst-port }

[Leo Laohoo]

Wait ... the issue is the link between the 2960X and a 3850?

[SK41570]

Hi Leo, yes appears to be the port channel between the 2960X and 3850 switches.

[Leo Laohoo]

Let me see if I can sum this up properly: 

1. Two ports, in an etherchannel, from the 2960X to the 3850
2. One port is working fine
3. The other port do not pass traffic but the link is "up/up".  

Is this accurate?  

[SK41570]

HI Leo, please see my answers below. Hopefully that helps clear up the issue:

1. Two ports, in an etherchannel, from the 2960X to the 3850 - Yes 2 ports in configured in a port channel using LACP. 
2. One port is working fine - Yes when the faulty interface in the port channel is disabled all traffic to all VLANs works correctly. 
3. The other port do not pass traffic but the link is "up/up".  The other port is up/up and does pass traffic. It appears to be only 1 VLAN which it does not pass traffic correctly.  

[Leo Laohoo]

Have you tried COLD reboot (pull the power cable completely)?

[SK41570]

No I am troubleshooting remotely at the moment. I can schedule some downtime and be on site to give it a reboot but it would be nice to know what's causing it. 

[Leo Laohoo]

Remember, COLD REBOOT:  The power cable needs to be pulled. 

There are two bugs exclusive to the 3650/3850:  CSCvj76259 and CSCvd46008

Bug IDs might say that the ports stop delivering PoE but it's half correct.  When the MOSFET "bug" hits, the port(s) stop passing traffic completely.  Layer 1/2 might show "up/up" but that's about it.  

To confirm MOSFET, pull the power cable.  If COLD reboot fixes, then it is MOSFET.  If this is the case, do not waste any more time and RMA the switch.  COLD reboot is not a permanent fix -- it is a workaround.  This means there is no guarantee the issue will not come back to other port(s).  

The only guaranteed "fix" is to RMA the switch.  

NOTE:  I hope I am wrong.

[SK41570]

Thank you Leo, I will have a read through them bug articles.