Ports on my 3650 Switch in a stack not working.

[philip.ilori1]

I recently upgraded about 8 stacks in my network with no issues. However, one stack is giving a lot of headaches. Port 1 - 35 of Switch 3 in the stack refused to work. There's link light, but the show interface output shows down/down. However, ports 36-48 work fine. I have replaced the switch with a tested switch, and replaced the stack cables. The issue still remains.

[Mark Malone]

if you roll back as a test are they working to rule out the software ?

seems suspect issue is still there after swapping in new switch too , mught be triggering something in software  

Is there anything in logs to indicate hardware failure

[philip.ilori1]

I have not attempted rolling back since the other 7 stacks work fine with the same Code. There is nothing in the logs to indicate hardware failure. I only got to know about this when a user complained of connectivity issue.

[Mark Malone]

Im just wondering with that question did you trigger some strange bug , the fact a new switch was inserted indicates it wasnt hardware but maybe something else as the issue followed the replacement , the only significant change was the upgrade which was to introduce new software , the switch is only slightly operating with a few ports what happens when you remove it from the stack do all ports work when standalone , the next step personally for me would be roll up or back and see if the software was the cause especially when nothing indicates anything unusual in logs, even though all other switches are good the bug could have been triggered by some type of traffic in the switch itself that may be independant to the clients attached or any application , it would be very coincidental for 2 switches to experience identical hardwrae problems like that in a network at the same stage

[philip.ilori1]

All ports on both switched worked when in standalone mode. I could roll back to the old code and see what happens. My worry is that why is it only on member 3 (on 2 different switches)?

[Julio E. Moisa]

Hi 

What you see executing show switch?

[philip.ilori1]

sh switch
Switch/Stack Mac Address : 881d.fcb0.8700 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 881d.fcb0.8700 15 V01 Ready
2 Member a0ec.f9ee.6680 1 V01 Ready
3 Member 3c08.f6fd.3500 2 V01 Ready
4 Member a0ec.f9ee.8680 1 V01 Ready
5 Member a0ec.f9ee.2d80 1 V01 Ready
6 Standby a0ec.f9cf.f900 1 V01 Ready
7 Member a0ec.f9ee.6580 14 V01 Ready

[Leo Laohoo]

Post the complete output to the command "sh post".

[philip.ilori1]

sh post
Stored system POST messages:

Switch 1
---------
POST: MBIST Tests : Begin
POST: MBIST Tests : End, Status Passed

POST: CRYPTO Tests : Begin
POST: CRYPTO Tests : End, Status Passed

POST: PHY Loopback: loopback Test : Begin
POST: PHY Loopback: loopback Test : End, Status Passed

POST: Inline Power Controller Tests : Begin
POST: Inline Power Controller Tests : End, Status Passed

POST: Thermal, Temperature Tests : Begin
POST: Thermal, Temperature Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: SIF Tests : Begin
POST: SIF Tests : End, Status Passed

Switch 2
---------
POST: MBIST Tests : Begin
POST: MBIST Tests : End, Status Passed

POST: CRYPTO Tests : Begin
POST: CRYPTO Tests : End, Status Passed

POST: PHY Loopback: loopback Test : Begin
POST: PHY Loopback: loopback Test : End, Status Passed

POST: Thermal, Temperature Tests : Begin
POST: Thermal, Temperature Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: SIF Tests : Begin
POST: SIF Tests : End, Status Passed

Switch 3
---------
POST: MBIST Tests : Begin
POST: MBIST Tests : End, Status Passed

POST: CRYPTO Tests : Begin
POST: CRYPTO Tests : End, Status Passed

POST: PHY Loopback: loopback Test : Begin
POST: PHY Loopback: loopback Test : End, Status Passed

POST: Thermal, Temperature Tests : Begin
POST: Thermal, Temperature Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: SIF Tests : Begin
POST: SIF Tests : End, Status Passed

Switch 4
---------
POST: MBIST Tests : Begin
POST: MBIST Tests : End, Status Passed

POST: CRYPTO Tests : Begin
POST: CRYPTO Tests : End, Status Passed

POST: PHY Loopback: loopback Test : Begin
POST: PHY Loopback: loopback Test : End, Status Passed

POST: Thermal, Temperature Tests : Begin
POST: Thermal, Temperature Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: SIF Tests : Begin
POST: SIF Tests : End, Status Passed

Switch 5
---------
POST: MBIST Tests : Begin
POST: MBIST Tests : End, Status Passed

POST: CRYPTO Tests : Begin
POST: CRYPTO Tests : End, Status Passed

POST: PHY Loopback: loopback Test : Begin
POST: PHY Loopback: loopback Test : End, Status Passed

POST: Thermal, Temperature Tests : Begin
POST: Thermal, Temperature Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: SIF Tests : Begin
POST: SIF Tests : End, Status Passed

Switch 6
---------
POST: MBIST Tests : Begin
POST: MBIST Tests : End, Status Passed

POST: CRYPTO Tests : Begin
POST: CRYPTO Tests : End, Status Passed

POST: PHY Loopback: loopback Test : Begin
POST: PHY Loopback: loopback Test : End, Status Passed

POST: Thermal, Temperature Tests : Begin
POST: Thermal, Temperature Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: SIF Tests : Begin
POST: SIF Tests : End, Status Passed

Switch 7
---------
POST: MBIST Tests : Begin
POST: MBIST Tests : End, Status Passed

POST: CRYPTO Tests : Begin
POST: CRYPTO Tests : End, Status Passed

POST: PHY Loopback: loopback Test : Begin
POST: PHY Loopback: loopback Test : End, Status Passed

POST: Thermal, Temperature Tests : Begin
POST: Thermal, Temperature Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: SIF Tests : Begin
POST: SIF Tests : End, Status Passed

[Julio E. Moisa · Accepted solution]

What is the faulty switch?

*1 Active 881d.fcb0.8700 15 V01 Ready 
2 Member a0ec.f9ee.6680 1 V01 Ready 
3 Member 3c08.f6fd.3500 2 V01 Ready 
4 Member a0ec.f9ee.8680 1 V01 Ready 
5 Member a0ec.f9ee.2d80 1 V01 Ready 
6 Standby a0ec.f9cf.f900 1 V01 Ready  
7 Member a0ec.f9ee.6580 14 V01 Ready  <--- it should be the standby 

[philip.ilori1]

Thanks for pointing that out. I will put in a change control to reload the stack.

[Leo Laohoo]

The output to the "sh post" command looks fine. No need to reload the stack, just reload the switch member having the problem.

What IOS version is the stack running on?

[philip.ilori1]

The stack is running version 3.6.6, and yes I reloaded the slot already.

[Leo Laohoo]

If the slot has been reloaded, has the problem re-occured?

[philip.ilori1]

Yes. I reloaded the switch, but the problem is still there. I will reload the entire stack tomorrow morning, and update on the outcome.