Cisco 3850 SFP Issues

[CSCO11550753]

Live Issue: I recently upgraded a Cisco 3850 Switch stack to 16.03.06 and started to see some odd behaviour. I had a trunk port that was configured on the newly upgraded switch with a number of VLAN's the trunk port was showing as up but no MAC addresses where showing in the MAC Table on that interface. Decided to shut/no shut the Port-Channel and we lost connectivity to the edge switch. When we removed the SFP module (Cisco) from the port on the core switch we could not see any light. Plugged the SFP back into the core switch and the edge ports come up/up but not the core ports.

When running the command show inv the SFP's where not showing as being connected.

Solution: Reboot the Cisco 3850 Core Switch. This then brought all the ports back up and all inserted SFP where re-discovered.

Future Problem: When wanting to add a new edge switch to the 3850 core we would need to make sure that the new switch was connected and all ports configured and then reboot the core switch so that the SFP would become visible and bring the port up. I dont want to have to reboot the switch every time a port goes down and the SFP disapears from the Core switch or when I want to add a new SFP to the system.

Has anyone else experienced this type of issue before?

[Mark Malone]

Hi
yes thats not normal behavior , is that happening each time or was it a once off , what 3850 platform are you on exactly ?

Did the logs show anything unusual before reload ,?
did you collect the show tech before the reload ?

[CSCO11550753]

WS-C3850-24S switch stack with stacking modules.

IOS:16.03.06

Didnt take a show tech but the logs seemed fine and yes it happens each time a port goes down. Something I have never seen before. Cant upload any output due to the nature of the network.

Another strange thing is we a Juniper firewall that was connected to the core switch via. fibre and now its down/down we didnt even touch the firewall so I think its pointing towards the cores behaviour.

My worry is that the place has scheduled power outages and if we take a port on the core off line we may need to reboot the whole core switch again.

Cheers

[Mark Malone]

You could replicate issues grab the show tech and log it with TAC incase its hardware issue but you could also try get off that software version first and try stabilize it if you don't have TAC support , it may be some nasty bug causing it that a fresh software version may fix

there's a newer edition MD release ahead of 3.6 ....

CAT3850/3650 UNIVERSAL
cat3k_caa-universalk9.16.03.07.SPA.bin

https://software.cisco.com/download/home/286257125/type/282046477/release/Denali-16.3.7

[CSCO11550753]

I currently have a colleague setting up the same kind of hardware environment to try and replicate the issue on IOS .06. If it has the same issue he is going to upgrade to .07 to see if its the IOS. If not we are going to have to try and log a TAC case. Its quite hard to do as the nature of the network it needs to be kept in the UK.

I think we may have to bump the hardware to .07 though as without that I think TAC will just advise that anyway.

[Mark Malone]

Yes logically that will be there process but they may see something in the show tech too that could relate it to hardware or a faulty board, they may also provide exact bug id if software related s o you know upgardingt is the path as sometime si have had to downgrade as the issue was not yet fixed in latest release and was not relevant in earlier versions

the problem with trying to replicate it on another set of switches is if its a bug you dont know yet what the trigger was that set it off , it could be anything from a specific process hanging to an sfp version number they dont like

hopefully upgrade fixes it though and its not an RMA , you could also run the show tech your self through the Cisco CLI analyzer tool , if its a known bug it will tell you

[mdieken011]

Did you happen to come across a resolution to this issue?  I have pretty much the same issue on one of my closet switches.  I went from 3.6.3 to 16.3.6  to try to solve it but after a few months it is happening again.  I appreciate any help on this.

[CSCO11550753]

We eventually upgraded to the 06.03.08 IOS version and this seemed to have fixed the issue. It was only implemented a couple of weeks ago but it seemed to have been stable since then. If I remember the fix at the time before we upgraded was to make sure all of your SFP's are installed and then reload. It recognises the SFP's as part of the POST etc.... and then works as expected. Not ideal but seemed to have worked. Hope this helps.

[mdieken011]

What concerns me is I did an upgrade almost a year ago to solve this exact issue.  They have been running fine for 9 months and it just happens.  I will try the reboot fix but I have to keep an eye on them.