Cisco 3650 - IOS 16.12.05b.SPA with Packet Loss

[simoesmarco8626982]

HI all!

hope to find everyone well

Yesterday had a weird one, was in an installation trying to troubleshoot packet loss and discovered that the source was a Cisco 3650 in one of the buildings. The cisco was forwarding packets in three directions, one of them being copper and the other two being optical. Found that the packets being sent by copper and then via radio to the other cisco on the other side were being received perfectly by the recorder (this is a CCTV system). But the same packets being sent by optical connected to radio would reach the recorder on the other side incorrectly with packets being lost. I checked all the switches and none of the switches were detecting packets being lost and all my counters were in 0.  

The logical explanation would be the radios are dropping packets, but these are 80Ghz radios in a licensed frequency, and upon checking all of them were perfect and working as intended. 

In the end, already desperate, decided to update the IOS from this switch from the version 16.03.06.SPA to 16.12.05b.SPA, and amazingly all the drops stopped. 

I was checking the bug tool and couldn't see anything that could explain this behavior. But my guess is, something in this IOS was not playing along with this Cisco compatible SFPs, but the incredible is nothing was being reported as being wrong.

Do you agree with my hipotisis? Any idea please?

Thank you 

[balaji.bandi · Accepted solution]

With the explanation, we do see some bugs in version 16.3.X and suggested others to upgrade. (they were fixed ) no reason.

every setup is different, but you made a good forward step to upgade to check, that fixed the issue. (this is really proactive step), than investinng and wasting time. Sometime or other users may not have same scenario as you, so they might have not reported. or some partner working with cisco may seen this, may be if you would have open TAC case, they may refer related to bug, end they sugges to upgrade stable Code to fix the issue.

good work !

[simoesmarco8626982]

Thank you Balaji!

Need to write a little report on this and will implement this.

But it's really difficult when the switches say all is good and in fact it isn't... Thankfully all is fixed now and the network is stable 

[balaji.bandi]

Appriciated your input, this help any of the community memember have same issue can resolve with the solution. +5 for you.

[Reza Sharifi]

Hi,

During the event, and before the upgrade, did you see anything in the logs indicating a problem with the SFP or the fiber?

Are all the other working switches running 16.03.06.SPA?

Thanks!

[simoesmarco8626982]

Hi Reza,

Thank you for the reply.

I checked the sfps and they were all working perfectly. The signal level was strong and I was having like 3 to 4db drop what is consistent with the splices and connectors I had in the installation.

Checked the logs and all was good as well.

The other switches didn't had this version and the entire installation was a mix of Denali, Gilbraltar releases, and some older ones, but this release specific, only this offending switch had it and it was the one giving issues.

Now, I've updated all the switches to the release that fixed my issues.