Catalyst 9200 & 9300 - SFP+ - FCS errors under max load

[simoesmarco8626982]

Hi all,

Hope to find everyone well.

Hope you can help me, I have a production system with 9200 and 9300 working with Cisco Compatible 10Gbps SFP+ for the core of the network. The network is using OSPF and everything is working correctly. 

To test the throughput of the core to prove that in fact we could achieve 10Gbps and that everything was good in the entire ring we rented two network test units and executed the tests. This tests were executed without me but the people that executed the tests told me that something was wrong in the network but that they couldn't understand what. 

I went today to check the switches and found that the interfaces have in the counters thousands of input errors (FCS errors). And of 8 switches, 6 have this in the 10Gbps SFPs interfaces and the remaining two switches despite not staring FCS errors they have input errors (symbol error) in the counters. 

Now, it seems that if the system is not under full load everything is working fine and the counters don't increase but when they did the test the FCS errors started increasing. 

The signal levels of the fibers are all good.

I'm inclined for this to be something wrong with this specific SFPs but would like to hear your opinion please.

Thank you

[balaji.bandi]

First, i check what is the IOS XE code running, then I will upgrade to 17.3.5 and the latest and test again.

Can you also post :

show controllers ethernet-controller te x/x

show interfaces ten x/x counters errors

some bugs around for reference if that help you :

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9600-series-switches/217413-understand-why-fcs-errors-input-errors.html

[Leo Laohoo]

1. What firmware are the switches on? 
2. Which of the switches are showing FCS, the 9200 or the 9300?
3. Post the complete output to the command "sh controll e <SFP PORT>".

[simoesmarco8626982]

Thank you @Leo Laohoo @balaji.bandi  

Truly wasn't aware that this could even be a bug, and was putting my eggs on the Compatible SFPs since I had a few issues with them and the DOM. But I just got all the information and reads like this.

In the 9300 Modular with the 8x10Gbps SFP+ module I get the below in the two switches that I have: 

 

 I'm not having FCS errors but I'm getting SymbolErr Frames and for what I read SymbolErr is realted with L1 issues. The weird now is that the SFPs are all the same in all the switches. 

In the 9300L and 9200L I'm not getting the SymbolErrors but I'm getting FCS errors that fits with the bug posted above since this ones are multigigabit:

 The IOS version in all the 9300 and 9200 is the 17.03.04

Thank you

[Leo Laohoo]

Switch to different SFP ports on both switches.

[simoesmarco8626982]

Thank you all!

I have now updated all the switches to version 17.06.03 so hopefully this will work correctly now.

Regarding the symbol errors, I'm going to try first just in case with the new IOS, because I saw they are appearing in the ports connected to the 10Gbps Radios but not in the ports connected between switches so hopefully the new IOS will fix this as well. 

If that doesn't work then I'll swap ports.

Thank you all!

[balaji.bandi]

sure - we happy to listen the outcome,. so that will helpfull for the community.

[simoesmarco8626982]

Hi @balaji.bandi @Leo Laohoo , finally I can update this thread after several months since only now we could find the source of the issue. Even after updating the IOS the issues were still present. This switches connect to Siklu Radios and this siklus bridge the switches. 

Yesterday having a representant of Siklu on site he for the sake off decided on the radio side to disable the pause frame to be sent from the radios to the switches, and this made the switches to stop reporting any kind of error being symbol frames of CRC errors.

This inserts itself in what was in the bug report, but this are all Cisco's 9300T with the 10Gbps expansion card, and the IOS is updated for the version 17.06.03 so none of the kind of issues should be present. Could this be a Siklu issue or a Cisco issue? I haven't opened a TAC case yet because I'm afraid Cisco will say, that's a Siklu issue and Siklu will say it's a Cisco issue and I'm in the middle. 

Any opinions please?

Thank you

[Leo Laohoo · Accepted solution]

Raise a TAC Case to make it "official".  

Just a reminder, that if this is a bug on the Cisco side, it will potentially take several years to fix unless a "sponsor" or "executive sponsor" inside Cisco can for push this bug to be fixed.  

[simoesmarco8626982]

Thank you @Leo Laohoo I'll do that, and also at the same time I'll disable the pause frames from the radios, because curiously the radios are sending pause frames, but if they are disabled the radios continue to transmit without dropping packets. This lead me to make a question to the radio manufacturer, if the radios are able to transmit without dropping packets with flow control off, why are they sending pause frames to begin with? 

[simoesmarco8626982]

Finally @Leo Laohoo I think i discovered what the issue is and for sure I should have stated this before. The system is running MacSec the switches are connected from switch to switch with the radios in the middle forwarding the packets. But if the radios inject pause frames in the macsec link the switches will not recognize this and will pass it as symbol and FCS errors for sure. 

This is the right assumption correct?

Thank you