CRC and Input Errors - Cisco ASR 907

[tppatrick1]

Good day, everyone,

I am experiencing CRC and input errors on one of my core fibre links, a 10Gig link using ER SFP modules. I noticed that the reliability on one end of the link is 248/255 - I read on one of the threads where one gentleman said this setting's reliability settings should be 255/255. As per my understanding, this is a layer one issue, most likely caused by a faulty patch lead or mid-coupler.

Question 1: The input and CRC errors are being caused by the faulty cables on the side where I am noticing the errors?                Question 2: Do we need to change the patch leads on the other side where the interface is running clean with no input & CRC errors?

This P2P link." running ISIS Level 2

Patrick

[Ramblin Tech · Accepted solution]

One end of the link is reporting that the frames it receives do not have valid CRCs, as calculated by the receiver's circuitry. Based on that information alone, it is not possible to definitively say whether the issue is with the receiver's CRC circuitry (it is mis-calculating the CRCs), the transmitter's circuitry (it is sending frames with bad CRCs), or any of the media segments in between.  However, experience tells us that it is more likely that valid frames are being sent by the transmitter and then being corrupted somewhere in the media.

You can eliminate the transmitting and receiving ends as being the source of the CRC errors by connecting each to some other local device (or looping these SFPs back to themselves) with short fiber jumpers, though you might need to insert attenuators to prevent saturation of their respective receivers, as the ER SFPs might launch too hot for a short jumper.  Assuming the transmitter and receiver run cleanly with local terminations, then it is a methodical process of elimination to swap out each media segment to determine if the CRC errors are eliminated.

Back to your questions:

Question 1: The input and CRC errors are being caused by the faulty cables on the side where I am noticing the errors?

There is no way to determine this without testing. The CRC errors could be introduced close to the transmitter, just as easily as they could be introduced close to the receiver.

Question 2: Do we need to change the patch leads on the other side where the interface is running clean with no input & CRC errors?

Same answer as #1.

I should also note that I am answering based on the assumption of dark fiber between your transmitter and receiver. If this link has active Ethernet components (eg, L2 switches) in the path, then the focus should be on the media segment between the receiver with the errors and the last switch, as CRC errors do not propagate through these devices. That is, a properly functioning L2 switch will discard any Ethernet frames it receives with a bad CRC.

[tppatrick1]

Hi @Ramblin Tech 

I appreciate the detailed explanation. There are no devices between this link, except for one patch through which I requested the fibre provider to clean the connectors and replace the patch lead. They have already done this.

I will conduct tests with other local devices. In the past, I had a negative experience when I looped back a ZR SFP module. After the loop test, the SFP stopped working. I think the receiver sensitivity was damaged because the transmitter power was too high for the short distance. I've realized that the distance of this link is too short for ER optics. The total distance is 11000 meters, and this could potentially cause damage to the optics.

Once I test it with local devices, I can determine if I need to replace the optics. Please refer to the attached dB levels.

Thank you for taking the time to explain in detail once again. I appreciate it.

I will provide you with the results tomorrow.

Partick

[Ramblin Tech · Accepted solution]

If you are using a short cable with an optic that has a high launch power (eg, ZR, ER, WDM) use an attenuator to reduce the power at the receiver. They are available in various dB values. I randomly grabbed these links (I have no connection to the vendors) and there are plenty of others:

https://www.fibersavvy.com/collections/fiber-attenuators

https://fibercablesdirect.com/19-fiber-optic-attenuators

[tppatrick1]

Hi @Ramblin Tech

Thank you so much for all your responses. I wanted to update everyone that my issue has now been resolved. After testing with the local switch, I still observed errors on the PE port and even encountered packet drops when pinging across to the local switch. To resolve this, I moved the configuration to another line card with a free 10 Gig port and found no errors on the new PE port, and the pings to the local switch were without any drops.

Next, we connected the new 10 Gig port to the remote PE router on the other end of the P2P link, and we did not observe any errors whatsoever. All other services previously experiencing slow network issues are working perfectly well over this link.

Once again, thank you to everyone who contributed to this issue!

@Georg Pauwen Thank you!

@Leo Laohoo  Thank you!

Patrick. 

[Leo Laohoo · Accepted solution]

Get an OTDR done by a professional to determine if the fibre run has a fault in it. 

Get a fibre optic inspector (a camera) and check the FOBOT and patch cables if they are dirty. 

Finally if the OTDR comes out clean and the fibre optic inspector passes the FOBOT and the patch cables, replace the optics one by one.

[tppatrick1]

Hi @Leo Laohoo 

Thank you so much for your assistance. The fibre has been tested with an OTDR and appears to be okay. I have attached the dB levels above. I agree with your suggestion to have the fibre provider check the FOBOT, as the link terminates on a cross-connect on both ends and those trays could also be the issue.

Thank you,

Patrick

[Georg Pauwen · Accepted solution]

Hello,

can you show the entire output of the interface, e.g.:

sh int ten 0/4/0

Also, can you find out which RSP you have (1/2/3) ?

[tppatrick1]

Hi @Georg Pauwen 

Thank you for responding. 

I have shutdown one end of the link due to a slow network and freezing video traffic when traffic traverses over that link.

For the HAR-PE1, where we are noticing the errors, I am using RSP1 the standby. The primary RSP0 got damaged, and the client is sorting out Cisco maintenance support so we can RMA.
The other end, GZR-PE1, is using RSP0 the primary. The standby RSP1 is faulty, too and we are waiting for support to be sorted.

Please refer to the attached. 

Patrick.