Weird fiber issues

[gchialliacn]

I'm having issues at my client and I'm really out of ideas. Hope someone can help me.

I have two 4506 switches at the Distribution layer, connected to stack of 2960X switches at the access side. Tengig uplinks on single mode fiber. SFPs are 10G-LRs.

The top stack member is connected to the Dist SW1, and the bottom stack member is connected to Dist SW2.

We started to observe CRC errors at Dist SW2. We replaced SFPs, but errors stayed. Cleaned fiber connectors: no change. The fiber vendor tested the fiber and it looks OK. Loss levels below 3 dB and even at the switch side it always show receive power above -5 dBm.

We swapped fibers all the way, and errors moved to Dist SW1. The swapped Tx and Rx sides and errors moved to the Access side.

The fiber company re-did all the fiber connectors (replaced MTRJ connectors with LC all the way). Loss levels still below 3dB.

It doesn't look like this could be a device issue, since errors follow the fiber across 4 devices and 2 different platforms.

Is there any fiber related issue that won't appear as loss, but could distort the light pulses and cause CRC errors at the receiving side?

Thanks in advance.

[david scott]

Hi,

Are you using multimode or single mode fibre? It's single mode for LR.

Are the fibre plugs and sockets the same? Blue into blue, green into green?

The transmit power for LR is right on the maximum receive power for LR, so while it *shouldn't* overpower the receiver you could try an inline 5dB attenuator on short runs. 5dB will still leave plenty of power budget.

Cheers.

[gchialliacn]

Yes, it is single mode fiber. Connectors are all blue.

Regarding the receive power, according to the SFP-10G-LR datasheet the maximum is 0.5 dBm. I don't believe I would need to attenuate the signal if I'm getting -5.0 dBm. Right?

Another data point is that this client was using 6500s before, with XENPAKs. They never had any issue. Could the fiber be incompatible with the SFP+ modules?

Also, there are two buildings in the campus. We are having the issue in the older building. The newer building has newer fiber and is not presenting these issues.

Thanks

[Leo Laohoo]

I have two 4506 switches at the Distribution layer, connected to stack of 2960X switches at the access side. Tengig uplinks on single mode fiber. SFPs are 10G-LRs.

I'm not yet pointing any "fingers" yet but do you know that 2960X has been known to have issues with the SFP ports?  Regardless of the IOS you're running, you can either get the SFP ports to "work" well and crash something else or stabilize other functions of your switch but ruin your SFP.  That's the latest "story".  Search here or Google for "2960X", "SFP" and "issues" and you'll see hits.  A lot of hits. 

Ok, here's one way of testing.  Directly connect a 2960X to your switch and see if you see the same issue.  Setting it up this way is the best way to determine if you've got a backbone issue, an optical transmission issue or a fibre optic cable issue. 

Can you also post the outputs to the command "sh interface <BLAH>" of both interfaces having errors?

[gchialliacn]

From what I see the SFP issues on the 2960Xs are mostly about the switch not recognizing the SFP. Haven't seen reports of CRCs.

We are going to try moving one of the 2960X to the DC side and plug directly to the 4506s and see what happens, as suggested by you.

This is the show interface output of one of the interfaces with errors on the 4506 side:

TenGigabitEthernet3/5 is up, line protocol is up (connected)
  Hardware is Ten Gigabit Ethernet Port, address is 1005.ca6b.ece0 (bia 1005.ca6b.ece0)
  Description: ******
  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR
  input flow-control is on, output flow-control is on
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters 4w2d
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 313000 bits/sec, 138 packets/sec
  5 minute output rate 16636000 bits/sec, 1531 packets/sec
     1974721679 packets input, 673075433436 bytes, 0 no buffer
     Received 31177677 broadcasts (27111655 multicasts)
     0 runts, 9939 giants, 0 throttles
     6517561 input errors, 6507622 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     4430086868 packets output, 5571376469640 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

Thanks