10Gig (tengigabit) CRC Errors between N5K500 and 3750X

Michael Weller · ‎07-16-2012

Dear Forum,

We have an issue with a 10 Gig connection between a Nexus 5548 and a 3750X 10Gig. There is also a HSRP group running over this link with aggressive timeouts which recently behaves somewhat unstable. We noticed a bunch of CRC errors on this link on the 3750X side (about 300 per hour) and assume that these are related to the HSRP troubles once or twice a day.

Originally the connection was made with a twinax cable. We already changed to MMF fiber with SR SFP+'s. The 10Gig Uplink Module in the 3750x has been replaced already. All to no avail. Now it's up to replace the 3750 (receives CRC) or the Nexus Chassis (sending device)

I was wondering if there is some experience on the list what is the most probably cause and solution here.

Thanks in advance,

Michael.

Leo Laohoo · ‎07-16-2012

On the 3750X, can you post the output to the command "sh controll e ten " please?

Michael Weller · ‎07-16-2012

Sure:

C3750#show controllers ethernet-controller t
C3750#show controllers ethernet-controller tenGigabitEthernet 3/1/1

     Transmit TenGigabitEthernet3/1/1         Receive
574708109169 Bytes                     518900683562 Bytes
   1439005510 Unicast frames              3807510591 Unicast frames
   1700275095 Multicast frames            2013035702 Multicast frames
     97077957 Broadcast frames             361871415 Broadcast frames
            0 Too old frames            750584488179 Unicast bytes
            0 Deferred frames           149720142824 Multicast bytes
            0 MTU exceeded frames        40440767697 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames             1299355 FCS errors
            0 3 collision frames                   1 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames                 981 Minimum size frames
            0 8 collision frames          2689693081 65 to 127 byte frames
            0 9 collision frames          1727005063 128 to 255 byte frames
            0 10 collision frames         1091861080 256 to 511 byte frames
            0 11 collision frames         1515886754 512 to 1023 byte frames
            0 12 collision frames          497582417 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 2 Symbol error frames
            0 Late collisions                      4 Invalid frames, too large
            0 VLAN discard frames         2956654982 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
     10050135 64 byte frames                       0 Valid frames, too small
   1897954866 127 byte frames
   2274249521 255 byte frames                      0 Too old frames
   1808046421 511 byte frames                      0 Valid oversize frames
   4247747354 1023 byte frames                     0 System FCS error frames
   1134905192 1518 byte frames                     0 RxPortFifoFull drop frame
    453339665 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

C3750#show controllers ethernet-controller tenGigabitEthernet 4/1/1

     Transmit TenGigabitEthernet4/1/1         Receive
401013315009 Bytes                     487542547741 Bytes
   2315452233 Unicast frames              2487385055 Unicast frames
    315842017 Multicast frames            1505902339 Multicast frames
     59648917 Broadcast frames             185748001 Broadcast frames
            0 Too old frames            119286086552 Unicast bytes
            0 Deferred frames           103246245303 Multicast bytes
            0 MTU exceeded frames        24880634311 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames              983530 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames               49004 Minimum size frames
            0 8 collision frames           445407751 65 to 127 byte frames
            0 9 collision frames          2540569644 128 to 255 byte frames
            0 10 collision frames          799266760 256 to 511 byte frames
            0 11 collision frames         2422515083 512 to 1023 byte frames
            0 12 collision frames           68360393 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames         2198817586 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
      4432313 64 byte frames                       0 Valid frames, too small
   2439601404 127 byte frames
   3641955592 255 byte frames                      0 Too old frames
   3604137484 511 byte frames                      0 Valid oversize frames
   2478595942 1023 byte frames                     0 System FCS error frames
   1831032940 1518 byte frames                     0 RxPortFifoFull drop frame
   1576089380 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

I'm also concerned about the too large frames. It seems to me the Nexus uses Jumbo-Frames (don't know if it should, might be a default setting).

Oh, BTW.. I just saw the N5k is not a 5548, but a 5020. Sorry.

Leo Laohoo · ‎07-16-2012

I'm very concerned about the high amount of FCS (Frame Check Sequence) line errors in both interfaces.

Forgot to ask the corresponding output to the commands "sh interface ten 3/1/1" and "sh interface ten 4/1/1".

Michael Weller · ‎07-16-2012

MTU is 1500 bytes, on both sides (I think) see output below. Strangely enough the NX has a non-zero Jumboframe counter (increasing).

C3750#sh int te3/1/1
TenGigabitEthernet3/1/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 4055.3992.111d (bia 4055.3992.111d)
Description: TO_100C_PO913
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
Media-type configured as connector
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:55, output 00:00:41, output hang never
Last clearing of "show interface" counters 04:10:35
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4011000 bits/sec, 1091 packets/sec
5 minute output rate 110431000 bits/sec, 16489 packets/sec
     37040147 packets input, 35868731879 bytes, 0 no buffer
     Received 1071851 broadcasts (935322 multicasts)
     0 runts, 0 giants, 0 throttles
     1002 input errors, 1002 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 935322 multicast, 0 pause input
     0 input packets with dribble condition detected
     182446795 packets output, 69097435882 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
C3750#sh int te4/1/1
TenGigabitEthernet4/1/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 4055.39bd.001d (bia 4055.39bd.001d)
Description: TO_100D_PO913
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
Media-type configured as connector
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:14, output 00:00:33, output hang never
Last clearing of "show interface" counters 04:10:38
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 161000 bits/sec, 215 packets/sec
5 minute output rate 29493000 bits/sec, 9580 packets/sec
     3264201 packets input, 318586322 bytes, 0 no buffer
     Received 1130048 broadcasts (1020339 multicasts)
     0 runts, 0 giants, 0 throttles
     501 input errors, 501 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1020339 multicast, 0 pause input
     0 input packets with dribble condition detected
     168673910 packets output, 87266013753 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

N5K connected to 3/1/1:

Ethernet1/1 is up
Hardware: 1000/10000 Ethernet, address: 0005.9b1f.8648 (bia 0005.9b1f.8648)
Description: CHANNEL_TO_RO015A
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 2d04h
Last clearing of "show interface" counters never
30 seconds input rate 96695496 bits/sec, 16464 packets/sec
30 seconds output rate 4715024 bits/sec, 1868 packets/sec
Load-Interval #2: 5 minute (300 seconds)
    input rate 148.33 Mbps, 20.23 Kpps; output rate 2.90 Mbps, 1.13 Kpps
RX
    26173471641 unicast packets 6446061938 multicast packets 24295222 broadcas
t packets
    32643828801 input packets 16853116460189 bytes
    7772805495 jumbo packets 0 storm suppression packets
    0 runts 0 giants 0 CRC 0 no buffer
    0 input error 0 short frame 0 overrun   0 underrun 0 ignored
    0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
    0 input with dribble 0 input discard
    0 Rx pause
TX
    48369101239 unicast packets 462061134 multicast packets 81589744 broadcast
packets
    48912752123 output packets 56533126796163 bytes
    35421478216 jumbo packets
    6 output errors 0 collision 0 deferred 0 late collision
    0 lost carrier 0 no carrier 0 babble 0 output discard
    0 Tx pause
3 interface resets

N5K connected to 4/1/1:
Ethernet1/1 is up
Hardware: 1000/10000 Ethernet, address: 0005.9b1f.8188 (bia 0005.9b1f.8188)
Description: CHANNEL_TO_RO0015A
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 2d04h
Last clearing of "show interface" counters never
30 seconds input rate 20643816 bits/sec, 10541 packets/sec
30 seconds output rate 142592 bits/sec, 180 packets/sec
Load-Interval #2: 5 minute (300 seconds)
    input rate 25.50 Mbps, 9.49 Kpps; output rate 159.20 Kbps, 142 pps
RX
    119855915845 unicast packets 6017690609 multicast packets 20318382 broadca
st packets
    125893924837 input packets 124351081100162 bytes
    70763564145 jumbo packets 0 storm suppression packets
    0 runts 0 giants 1 CRC 0 no buffer
    1 input error 0 short frame 0 overrun   0 underrun 0 ignored
    0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
    0 input with dribble 0 input discard
    0 Rx pause
TX
    20958491666 unicast packets 503446052 multicast packets 52796318 broadcast
packets
    21514734036 output packets 29748981423583 bytes
    18964491576 jumbo packets
    0 output errors 0 collision 0 deferred 0 late collision
    0 lost carrier 0 no carrier 0 babble 0 output discard
    0 Tx pause
3 interface resets

Leo Laohoo · ‎07-16-2012

Ok, no output drops on the 3750X. That's good news.

Out of curiousity, what kind of fibre optic cable are you using? Are you using OM3 or OM4?

Michael Weller · ‎07-16-2012

That's only OM3.. but only a very short cable, about 5m. Note that it used to be a direct attached SFP-H10GB-CU5M before.

Then, when the FCS errors were noticed, it was already replaced with this fibre connection last weekend. Hence I'd exclude an error in the cables or sfps since they've already been replaced with a completely different technology.

Those two lines are btw channeled with LACP, it's a multichassis VPC on the Nexus side.

Leo Laohoo · ‎07-16-2012

Hmmm ... 5 meter cable? Is this a SINGLE piece? Meaning this fibre optic doesn't go to a FOBOT?

Judging from the first output, the problem is coming from the Nexus side.

One more thing: both Ten3/1/1 & Ten4/1/1 are coming from the same 5020?

If the answer is yes, then i think it's coming from the appliance.

ASIC: what ports in the 5020? Can you move one link to another port group?

Sent from Cisco Technical Support iPad App

Michael Weller · ‎07-16-2012

Yes, one single piece of fibre, no fobot. The fibres have been brought in place instead of the SFP-H10GB using onsite spare parts, for testing if there's a problem.

No it's not the same 5020. It's a vpc pair. Eth1/1 on each chassis, see show int output above. What confuses me here is, that these are two N5K chassis and both produce CRC failures. Likewise, connection ends up on two different 3750chassis (combined in a stack), each reporting CRC failures. There is no single port ASIC affected. Do you know by heart how the ports are grouped on asics? I'd need to search for this info. But agains, ports are distributed over chassis.

Of course, the 5020 are in a vpc cluster and the 3750x are stacked. So errors might somehow spread over the systems.

I'm also confused the 5020s report incoming jumbo frames though the system mtu on the 3750 is 1500, least to talk about the interface mtu.

Leo Laohoo · ‎07-16-2012

Ok. You got me stump.

The FCS counters from the 3750X means the errors are coming from the 5K.

By the way, does the serial number(s) of the SFP+ start with an "A"?

Michael Weller · ‎07-17-2012

No Problem,

since I wasn't sure which device to replace by RMA I opened a TAC case. The Tac is currently confused too. They deep analysed the Nexus w/o finding a problem and are now analysing the 3750, maybe looking for a software issue too.

The main point about this thing is that there is no common denominator.. TwinAx or Fibre, two 5020, two 3750. And still two ports showing the same effect.

SNs of the SFP-10G-SR begin with FNS btw. They should be ok and original, no OEM or similar.

If a solution is found, I'll post a note.

Leo Laohoo · ‎07-17-2012

FNS means its made by Finnisar. It should be OK.

Normally, if I had a case like this, I'd replace both SFP+ and fibre optic cable three time.

But since it's happening to two different machines I'm no longer sure this is the case.

What IOS is the 3750X running?

Sent from Cisco Technical Support iPad App

Michael Weller · ‎07-17-2012

> But since it's happening to two different machines I'm no longer sure this is the case.

And.. the crc had been detected first with twinax cables. The SFP+ are already a replacement for the Twinax.

12.2(55)SE1; IP-Services. Actually it's a mixed stack: 2 3750X and 2 3750G-12S. I'm wondering if there is an issue with the current stack master.

Leo Laohoo · ‎07-17-2012

12.2(55)SE1; IP-Services.  Actually it's a mixed stack: 2 3750X and 2 3750G-12S. I'm wondering if there is an issue with the current stack master.

Uh-oh.

I'm no big fan of this IOS level. Try 12.2(55)SE4 or 12.2(55)SE5 if you can get away with upgrading the IOS.

Cory Peterson · ‎08-08-2012

Hello,

We have noticed the exact same behaviour between our 3750x and Nexus 7k boxes running on Twinax. With the same rate of CRC errors.

Has TAC Been able to idntifiy this issue at all? Did you try to update the IOS?

We are running 12.2(58)SE2 on the 3750x.

Thanks!

Cory