Re: Input and CRC errors on every device connected to core switch

tyler.perkey · ‎06-04-2020

Hi all,

I have 4 Cisco 2960x switches, 1 acting as a "core" with port channels to the other 3 distribution switches. 1 of the port channels is fiber and the other 2 are copper. On the interfaces of every distribution switch I have CRC and Input errors on the links going to the core. It's too big of a coincidence that all 3 would have bad cables or dirty fiber, we've already tried different ports and changing fiber and we still get errors. The configs are pretty simple, nothing real complicated and all interfaces are set at 1000/full with 1500mtu size. Other than a bug I have no idea what could be causing these errors.

Reza Sharifi · ‎06-04-2020

Hi,

Does a make a difference if for testing you just use a single physical interface (no Portchannel) for one of the uplinks?

HTH

Leo Laohoo · ‎06-04-2020

Post the complete output to the following commands:

sh interface <PHYSICAL PORT> controll

tyler.perkey · ‎06-05-2020

Output of show int (int) controller

CoreSW1#sh int gi1/0/19 controll
GigabitEthernet1/0/19 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is a0b4.3970.2313 (bia a0b4.3970.2313)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:10, output 00:00:07, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 3000 bits/sec, 3 packets/sec
89725 packets input, 52506584 bytes, 0 no buffer
Received 9891 broadcasts (9749 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 9749 multicast, 0 pause input
0 input packets with dribble condition detected
967999 packets output, 165241496 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Transmit GigabitEthernet1/0/19 Receive
165241496 Bytes 52506584 Bytes
507541 Unicast frames 79834 Unicast frames
370588 Multicast frames 9749 Multicast frames
89870 Broadcast frames 142 Broadcast frames
0 Too old frames 48463651 Unicast bytes
0 Deferred frames 3779705 Multicast bytes
0 MTU exceeded frames 9088 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 0 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 17215 Minimum size frames
0 8 collision frames 31250 65 to 127 byte frames
0 9 collision frames 1793 128 to 255 byte frames
0 10 collision frames 9378 256 to 511 byte frames
0 11 collision frames 490 512 to 1023 byte frames
0 12 collision frames 678 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 0 Invalid frames, too large
0 VLAN discard frames 28921 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
17211 64 byte frames 0 Valid frames, too small
468069 127 byte frames
404044 255 byte frames 0 Too old frames
56574 511 byte frames 0 Valid oversize frames
14027 1023 byte frames 0 System FCS error frames
2861 1518 byte frames 0 RxPortFifoFull drop frame
5213 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames

Stack3#sh int gi1/0/47 controll
GigabitEthernet1/0/47 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is b000.b48f.f32f (bia b000.b48f.f32f)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:09, output 00:00:03, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 78
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 136000 bits/sec, 71 packets/sec
5 minute output rate 27000 bits/sec, 14 packets/sec
139943147 packets input, 134954966430 bytes, 0 no buffer
Received 13699524 broadcasts (11345073 multicasts)
0 runts, 0 giants, 0 throttles
117689 input errors, 117689 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 11345073 multicast, 0 pause input
0 input packets with dribble condition detected
31842558 packets output, 14517709322 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Transmit GigabitEthernet1/0/47 Receive
1632809248 Bytes 1810991361 Bytes
31189341 Unicast frames 126243673 Unicast frames
571642 Multicast frames 11345074 Multicast frames
81586 Broadcast frames 2354451 Broadcast frames
0 Too old frames 3683935091 Unicast bytes
0 Deferred frames 1552799681 Multicast bytes
0 MTU exceeded frames 301575494 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 117689 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 552122 Minimum size frames
0 8 collision frames 29578760 65 to 127 byte frames
0 9 collision frames 11073335 128 to 255 byte frames
0 10 collision frames 8576401 256 to 511 byte frames
0 11 collision frames 5713409 512 to 1023 byte frames
0 12 collision frames 41325023 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 10 Invalid frames, too large
0 VLAN discard frames 43241827 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
552171 64 byte frames 0 Valid frames, too small
18192742 127 byte frames
2908651 255 byte frames 0 Too old frames
1223308 511 byte frames 0 Valid oversize frames
1418430 1023 byte frames 0 System FCS error frames
5043577 1518 byte frames 0 RxPortFifoFull drop frame
2503690 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames

Leo Laohoo · ‎06-05-2020

@tyler.perkey wrote:
117689 FCS errors

Depending on the firmware of the switch, run a TDR on Gi 1/0/47.

tyler.perkey · ‎06-05-2020

TDR test:

Stack3#show cable-diagnostics tdr int gi1/0/47
TDR test last run on: June 05 11:57:57

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/47 1000M Pair A 22 +/- 10 meters Pair A Normal
Pair B 22 +/- 10 meters Pair B Normal
Pair C 22 +/- 10 meters Pair C Normal
Pair D 22 +/- 10 meters Pair D Normal
Harper-SW3#test cable-diagnostics tdr interface gi1/0/48
TDR test started on interface Gi1/0/48
A TDR test can take a few seconds to run on an interface
Use 'show cable-diagnostics tdr' to read the TDR results.
Stack3#show cable-diagnostics tdr int gi1/0/48
TDR test last run on: June 05 11:58:47

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/48 1000M Pair A 21 +/- 10 meters Pair B Normal
Pair B 21 +/- 10 meters Pair A Normal
Pair C 21 +/- 10 meters Pair D Normal
Pair D 21 +/- 10 meters Pair C Normal

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/21 1000M Pair A 21 +/- 10 meters Pair B Normal
Pair B 21 +/- 10 meters Pair A Normal
Pair C 21 +/- 10 meters Pair D Normal
Pair D 21 +/- 10 meters Pair C Normal
CoreSW1#test cable-diagnostics tdr interface gi1/0/22
TDR test started on interface Gi1/0/22
A TDR test can take a few seconds to run on an interface
Use 'show cable-diagnostics tdr' to read the TDR results.
CoreSW1#show cable-diagnostics tdr int gi1/0/22
TDR test last run on: June 05 12:00:08

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/22 1000M Pair A 21 +/- 10 meters Pair A Normal
Pair B 21 +/- 10 meters Pair B Normal
Pair C 21 +/- 10 meters Pair C Normal
Pair D 21 +/- 10 meters Pair D Normal
CoreSW1#

Leo Laohoo · ‎06-05-2020

The output looks good to me.
What is connected to Gi 1/0/47?

tyler.perkey · ‎06-05-2020

That would be one of the other interfaces shown on Stack3. It's part of the port channel between Core and Stack3

Leo Laohoo · ‎06-05-2020

If it is another switch port, run a TDR that way.

tyler.perkey · ‎06-07-2020

The TDR for the 4 ports shown are links between core1 and stack3.

Leo Laohoo · ‎06-07-2020

Gi 1/0/47 is a copper cable. Replace it and clear the counters. Observe for 1- and 7 days if the errors are still incrementing or not.

tyler.perkey · ‎06-07-2020

That has been tried in the past, counters were cleared, core switch was replaced under Cisco RMA because counters keep incrementing. We have certain even that's will nearly shut the entire network down and the errors will start to increment. We don't have constant issues, just sporadic events.

Leo Laohoo · ‎06-07-2020

So the link between the two switches is a single-piece wire? Is there any patch panel between the two?

tyler.perkey · ‎06-08-2020

Correct, links between stack switches and core at straight runs of copper. We also have the errors on the 1 stack that uses fiber back to the core.

Leo Laohoo · ‎06-08-2020

What firmware is the core switch running on?