CRC errors increasing, but pingtests are 100% ok!

SlevinKelevra · ‎06-27-2013

Gents,

My crc errors are steadily increasing on the Ethernet segment, but when i ping the BGP neighbor (so packets travel accross this "bad" segment) i do not see any packets going bad. Anyone got an explanation for this?

How is this possible, i would expect at least something going wrong with my ping test.

Unfortunately Google could not help me.

xxxx#ping xxx.22x.193.174 repeat 1000 size 1400

Type escape sequence to abort.
Sending 1000, 1400-byte ICMP Echos to xxx.22x.193.174, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 232/251/552 ms <===

xxxx#sh int Fa0/0

FastEthernet0/0 is up, line protocol is up

Hardware is Gt96k FE, address is 001f.9ee5.d32a (bia 001f.9ee5.d32a)

Description: ISP

Internet address is xxx.12x.110.226/27

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:18, output 00:00:00, output hang never

Last clearing of "show interface" counters 00:05:28

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 214000 bits/sec, 76 packets/sec

5 minute output rate 232000 bits/sec, 72 packets/sec

25999 packets input, 9368996 bytes

Received 7 broadcasts, 0 runts, 0 giants, 0 throttles

66 input errors, 66 CRC, 0 frame, 0 overrun, 0 ignored <===

0 watchdog

0 input packets with dribble condition detected

25199 packets output, 10021048 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

6 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

vmiller · ‎06-27-2013

CRCs are a physical layer problem. Ping won't help there.

check the cable, on this device and the one its conected to, if possible move ports on one device at a time

note the time of your last clearing of counters in the show interface, you picked up a few more crcs..

SlevinKelevra · ‎06-27-2013

Can you tell me why ping will not help me there, is there any other way to test a circuit? In my opinion pings simulate traffic and crc errors are just bad packets, caused by a physical problem. An Ethernet node will check every packet before sending it out if is valid if not it will be dropped, so if there are CRC errors i should see bad packets.

PE ---- DSL --- Ethernet node provider ---- Router ----- LAN

- Cables has been replaced, i still need to change the port on the router though

vmiller · ‎06-27-2013

Ping tests IP only.So we know at this point layer 3 works.PING wan't give you details of lower level issues (i.e. CRC).

If the Ethernet node provider listed in your diagram is owned by the service provider I would get them engaged.

Make sure all the settings on your end are ok (speed,duplex....)

Note: 66 input errors, 66 CRC

this is from the cisco guide on ethernet troubleshooting :

CRC

Indicates that the cyclic redundancy checksum generated by the originating LAN station or far-end device does not match the checksum calculated from the data received. On a LAN, this usually indicates noise or transmission problems on the LAN interface or the LAN bus itself. A high number of CRCs is usually the result of collisions or a station transmitting bad data.

Eehab Suliman · ‎06-27-2013

Although a CRC is an input error your ping test was a good start because you get ICMP back.

Sometimes a high number of CRC or FCS errors on multiple ports may indicate a capacitor problem.

As the capacitor degrades, the network interface controllers will be unable to reliably decode packets from the network.

However, a capacitor problem will affect all interfaces. If you are seeing the crc errors on one circuit, this is not a capacitor problem.

Please provide the output of:

show controller eth xx/yy
show controllers ethernet-controller | inc FCS errors

I'd reinsert the GBIC, clear counters and replace the cable. If all this doesnt explain the behavior you are seeing, then you are likely running into a bug and I would need the platform and code version.

Checking with your carrier is always a good idea when troubleshooting circuit issues.

kind regards, Eehab

SlevinKelevra · ‎06-27-2013

Eeham, here is my output, the | include FCS command did not work, so i have pasted the output here, hope this helps.

Do know that the cable already has been replaced. Below you will find the controller output for FastEthernet 0/0 (which has errors)

The FastEthernet 0/1 is clean.

Problem interface

xxxxx#sh controllers fastEthernet 0/0
Interface FastEthernet0/0
Hardware is GT96K FE ADDR: 6318CB4C, FASTSEND: 61C3E1D0, MCI_INDEX: 0
DIST ROUTE ENABLED: 0Route Cache Flag: 11
GPIO 2 CONF= 67FF07A8 GPIO 2 IO= 187D303F CIU arbit = 80F002BF
PHY add register = 0xA4 PHY data register = 0xFC50000
Port Conf Reg= 0x8080 ENABLE HT8K HMOD0 FDX
Port Conf Ex Reg= 0x1CCF00
TX1:1 RXPRI=DE(00) ~DPLXAUTO ~FLCNTL ~FLNKP MFL64KB FE ~SPDAUTO RMII
Port Com Reg= 0x0
Port Status Reg= 0xB 100MB FDPX FCTL EN LNK UP ~PAUSED TX oFF
Serial Param Reg= 0x218823 Hash table pointer= 0xE698960
Source ADDR L= 0xD32A Source ADDR H= 0x1F9EE5
SDMA conf reg= 0x223C RETX 15 RX BE TX BE FRINT BSIZE 4
SDMA com reg= 0x1010080 SRT TXL STP TXH EN RX
IMASK= 0x90003DCD ICause= 0x0
Serial 0 mask 30FFFC3Serial 0 cause 0
IpDiffservP0L= 0x0 IpDiffservP0H= 0x0 IpDiffservP1L= 0x0 IpDiffservP1H= 0x0
IP VLAN TAG PRI= 0xF0CC IP VLAN TAG PRI= 0xF0CC
First rxd Q0= 0xE6D8AB0 Curr rxd Q0= 0xE6D8AB0
First rxd Q1= 0xE6D8E40 Curr rxd Q1= 0xE6D8E40
First rxd Q2= 0xE6D92A0 Curr rxd Q2= 0xE6D92A0
First rxd Q3= 0xE6D9700 Curr rxd Q3= 0xE6D9700
First txd Q0= 0xE6DA120 First txd Q1= 0xE6DA3A0
gt96kfe_instance=0x6318DC14, registers=0x24084800
RxRing entries=64, tx ring entries=128
RxR0=0x E6D89E0, RxR1=0x E6D8E40, RxR2=0x E6D92A0, RxR3=0x E6D9700
Malloc RxR0=0xEE6D89E0, RxR1=0xEE6D8E40, RxR2=0xEE6D92A0, RxR3=0xEE6D9700
SDOW RxR0=0x6318E20C, RxR1=0x6318E340, RxR2=0x6318E474, RxR3=0x6318E5A8
HEAD RxR0=0xD, RxR1=0x0, RxR2=0x0, RxR3=0x0
TAIL RxR0=0x0, RxR1=0x0, RxR2=0x0, RxR30x0

tx_limited=0(128)
TxR0=0x E6D9B60, TxR1=0x E6DA3A0
COUNT TxR0=0x0, TxR1=0x0
Head TxR0=0x5A, TxR1=0x0 Tail TxR0=0x5A, TxR1=0x0

PHY registers:
Register 0x00:   2100 780D 0040 61E4 0101 0000 0004 2001
Register 0x08:   0000 0000 0000 0000 0000 0000 0000 0000
Register 0x10:   1000 0300 0000 0000 0200 0126 0600 0000
Register 0x18:   0037 000D 9F00 008A 0827 0000 0000
Bytes_recvd 1187451105 Bytes_sent 1038526126 Frames_recvd 175676163 Frames_sent 169961633
total_bytes_RX 1402562577 Total_frames_RX 182010338 Bcast_frames_recvd 112118
Mcast_frames_RX 36708 CRC_err 765404 Ovr_sized_frames 0
Fragments 5568771 Jabber 0 collision 0
Late_collision 0 64B frame 340390; 65_127B_frames 20548319
128_255B_frames 175024394 256_511B_frames 79829283 512_1023B_frames 28252734
1023_maxB_frames 41642676 Rx_error 0 Dropped_frames 0
Mcast_frames_tx 3664 Bcast_frames_tx 7 Sml_frame_recvd 0

Software MAC address filter(hash:length/addr/mask/hits):
0x00: 0 ffff.ffff.ffff 0000.0000.0000         0
0xB4: 0 001f.9ee5.d32a 0000.0000.0000         0
0xC0: 0 0100.0ccc.cccc 0000.0000.0000         0

======= Driver Counters =======
Number of Transmitter Hang = 0
tx_more_col_err= 0 tx_one_col_err= 0
tx_exc_collision_err= 0 tx_late_collision_err= 0
tx_underrun_err= 0 tx_error_intr= 0
rx_soft_overflow_err= 0 rx_overflow_err= 0

Interface which is clean and has no problems

xxxx#sh controllers fastEthernet 0/1
Interface FastEthernet0/1
Hardware is GT96K FE ADDR: 63199E90, FASTSEND: 61C3E1D0, MCI_INDEX: 0
DIST ROUTE ENABLED: 0Route Cache Flag: 613
GPIO 2 CONF= 67FF07A8 GPIO 2 IO= 187D303F CIU arbit = 80F002BF
PHY add register = 0xA4 PHY data register = 0xFC40000
Port Conf Reg= 0x8080 ENABLE HT8K HMOD0 FDX
Port Conf Ex Reg= 0x1CCF00
TX1:1 RXPRI=DE(00) ~DPLXAUTO ~FLCNTL ~FLNKP MFL64KB FE ~SPDAUTO RMII
Port Com Reg= 0x0
Port Status Reg= 0xB 100MB FDPX FCTL EN LNK UP ~PAUSED TX oFF
Serial Param Reg= 0x218823 Hash table pointer= 0xE7AAC20
Source ADDR L= 0xD32B Source ADDR H= 0x1F9EE5
SDMA conf reg= 0x223C RETX 15 RX BE TX BE FRINT BSIZE 4
SDMA com reg= 0x1010080 SRT TXL STP TXH EN RX
IMASK= 0x90003DCD ICause= 0x0
Serial 0 mask 30FFFC3Serial 0 cause 0
IpDiffservP0L= 0x0 IpDiffservP0H= 0x0 IpDiffservP1L= 0x0 IpDiffservP1H= 0x0
IP VLAN TAG PRI= 0xF0CC IP VLAN TAG PRI= 0xF0CC
First rxd Q0= 0xE7EB050 Curr rxd Q0= 0xE7EB050
First rxd Q1= 0xE7EB100 Curr rxd Q1= 0xE7EB100
First rxd Q2= 0xE7EB560 Curr rxd Q2= 0xE7EB560
First rxd Q3= 0xE7EB9C0 Curr rxd Q3= 0xE7EB9C0
First txd Q0= 0xE7EC060 First txd Q1= 0xE7EC660
gt96kfe_instance=0x6319AF58, registers=0x24088800
RxRing entries=64, tx ring entries=128
RxR0=0x E7EACA0, RxR1=0x E7EB100, RxR2=0x E7EB560, RxR3=0x E7EB9C0
Malloc RxR0=0xEE7EACA0, RxR1=0xEE7EB100, RxR2=0xEE7EB560, RxR3=0xEE7EB9C0
SDOW RxR0=0x6319B46C, RxR1=0x6319B5A0, RxR2=0x6319B6D4, RxR3=0x6319B808
HEAD RxR0=0x6, RxR1=0x0, RxR2=0x0, RxR3=0x0
TAIL RxR0=0x0, RxR1=0x0, RxR2=0x0, RxR30x0

tx_limited=0(128)
TxR0=0x E7EBE20, TxR1=0x E7EC660
COUNT TxR0=0x0, TxR1=0x0
Head TxR0=0x77, TxR1=0x0 Tail TxR0=0x77, TxR1=0x0

PHY registers:
Register 0x00:   2100 780D 0040 61E4 0101 0000 0004 2001
Register 0x08:   0000 0000 0000 0000 0000 0000 0000 0000
Register 0x10:   1000 0300 0000 0000 0200 00EC 0300 0000
Register 0x18:   0037 000D 9F00 008A 0827 0000 0000
Bytes_recvd 4005330696 Bytes_sent 3945401753 Frames_recvd 169953771 Frames_sent 175031935
total_bytes_RX 4005330696 Total_frames_RX 169953771 Bcast_frames_recvd 3935
Mcast_frames_RX 1385954 CRC_err 0 Ovr_sized_frames 0
Fragments 0 Jabber 0 collision 0
Late_collision 0 64B frame 21752406; 65_127B_frames 87390251
128_255B_frames 91003053 256_511B_frames 77320709 512_1023B_frames 26603794
1023_maxB_frames 40915493 Rx_error 0 Dropped_frames 0
Mcast_frames_tx 3664 Bcast_frames_tx 6100 Sml_frame_recvd 0

Software MAC address filter(hash:length/addr/mask/hits):
0x00: 0 ffff.ffff.ffff 0000.0000.0000         0
0xB5: 0 001f.9ee5.d32b 0000.0000.0000         0
0xC0: 0 0100.0ccc.cccc 0000.0000.0000         0

======= Driver Counters =======
Number of Transmitter Hang = 0
tx_more_col_err= 0 tx_one_col_err= 0
tx_exc_collision_err= 0 tx_late_collision_err= 0
tx_underrun_err= 0 tx_error_intr= 0
rx_soft_overflow_err= 0 rx_overflow_err= 0

Here is some extra info about the router, ios and cards that are installed.

xxxxx#sh ver

Cisco IOS Software, 1841 Software (C1841-ADVIPSERVICESK9-M), Version 12.4(25b), RELEASE SOFTWARE (fc1)

Cisco 1841 (revision 7.0) with 234496K/27648K bytes of memory.

Processor board ID FCZ1213133A

2 FastEthernet interfaces

1 Virtual Private Network (VPN) Module

Let me know if you need anything more!

Eehab Suliman · ‎06-27-2013

Based on everything we looked at, this is:

1- software bug

2- defective port

I can research this further tonight to see if there are any known bugs on the 1841 platform.

In the mean time you can move the circuit to fast ethernet 0/1. If the CRCs are not observed then you know you are running into a hardware issue and will have to replace the part.

cheers, Eehab

SlevinKelevra · ‎06-28-2013

Thhx Eehab, unfortunately i manage this router remotely, but i have arranged someone on site to help me with this. Will get back to you once this has been done.

Eehab Suliman · ‎06-28-2013

Np, Remco!

cheers, Eehab

Kimberly Adams · ‎06-28-2013

Remco,

I hate to aske the obvious here, but I have seen a lot of these kinds of issues being something simple like a speed/duplex issue in the configuration with the link partner. What is your interface configured for in this regard? Also, is the other side of this link also set the same way?

Thanks,

Kimberly

Thanks and Cheers! Kimberly Please remember to rate helpful posts.

chandaniba1 · ‎06-28-2013

Often the duplex mismatch causes slow in network speed and runts, it is important to have the same duplex setting at both ends which may be the reason of CRC errors

Sent from Cisco Technical Support iPhone App

SlevinKelevra · ‎06-30-2013

xxxxx#show cdp entry *
-------------------------
Device ID: LL-837
Entry address(es):
IP address: xxx.125.x10.2x5
Platform: Cisco C837, Capabilities: Router
Interface: FastEthernet0/0, Port ID (outgoing port): Ethernet0
Holdtime : 162 sec

Version :
Cisco IOS Software, C837 Software (C837-K9O3Y6-M), Version 12.3(8)T11, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Wed 10-Aug-05 17:33 by dchih

advertisement version: 2
Duplex: full <==========

Leo Laohoo · ‎06-30-2013

Interface: FastEthernet0/0,  Port ID (outgoing port): Ethernet0

Kimberly is right.

The downstream device is a Cisco 837. I believe the downstream port is an Ethernet port. So I think it's negotiated to half duplex?

SlevinKelevra · ‎07-01-2013

Hmzz, thats interesting, did email the ISP and they have told me that their port was configured (hard coded) as 100 full, but Ethernet only goes so far as 10..., but i believe them. Looked up at Cisco site. I do not know why it is saying indeed Ethernet... it doesnt make sense, my port is configured as 100 full else the connection wouldnt even be up and running.

" The Cisco 837 router supports ADSL over POTS (basic telephone service). All models offer a four-port 10/100 Ethernet LAN switch for connecting multiple PCs or network devices in a small-office network."

I have send out an engineer to the site to replace port and possible WIC. Will keep you guys updated!

Leo Laohoo · ‎07-01-2013

Are both ports configured for "auto/auto"?