3945 input errors and overruns

ylingf · ‎07-17-2014

Guys,

I have a 3945 that is seeing incrementing input errors and overruns. Buffer seems to be mostly fine, I added more buffers just to be sure. I read that the overrun error is usually interface hardware could not handle inbound traffic faster enough. But the traffic rate is at around 22Mbps over 30 second average. Not a lot for a gigabit interface. The overrun error increment by about 1 per two thousand packets. Also, rx_overflow_err is incrementing around 400 per second.

Is there a fix for this? Thanks a lot guys!

#sh int g0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is PQ3_TSEC, address is 7c69.f66c.cc80 (bia 7c69.f66c.cc80)
Description: Stealth Internet
Internet address is 207.251.103.122/30
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 5/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:36:20
Input queue: 0/1000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 22862000 bits/sec, 4714 packets/sec
30 second output rate 2893000 bits/sec, 1857 packets/sec
10501821 packets input, 6532404997 bytes, 0 no buffer
Received 1 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
4671 input errors, 0 CRC, 0 frame, 4671 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
4163926 packets output, 778284915 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

#sh buffers
Buffer elements:
1048 in free list
599358 hits, 0 misses, 1626 created

Public buffer pools:
Small buffers, 104 bytes (total 201, permanent 200, peak 201 @ 00:11:47):
197 in free list (50 min, 250 max allowed)
174601 hits, 12 misses, 35 trims, 36 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 200, permanent 200):
198 in free list (50 min, 250 max allowed)
924631 hits, 10 misses, 30 trims, 30 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 200, permanent 200):
200 in free list (50 min, 250 max allowed)
149483 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 11 @ 20:18:12):
10 in free list (0 min, 100 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 10, permanent 10, peak 210 @ 00:11:25):
10 in free list (5 min, 10 max allowed)
0 hits, 0 misses, 18 trims, 18 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 5, permanent 0, peak 5 @ 20:18:12):
5 in free list (4 min, 10 max allowed)
0 hits, 0 misses, 14 trims, 19 created
0 failures (0 no memory)

Interface buffer pools:
CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 20:18:28):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 7 trims, 8 created
0 failures (0 no memory)
CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 20:18:28):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 7 trims, 8 created
0 failures (0 no memory)
Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ 20:18:07):
101 in free list (132 min, 132 max allowed)
168 hits, 0 misses
IPMUX SF buffers, 1500 bytes (total 500, permanent 500):
500 in free list (0 min, 1000 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 20:18:28):
26 in free list (25 min, 50 max allowed)
0 hits, 0 misses, 7 trims, 8 created
0 failures (0 no memory)
pak coalesce engine buffers, 1536 bytes (total 522, permanent 256, peak 522 @ 01:17:35):
9 in free list (0 min, 512 max allowed)
261605 hits, 136 misses, 6 trims, 272 created
0 failures (0 no memory)
512 max cache size, 512 in cache
253257 hits in cache, 0 misses in cache
SEC Eng Packet buffers, 1700 bytes (total 1024, permanent 1024):
0 in free list (0 min, 1024 max allowed)
1024 hits, 0 fallbacks
1024 max cache size, 1024 in cache
0 hits in cache, 0 misses in cache
IPC buffers, 4096 bytes (total 2, permanent 2):
1 in free list (1 min, 8 max allowed)
1 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 20:18:28):
3 in free list (2 min, 4 max allowed)
0 hits, 0 misses, 7 trims, 8 created
0 failures (0 no memory)
CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 20:18:28):
2 in free list (1 min, 2 max allowed)
0 hits, 0 misses, 7 trims, 8 created
0 failures (0 no memory)
IPC Medium buffers, 16384 bytes (total 2, permanent 2):
2 in free list (1 min, 8 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 20:18:42):
17 in free list (16 min, 16 max allowed)
0 hits, 0 misses, 1218 trims, 1219 created
0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 768, permanent 768):
256 in free list (128 min, 1024 max allowed)
512 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
512 max cache size, 512 in cache
84996 hits in cache, 0 misses in cache

Particle Clones:
1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 256 bytes (total 768, permanent 768):
256 in free list (128 min, 1024 max allowed)
512 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
512 max cache size, 512 in cache
0 hits in cache, 0 misses in cache
Normal buffers, 1548 bytes (total 3840, permanent 3840):
3840 in free list (128 min, 4096 max allowed)
896 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)

Private particle pools:
HQF Particle buffers, 0 bytes (total 2000, permanent 2000):
2000 in free list (500 min, 2000 max allowed)
378558 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
IDS SM buffers, 240 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
128 hits, 0 fallbacks
128 max cache size, 128 in cache
0 hits in cache, 0 misses in cache
SEC Eng Particle Header buffers, 256 bytes (total 1024, permanent 1024):
0 in free list (0 min, 1024 max allowed)
1024 hits, 0 fallbacks
1024 max cache size, 1024 in cache
0 hits in cache, 0 misses in cache
IPMUX particle pool buffers, 512 bytes (total 500, permanent 500):
0 in free list (0 min, 1000 max allowed)
500 hits, 1 misses
1000 max cache size, 500 in cache
0 hits in cache, 0 misses in cache
GigabitEthernet0/0 buffers, 1664 bytes (total 768, permanent 768):
0 in free list (0 min, 768 max allowed)
768 hits, 0 fallbacks
768 max cache size, 511 in cache
101226237 hits in cache, 0 misses in cache
GigabitEthernet0/1 buffers, 1664 bytes (total 768, permanent 768):
0 in free list (0 min, 768 max allowed)
768 hits, 0 fallbacks
768 max cache size, 512 in cache
248587186 hits in cache, 0 misses in cache
GigabitEthernet0/2 buffers, 1664 bytes (total 768, permanent 768):
0 in free list (0 min, 768 max allowed)
768 hits, 0 fallbacks
768 max cache size, 512 in cache
12843 hits in cache, 0 misses in cache
SEC Eng Particle buffers, 1700 bytes (total 1024, permanent 1024):
0 in free list (0 min, 1024 max allowed)
1024 hits, 0 misses
1024 max cache size, 1024 in cache
0 hits in cache, 0 misses in cache

#sh controller g0/0
Interface GigabitEthernet0/0
Hardware is PQUICC3 M8572E
ADDR: 0x1C5C914, FASTSEND: 0x401C114
Route Cache Flag: 0x11

= Media RJ45 =

PHY REGISTERS:
Display for CRC check for page:6

PACKET_GENERATION reg_addr: 16: 0x0010
CRC COUNTERS reg_addr: 17: 0xFF00

page number:0

Copper Contrl @ reg_addr: 0: 0x1140
Copper Status @ reg_addr: 1: 0x796D
PHY Idf1 @ reg_addr: 2: 0x0141
PHY Idf2 @ reg_addr: 3: 0x0C97
Autoneg Adv @ reg_addr: 4: 0x0DE1
Autoneg Lnk Ptr abty @ reg_addr: 5: 0xC1E1
Autoneg Exp @ reg_addr: 6: 0x000D
Autoneg Nx Pg Tx @ reg_addr: 7: 0x2801
Autoneg Lk Prt Nx Pg @ reg_addr: 8: 0x4E1E
1000B-T Ctr @ reg_addr: 9: 0x0E00
1000B-T St @ reg_addr: 10: 0x3800
Copper Ext St @ reg_addr: 15: 0x3000
Copper Specific Ctl1 @ reg_addr: 16: 0x6060
Copper Specific St1 @ reg_addr: 17: 0xAC48
Copper Interrupt En @ reg_addr: 18: 0x0000
Copper Specific St2 @ reg_addr: 19: 0x0000
Copper Rcv Er Ctr @ reg_addr: 21: 0x0000
Copper Specific Ctl2 @ reg_addr: 26: 0x8040

page number:1

Fiber Contrl @ reg_addr: 0: 0x1140
Fiber Status @ reg_addr: 1: 0x794D
PHY Idf1 @ reg_addr: 2: 0x0141
PHY Idf2 @ reg_addr: 3: 0x0C97
Autoneg Adv @ reg_addr: 4: 0x0060
Autoneg Lnk Ptr abty @ reg_addr: 5: 0x0000
Autoneg Exp @ reg_addr: 6: 0x0004
Autoneg Nx Pg Tx @ reg_addr: 7: 0x2001
Autoneg Lk Prt Nx Pg @ reg_addr: 8: 0x0000
Fiber Ext St @ reg_addr: 15: 0x3000
Fiber Specific Ctl1 @ reg_addr: 16: 0x0000
Fiber Specific St1 @ reg_addr: 17: 0x0018
Fiber Interrupt En @ reg_addr: 18: 0x0000
Fiber Specific St2 @ reg_addr: 19: 0x0000
Fiber Rcv Er Ctr @ reg_addr: 21: 0x0000
Fiber Specific Ctl2 @ reg_addr: 26: 0x8085

page number:2

MAC Contrl @ reg_addr: 0: 0x0040
PHY Idf1 @ reg_addr: 2: 0x0141
PHY Idf2 @ reg_addr: 3: 0x0C97
MAC Specific Ctl1 @ reg_addr: 16: 0x0288
MAC Specific St1 @ reg_addr: 17: 0x0440
MAC Interrupt En @ reg_addr: 18: 0x0000
MAC Specific St2 @ reg_addr: 19: 0x0000
MAC Specific Ctl2 @ reg_addr: 26: 0x0005

MPC8500 TSEC Registers:
IADDR0 : 0x00000000, IADDR1 : 0x00000000
IADDR2 : 0x00000000, IADDR3 : 0x00000000
IADDR4 : 0x00000000, IADDR5 : 0x00000000
IADDR6 : 0x00000000, IADDR7 : 0x40000000
GADDR0 : 0x00000000, GADDR1 : 0x00000004
GADDR2 : 0x00001000, GADDR3 : 0x00000000
GADDR4 : 0x01000000, GADDR5 : 0x20000000
GADDR6 : 0x00000000, GADDR7 : 0x00000001
IEVENT : 0x00000000, IMASK : 0xB3708180
EDIS : 0x00000000, ECNTRL : 0x00001022
MINFLR : 0x00000000, PTV : 0x00000060
DMACTRL : 0x000000C1, TBIPA : 0x00000011
TSTAT : 0x80008000, TBDLEN : 0x00000000
CTBPTR : 0x00000000, TBPTR : 0x3CFB3818
TBASE : 0x3CFB32C0, OSTBD : 0x00000000
OSTBDP : 0x00000000, RCTRL : 0x00000004
RSTAT : 0x00000080, RBDLEN : 0x00000000
CRBPTR : 0x00000000, MRBLR : 0x00000600
MACCFG1 : 0x0000002F, MACCFG2 : 0x00007225
RBASE : 0x3CFB2A80, IPGIFG : 0x40605060
HAFDUP : 0x0001F037, MAXFRML : 0x00002580
MIIMCFG : 0x00000015, MIIMCOM : 0x00000000
MIIMADD : 0x00000300, MIIMCON : 0x00000000
MIIMSTAT : 0x0000FFFF, MIIMIND : 0x00000000
IFSTAT : 0x00000008, TCTRL : 0x00000000
ATTR : 0x000048C0, ATTRELI : 0x00500000
MACSTNADDR1 : 0x80CC6CF6
MACSTNADDR2 : 0x697C0000
FIFO_TX_THR : 0x00000100
FIFO_TX_STARVE : 0x000000E0
FIFO_STARVE_SHUTOFF : 0x00000100
FIFO_PAUSE_CTRL: 0x00000000
============RMON MIB Registers==============
TR64=2055300, TR127=2624725
TR255=1156119, TR511=1464021
TR1K=3744253, TRMAX=941666
TRMGV=0
RBYT=2432586826, RPKT=1693178
RFCS=0, RMCA=0
RBCA=9, RXCF=0
RXPF=0, RXUO=0
RALN=0, RFLR=0
RCDE=0, RCSE=0
RUND=1, ROVR=0
RFRG=0, RJBR=0
RDRP=28887
TBYT=1497944823, TPKT=1904299
TMCA=1479, TBCA=6
TXPF=0, TDFR=0
TEDF=0, TSCL=0
TMCL=0, TLCL=0
TNCL=0, TDRP=0
TJBR=0, TFCS=0
TXCF=0, TOVR=0
TUND=0, TFRG=0
CAR1=0x00000000, CAR2=0x00003000
CAM1=0xFE03FFFF, CAM2=0x000FFFFD

Software MAC Address Filter (hash:length/addr/mask/hits)
--------------------------------------------------------
0x000: 0 ffff.ffff.ffff 0000.0000.0000 5
0x076: 0 7c69.f66c.cc80 0000.0000.0000 101934361
0x0C0: 0 0100.0ccc.cccc 0000.0000.0000 0
0x0C0: 1 0180.c200.0002 0000.0000.0000 0
0x0C5: 0 0180.c200.0007 0000.0000.0000 0
0x0CC: 0 0180.c200.000e 0000.0000.0000 0

Software filtered frames: 0
Promiscuous mode: 0

Statistics:
Rx Bytes 2432586826 Tx Bytes 1497945099
Rx Good Packets 1693177 Tx Good Packets 1904301
Rx Multicast 0
Rx Broadcast 9

Rx Bad Pkt Errors 1 Tx Bad Pkt Errors 0
Rx FCS Errors 0 Tx FCS Errors 0
Rx Runt Errors 1 Tx Runt Errors 0
Rx Oversize Errors 0 Tx Oversize Errors 0
Rx Length Errors 0 Tx Collisions 0
Rx Code Errors 0 Tx Late Collisions 0
Rx Dribble Errors 0 Tx Excessive Collisions 0
Tx Abort Errors 0

Port Stopped: N

Internal Loopback Set: N

Internal Driver Information:
throttled=0, enabled=0, disabled=0
rx_coalesce_failed=0, rx_framing_err=0, rx_overflow_err=719619, rx_buffer_err=0
rx_no_enp=0, rx_discard=0
tx_one_col_err=0, tx_more_col_err=0, tx_no_enp=0, tx_deferred_err=0
tx_underrun_err=0, tx_late_collision_err=0, tx_loss_carrier_err=0
tx_exc_collision_err=0, tx_buff_err=0, fatal_tx_err=0

Ring Buffer Information:
pq3_tsec_instance=0x1C5D700
rx ring entries=256, tx ring entries=256
rxring=0x3CFB2A80, rxr shadow=0x1C5DBA0, rx_head=120, rx_tail=0
txring=0x3CFB32C0, txr shadow=0x13AC9C84, tx_head=217, tx_tail=217, tx_count=0
Driver Level Counters: (Cumulative, Zeroed only at Reset)
Rx Frames=101927547, Bytes=1258377525
Tx Frames=48039129, Bytes=1275424370
tx_ring_limit_min = 64 tx_ring_limit_max = 256 tx_ring_limit_current = 240

The following two commands are with in a second of each other

#sh controller g0/0 | i rx_overflow_err
rx_coalesce_failed=0, rx_framing_err=0, rx_overflow_err=80110, rx_buffer_err=0
#sh controller g0/0 | i rx_overflow_err
rx_coalesce_failed=0, rx_framing_err=0, rx_overflow_err=80598, rx_buffer_err=0

rothomas2 · ‎07-17-2014

Well for 22mbps you do have considerable packet per second around 5K. Meaning your packets are rather small, probably some UDP traffic. I don't think you should underestimate 22mbps traffic on the 30 second average. 1 second is just so much time, when you think about the fraction of ms.

If you have a few packets arriving at the interface on the same ms fraction, and the input 'bus' is not able to read them fast enough to put them in memory buffer, packets will have to be dropped as an input drop. You can have as much buffer memory as you want, this wont help.

Consider this is real 'demand' traffic and depends a lot on usage pattern' You can very well get 1gbps of TCP packets without any input errors with iperf, if your packet arrive at the correct time.

Do the same test, with UDP and random test arrivals, and very small packets, and with not so much traffic on the average you can start increasing your input errors.

The solution by the book for this is 'port agregation' maybe you want to give that a try and see if your error count lowers or disappear.

ylingf · ‎07-18-2014

I agree microburst traffic could be the cause here. The 3948 is rated at 982 kpps with 64 byte packets. On one hand, we are not even at 1% of the rated capacity here based on pps. On the other hand, 982k of 64 byte packets plus 20 bytes overhead, is about 660Mbps. This means that if the traffic is coming in at line rate for 64 byte packets, there will be a lot of drops. My conclusion is that the 3945 platform is not capable of processing small packets at gigabit line rate.

Joseph W. Doherty · ‎07-18-2014

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Just to add to ylingf's post, the 660 Mbps would be best case, i.e. router doing nothing but simple packet forwarding. Cisco recommends the 3945 for up to 150 Mbps of WAN traffic.

What you might try is running the gig interface at 100, this will avoid gig bursts over running the capacity of the router.

rothomas2 · ‎07-18-2014

Almost no platform is capable of doing this micropacket processing. Still its not common. Cisco rates their products with packets size of 64bytes which turns into very small traffic rates. In real deployment you will much larger packet sizes that 64bytes almost 8 to 10 times larger.

The end result of this, is people oversizing their routing requirements everywhere.

If you want true performance, and line rate routing either the 65XX, 76XX or 3750 are just best choice.