overrun errors ASR1006

[Lennon]

Hello everyone!

I've got an ASR1006(ESP40, SIP10) router as a border running BGP, NAT and other services on it. As it has two uplinks with max traffic of 8Gb/s and 6Gb/s I observe a lot of overrun input errors on interface that increases 6-7Gb/s traffic. I've balanced traffic to other interface, changed SIP, ESP, and even 10G XFP optical transceiver. Made no sense, overrun errors apper on interface with traffic over 6Gb/s. Please help what could be the reason of this problem and what issue it delivers  to users connections?

[Georg Pauwen]

Hello,

post the output of:

show buffers

show interfaces x (where 'x' is the interface that has the input/overrun errors)

[Lennon]

show buffers
Buffer elements:
1644 in free list
672726751 hits, 0 misses, 1019 created

Public buffer pools:
Small buffers, 104 bytes (total 1200, permanent 1200):
1197 in free list (200 min, 2500 max allowed)
387733266 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 900, permanent 900, peak 5767 @ 7w0d):
898 in free list (100 min, 2000 max allowed)
811621182 hits, 87168 misses, 261445 trims, 261445 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 900, permanent 900, peak 904 @ 7w0d):
900 in free list (50 min, 1800 max allowed)
197688532 hits, 8 misses, 4 trims, 4 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 100, permanent 100, peak 101 @ 7w0d):
100 in free list (0 min, 300 max allowed)
11438774 hits, 58 misses, 1 trims, 1 created
58 failures (0 no memory)
Large buffers, 5024 bytes (total 100, permanent 100, peak 101 @ 7w0d):
100 in free list (0 min, 300 max allowed)
58 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
VeryLarge buffers, 8256 bytes (total 100, permanent 100):
100 in free list (0 min, 300 max allowed)
846231 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 20, permanent 20, peak 21 @ 7w0d):
20 in free list (0 min, 33 max allowed)
1222552 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)

Interface buffer pools:
CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 7w0d):
101 in free list (100 min, 200 max allowed)
7 hits, 0 misses, 2197 trims, 2198 created
0 failures (0 no memory)
BFD Private Pool buffers, 256 bytes (total 1024, permanent 1024):
512 in free list (128 min, 1024 max allowed)
512 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
512 max cache size, 512 in cache
73219829 hits in cache, 0 misses in cache
Generic ED Pool buffers, 512 bytes (total 101, permanent 100, peak 101 @ 7w0d):
101 in free list (100 min, 100 max allowed)
0 hits, 0 misses
CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 7w0d):
101 in free list (100 min, 200 max allowed)
5640 hits, 0 misses, 2197 trims, 2198 created
0 failures (0 no memory)
Syslog ED Pool buffers, 600 bytes (total 1057, permanent 1056, peak 1057 @ 7w0d):
1025 in free list (1056 min, 1056 max allowed)
369041 hits, 0 misses
EOBC0 buffers, 1524 bytes (total 256, permanent 256):
256 in free list (0 min, 256 max allowed)
78931151 hits, 0 fallbacks
CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 7w0d):
26 in free list (25 min, 50 max allowed)
299 hits, 0 misses, 2197 trims, 2198 created
0 failures (0 no memory)
IPC buffers, 4096 bytes (total 6552, permanent 6552):
6481 in free list (2184 min, 21840 max allowed)
37700178 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 7w0d):
3 in free list (2 min, 4 max allowed)
10 hits, 0 misses, 2197 trims, 2198 created
0 failures (0 no memory)
CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 7w0d):
2 in free list (1 min, 2 max allowed)
0 hits, 0 misses, 2197 trims, 2198 created
0 failures (0 no memory)
IPC Medium buffers, 16384 bytes (total 2, permanent 2):
2 in free list (1 min, 8 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
Private Huge IPC buffers, 18024 bytes (total 1, permanent 0, peak 1 @ 7w0d):
1 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 2197 trims, 2198 created
0 failures (0 no memory)
Private Huge buffers, 65280 bytes (total 1, permanent 0, peak 1 @ 7w0d):
1 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 2197 trims, 2198 created
0 failures (0 no memory)
IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 7w0d):
17 in free list (16 min, 16 max allowed)
0 hits, 0 misses, 355136 trims, 355137 created
0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 266, permanent 256, peak 266 @ 7w0d):
10 in free list (10 min, 512 max allowed)
253 hits, 3 misses, 0 trims, 10 created
0 failures (0 no memory)
256 max cache size, 256 in cache
208679001 hits in cache, 0 misses in cache

Particle Clones:
1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 256 bytes (total 384, permanent 384):
128 in free list (128 min, 1024 max allowed)
256 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
256 max cache size, 256 in cache
0 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 512, permanent 512):
384 in free list (128 min, 1024 max allowed)
128 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
128 max cache size, 128 in cache
0 hits in cache, 0 misses in cache

Private particle pools:
lsmpi_rx buffers, 416 bytes (total 8194, permanent 8194):
0 in free list (0 min, 8194 max allowed)
8194 hits, 0 misses
8194 max cache size, 0 in cache
340249188 hits in cache, 0 misses in cache
lsmpi_tx buffers, 416 bytes (total 4098, permanent 4098):
0 in free list (0 min, 4098 max allowed)
4098 hits, 0 misses
4098 max cache size, 4092 in cache
213940587 hits in cache, 0 misses in cache

show interfaces tenGigabitEthernet 1/0/0
TenGigabitEthernet1/0/0 is up, line protocol is up
Hardware is SPA-1X10GE-L-V2, address is c47d.4fb6.e040 (bia c47d.4fb6.e040)
Description: --
Internet address is --
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 20/255, rxload 101/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 10000Mbps, link type is force-up, media type is 10GBase-ER
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:03:41, output hang never
Last clearing of "show interface" counters 1w1d
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 3973142000 bits/sec, 405157 packets/sec
30 second output rate 799500000 bits/sec, 292713 packets/sec
301144229811 packets input, 375457380695735 bytes, 0 no buffer
Received 159213 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
473512631 input errors, 0 CRC, 0 frame, 473512631 overrun, 0 ignored
0 watchdog, 742146 multicast, 0 pause input
245107590709 packets output, 89493640422609 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
148146 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Georg Pauwen]

Hello,

try and increase the middle buffers to:

buffers middle permanent 1800
buffers middle min-free 200
buffers middle max-free 4000

That said, input errors could also indicate a physical problem with the cabling and/or the port. Do you have a chance to change out the cable, or use another port ?

[Lennon]

Hello,

i've changed these settings but overrun errors keep increasing:

18920897 input errors, 0 CRC, 0 frame, 18920897 overrun, 0 ignored

Cable, optical transceiver, SIP and SPA i've changed earlier and it didn't work. 

Also I see that SIP buffers are fully filled on interface to which I allocate most amount of traffic:

sh platform hardware slot 1 plim buffer settings
Interface 1/0/0
RX Low
Buffer Size 28901376 Bytes
Drop Threshold 28900416 Bytes
Fill Status Curr/Max 28372992 Bytes / 28372992 Bytes
TX Low
Interim FIFO Size 192 Cache line
Drop Threshold 109248 Bytes
Fill Status Curr/Max 2048 Bytes / 101376 Bytes
RX High
Buffer Size 4128768 Bytes
Drop Threshold 4127424 Bytes
Fill Status Curr/Max 0 Bytes / 0 Bytes
TX High
Interim FIFO Size 192 Cache line
Drop Threshold 109248 Bytes
Fill Status Curr/Max 0 Bytes / 0 Bytes

Any ideas how to fix it?

[Joseph W. Doherty]

". . . what could be the reason of this problem . . ."

Assuming it's not a "software defect", classically it indicates your interface hardware cannot keep up with the ingress data rate (which tracks with your mention of the issue arises when ingress rate begins to exceed 6 Gbps and worsens as the ingress rate increases).

". . . what issue it delivers  to users connections . . ."

The "usual" related to packets being dropped.

"As it has two uplinks with max traffic of 8Gb/s and 6Gb/s I observe a lot of overrun input errors on interface that increases 6-7Gb/s traffic."

You only have just one SPA, with a 10g port, in a SIP-10, correct?

[Lennon]

Can anyone help?

[Georg Pauwen]

Hello,

I looked around, and although you could QoS (link below), I don't really think this would solve your problem (which basically is ingress oversubscription).

Can you post the full running configuration sh run) of your router, as well as the output of 'sh ver' ?

Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Software Configuration Guide

https://www.cisco.com/c/en/us/td/docs/interfaces_modules/shared_port_adapters/configuration/ASR1000/asr1000-sip-spa-book/asr-spa-pkt-class.html