Interface Fa Throttles/rx_soft_overflow_err increasing -> buffer issue?

Dario Tigani · ‎07-15-2011

Hello,

I'm seeing an increase of Throttles and/or rx_soft_overflow_err on a Fastethernet interface.

I did some research and can see alot of references of this being caused by input errors.

Yet I'm not seeing any other counter increase on this interface but the two mentioned above.

I also checked show buffers and the buffers used to drop some packets from time to time it seems but I'm not sure what could be causing this.

If anyone has any idea, it would be very much appriciated.

Here's the related output:

=========================================INTERFACE================================================

ROUTER#sh int fa0/0

FastEthernet0/0 is up, line protocol is up

Hardware is Gt96k FE, address is dc7b.9495.bbfe (bia dc7b.9495.bbfe)

Description: LAN

Internet address is 10.104.80.254/24

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 00:29:14

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/100 (size/max)

5 minute input rate 59000 bits/sec, 20 packets/sec

5 minute output rate 167000 bits/sec, 20 packets/sec

53862 packets input, 14330135 bytes

Received 2832 broadcasts, 0 runts, 0 giants, 2 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog

0 input packets with dribble condition detected

58097 packets output, 40280849 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

29 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

=========================================CONTROLLER================================================

ROUTER#sh controllers fa0/0
Interface FastEthernet0/0
Hardware is GT96K FE ADDR: 64DD878C, FASTSEND: 62819414, MCI_INDEX: 0
DIST ROUTE ENABLED: 0Route Cache Flag: 11
GPIO 2 CONF= 67FF07A8 GPIO 2 IO= 187D303F CIU arbit = 80F002BF
PHY add register = 0xA4 PHY data register = 0xC440040
Port Conf Reg= 0x8080 ENABLE HT8K HMOD0 FDX
Port Conf Ex Reg= 0x1CCF00
TX1:1 RXPRI=DE(00) ~DPLXAUTO ~FLCNTL ~FLNKP MFL64KB FE ~SPDAUTO RMII
Port Com Reg= 0x0
Port Status Reg= 0xB 100MB FDPX FCTL EN LNK UP ~PAUSED TX oFF
Serial Param Reg= 0x218823 Hash table pointer= 0xE8A5B60
Source ADDR L= 0xBBFE Source ADDR H= 0xDC7B9495
SDMA conf reg= 0x223C RETX 15 RX BE TX BE FRINT BSIZE 4
SDMA com reg= 0x1010080 SRT TXL STP TXH EN RX
IMASK= 0x90003DCD ICause= 0x0
Serial 0 mask 30FFFC3Serial 0 cause 0
IpDiffservP0L= 0x0 IpDiffservP0H= 0x0 IpDiffservP1L= 0x0 IpDiffservP1H= 0x0
IP VLAN TAG PRI= 0xF0CC IP VLAN TAG PRI= 0xF0CC
First rxd Q0= 0xE8E5D50 Curr rxd Q0= 0xE8E5D50
First rxd Q1= 0xE8E6040 Curr rxd Q1= 0xE8E6040
First rxd Q2= 0xE8E64A0 Curr rxd Q2= 0xE8E64A0
First rxd Q3= 0xE8E6900 Curr rxd Q3= 0xE8E6900
First txd Q0= 0xE8E72B0 First txd Q1= 0xE8E75A0
gt96kfe_instance=0x64DD98D4, registers=0x24084800
RxRing entries=64, tx ring entries=128
RxR0=0x E8E5BE0, RxR1=0x E8E6040, RxR2=0x E8E64A0, RxR3=0x E8E6900
Malloc RxR0=0xEE8E5BE0, RxR1=0xEE8E6040, RxR2=0xEE8E64A0, RxR3=0xEE8E6900
SDOW RxR0=0x64DD9F0C, RxR1=0x64DDA040, RxR2=0x64DDA174, RxR3=0x64DDA2A8
HEAD RxR0=0x17, RxR1=0x0, RxR2=0x0, RxR3=0x0
TAIL RxR0=0x0, RxR1=0x0, RxR2=0x0, RxR30x0

tx_limited=0(128)
TxR0=0x E8E6D60, TxR1=0x E8E75A0
COUNT TxR0=0x0, TxR1=0x0
Head TxR0=0x3F, TxR1=0x0 Tail TxR0=0x3F, TxR1=0x0

PHY registers:
Register 0x00:   2100 780D 0040 61E4 0101 0000 0004 2001
Register 0x08:   0000 0000 0000 0000 0000 0000 0000 0000
Register 0x10:   1000 0300 0000 0000 0200 0E5E 0600 0000
Register 0x18:   0037 000D 9F00 008A 0827 0000 0000
Bytes_recvd 340788650 Bytes_sent 848651672 Frames_recvd 1322173 Frames_sent 1303600
total_bytes_RX 340798876 Total_frames_RX 1322225 Bcast_frames_recvd 46231
Mcast_frames_RX 54171 CRC_err 0 Ovr_sized_frames 0
Fragments 0 Jabber 0 collision 0
Late_collision 0 64B frame 796127; 65_127B_frames 425062
128_255B_frames 406157 256_511B_frames 292631 512_1023B_frames 92921
1023_maxB_frames 612990 Rx_error 0 Dropped_frames 0
Mcast_frames_tx 1783 Bcast_frames_tx 1142 Sml_frame_recvd 0

Software MAC address filter(hash:length/addr/mask/hits):
0x00: 0 ffff.ffff.ffff 0000.0000.0000         0
0x6A: 0 dc7b.9495.bbfe 0000.0000.0000         0
0xC0: 0 0100.0ccc.cccc 0000.0000.0000         0
0xC0: 1 0180.c200.0002 0000.0000.0000         0
0xC5: 0 0180.c200.0007 0000.0000.0000         0

======= Driver Counters =======
Number of Transmitter Hang = 0
tx_more_col_err= 0 tx_one_col_err= 0
tx_exc_collision_err= 0 tx_late_collision_err= 0
tx_underrun_err= 0 tx_error_intr= 0
rx_soft_overflow_err= 28 rx_overflow_err= 0

=========================================BUFFERS================================================

ROUTER#show buffers
Buffer elements:
1031 in free list (1119 max allowed)
660370 hits, 0 misses, 619 created

Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50, peak 59 @ 1d03h):
     48 in free list (20 min, 150 max allowed)
     294074 hits, 3 misses, 9 trims, 9 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 31, permanent 25, peak 43 @ 05:32:52):
     27 in free list (10 min, 150 max allowed)
     1851909 hits, 41 misses, 60 trims, 66 created
     1 failures (0 no memory)
Big buffers, 1536 bytes (total 79, permanent 50, peak 82 @ 05:03:50):
     79 in free list (5 min, 150 max allowed)
     711831 hits, 83 misses, 8 trims, 37 created
     53 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
     10 in free list (0 min, 100 max allowed)
     21 hits, 32 misses, 0 trims, 0 created
     32 failures (0 no memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 1 @ 06:22:30):
     1 in free list (0 min, 10 max allowed)
     0 hits, 32 misses, 2 trims, 3 created
     32 failures (0 no memory)
Huge buffers, 18024 bytes (total 3, permanent 0, peak 12 @ 1d00h):
     3 in free list (0 min, 4 max allowed)
     184185 hits, 87 misses, 109 trims, 112 created
     31 failures (0 no memory)

Interface buffer pools:
Syslog ED Pool buffers, 600 bytes (total 132, permanent 132):
     113 in free list (132 min, 132 max allowed)
     23 hits, 0 misses
IPC buffers, 4096 bytes (total 2, permanent 2):
     2 in free list (1 min, 8 max allowed)
     0 hits, 0 fallbacks, 0 trims, 0 created
     0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 768, permanent 768):
     256 in free list (128 min, 1024 max allowed)
     512 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     158 hits in cache, 0 misses in cache

Particle Clones:
1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 256 bytes (total 768, permanent 768):
     256 in free list (128 min, 1024 max allowed)
     512 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     1194893 hits in cache, 0 misses in cache
Normal buffers, 1548 bytes (total 768, permanent 768):
     768 in free list (128 min, 1024 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)

Private particle pools:
HQF buffers, 0 bytes (total 2000, permanent 2000):
     2000 in free list (500 min, 2000 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
IDS SM buffers, 240 bytes (total 128, permanent 128):
     0 in free list (0 min, 128 max allowed)
     128 hits, 0 fallbacks
     128 max cache size, 128 in cache
     0 hits in cache, 0 misses in cache
Serial0/0/0 buffers, 1536 bytes (total 64, permanent 64):
     0 in free list (0 min, 64 max allowed)
     64 hits, 0 fallbacks
     64 max cache size, 32 in cache
     1513129 hits in cache, 0 misses in cache
Serial0/0/1 buffers, 1536 bytes (total 64, permanent 64):
     0 in free list (0 min, 64 max allowed)
     64 hits, 0 fallbacks
     64 max cache size, 32 in cache
     32 hits in cache, 0 misses in cache
FastEthernet0/0 buffers, 1552 bytes (total 512, permanent 512):
     0 in free list (0 min, 512 max allowed)
     512 hits, 0 fallbacks
     512 max cache size, 256 in cache
     1276252 hits in cache, 0 misses in cache
FastEthernet0/1 buffers, 1552 bytes (total 512, permanent 512):
     0 in free list (0 min, 512 max allowed)
     512 hits, 0 fallbacks
     512 max cache size, 256 in cache
     256 hits in cache, 0 misses in cache

smogra · ‎07-16-2011

Please paste the sh ver output as well.

The basic point is confirmed that rx-overflow-error shows the traffic come in and the
switch is not able to handle them.
Several points are:
1/.  The switch was not able to switch the packets quick enough; 
2/.  The output rate is less than input rate so the packets were piled up in 
output interface; 
3/.  The clock btw the router bus and the FE NM is not quick enough to rx 
traffic quick enough.

The suggestion would be:
1/.  Check if cef is enabled in the router; Show cef interface Show cef drop Show ip
traffic ( twice )
2/.  Did you see the problem before ?  

When the packet comes to the switch , the Interface temporarily stores a packet that 
it has accepted in a hardware FIFO queue.  It then DMA's the packet into into the RX
Ring.Before the interface can DMA the packet it first needs to find a free descriptor.  
It scans the RX Ring looking for a descriptor that has the OWN bit set.  Once one is 
found, the OWN bit is cleared (Now the CPU owns that Descriptor), the data is DMA'd 
into the descriptors corresponding buffer/particle, and the START status is set. 
rx-soft-overflow means that there was no descriptor available on the rx-ring to put 
the packet on and thats where the rx-soft-overflow counter comes into play.

Some of the  reasons for that are :

1) High CPU Utilization (not able to service RX Interrupt quickly)
2) Large burst of packets (lot's of frames hit controller filling the RX ring 
before the CPU can clear the ring)
3) Buffers/particles are not available (the CPU tried to replace a RX ring 
buffer  but was not able to allocate one)
4) The CPU is taking too long within the RX Interrupt (too many switching 
decisions, features that slow it down, etc)

Cheers

Sweta

*Please rate the answer if the content was useful.

Dario Tigani · ‎07-17-2011

Hi Sweta,

Here's the show version:

ROUTER#sh ver
Cisco IOS Software, 1841 Software (C1841-ADVSECURITYK9-M), Version 12.4(22)T5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 28-Apr-10 03:54 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

BienHoa uptime is 4 days, 2 hours, 46 minutes
System returned to ROM by power-on
System restarted at 11:08:16 ICT Thu Jul 14 2011
System image file is "flash:c1841-advsecurityk9-mz.124-22.T5.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 1841 (revision 7.0) with 236544K/25600K bytes of memory.
Processor board ID FHK143771WX
2 FastEthernet interfaces
2 Serial(sync/async) interfaces
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity disabled.
191K bytes of NVRAM.
62720K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

Dario Tigani · ‎07-18-2011

I checked the output queue and processor utilization during the times when I see the throttles increase.

The cpu utilization is not abnormally high

The output queue is not full

Also, CEF is enabled for this interface.

Here's the output of show cef drop and show ip traffic in case it could help with troubleshooting;

=========================

ROUTER#sh ip traffic
IP statistics:
Rcvd: 2453864 total, 420309 local destination
         0 format errors, 0 checksum errors, 175 bad hop count
         0 unknown protocol, 1 not a gateway
         0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other
Frags: 503031 reassembled, 330 timeouts, 0 couldn't reassemble
         0 fragmented, 0 fragments, 0 couldn't fragment
Bcast: 43185 received, 352 sent
Mcast: 0 received, 0 sent
Sent: 421075 generated, 7594571 forwarded
Drop: 86 encapsulation failed, 0 unresolved, 0 no adjacency
         0 no route, 0 unicast RPF, 0 forced drop
         0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
         0 physical broadcast

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 19546 unreachable
        17615 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
        1 time exceeded, 0 info replies
Sent: 0 redirects, 43412 unreachable, 76 echo, 17615 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
        0 info reply, 348 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
Rcvd: 316474 total, 0 checksum errors, 387 no port
Sent: 175676 total

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

IP-EIGRP statistics:
Rcvd: 0 total
Sent: 0 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0

UDP statistics:
Rcvd: 66673 total, 0 checksum errors, 42020 no port
Sent: 25225 total, 0 forwarded broadcasts

OSPF statistics:
Rcvd: 0 total, 0 checksum errors
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

Sent: 0 total
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

ARP statistics:
Rcvd: 124204 requests, 612 replies, 0 reverse, 0 other
Sent: 4280 requests, 30210 replies (666 proxy), 0 reverse
Drop due to input queue full: 0
ROUTER#sh ip traffic
IP statistics:
Rcvd: 2453975 total, 420364 local destination
         0 format errors, 0 checksum errors, 175 bad hop count
         0 unknown protocol, 1 not a gateway
         0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other
Frags: 503045 reassembled, 330 timeouts, 0 couldn't reassemble
         0 fragmented, 0 fragments, 0 couldn't fragment
Bcast: 43189 received, 352 sent
Mcast: 0 received, 0 sent
Sent: 421121 generated, 7595499 forwarded
Drop: 86 encapsulation failed, 0 unresolved, 0 no adjacency
         0 no route, 0 unicast RPF, 0 forced drop
         0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
         0 physical broadcast

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 19546 unreachable
        17615 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
        1 time exceeded, 0 info replies
Sent: 0 redirects, 43414 unreachable, 76 echo, 17615 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
        0 info reply, 348 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
Rcvd: 316525 total, 0 checksum errors, 387 no port
Sent: 175680 total

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

IP-EIGRP statistics:
Rcvd: 0 total
Sent: 0 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0

UDP statistics:
Rcvd: 66677 total, 0 checksum errors, 42024 no port
Sent: 25225 total, 0 forwarded broadcasts

OSPF statistics:
Rcvd: 0 total, 0 checksum errors
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

Sent: 0 total
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

ARP statistics:
Rcvd: 124225 requests, 612 replies, 0 reverse, 0 other
Sent: 4280 requests, 30213 replies (666 proxy), 0 reverse
Drop due to input queue full: 0
ROUTER#sh cef drop
% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Drop Statistics
Slot Encap_fail Unresolved Unsupported No_route No_adj ChkSum_Err
RP 0 0 671241 0 819 1
ROUTER#show ip cef switching stat

       Reason                          Drop       Punt Punt2Host
RP LES Packet destined for us             0     385165       1592
RP LES No adjacency                     819          0          0
RP LES Bad checksum                       1          0          0
RP LES TTL expired                        0          0        150
RP LES Features                        4578          0     670738
RP LES Neighbor resolution req         2809        378          0
RP LES Total                           8207     385543     672480

All Total 8207 385543 672480
ROUTER#