7206 VXR NPE-G2 input/ignored errors increasing

[danimelocoton]

Hi guys

I have questioned by our NOC support about a possible issue in an GigabitEthernet interface of 7206VXR with NPE-G2.  I see input errors increasing at the same time as ignored errors. Deeping in the forum I find several discussions trying to explain this. Sorry if it's not the correct forum or it s a duplicate thread, I'm not able to find it. I hope you will help.

1) Microburst problem, Router cannot handle this traffic due to rx ring limits.

2) Possible cosmetic bug, CSCsf32623. (We dont use multicast and we have 15.2(4)S2 .

3) Possible hardware limitation (we are reaching 850Mbps in that interface, but input errors increase even in normal traffic hours  50-100Mbps)

During the peak hours, ratio between input errors ,and input packets is 1/3000.  

I dont see drops, and buffers misses you will see when you try show buffers command occured one week ago due to a big problem in the network. Same stats since then (we can't reload this router because affects more than 2000 clients=

The questions are

Is it a cosmetic bug? (We are not running multicast but version is 15.2(4)S2,

is it a hardware limitation problem? If yes...What hardware is recommended?

Can we try to tune something in config to fix this?. 

Is the ratio input Errors/packet errors affecting to clients traffic speed or traffic voice?. 

I try to send the result of commands.

Thanks.

show int gi0/1

Router#sho ver
Cisco IOS Software, 7200 Software (C7200P-ADVIPSERVICESK9-M), Version 15.2(4)S2, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 11-Dec-12 14:23 by prod_rel_team

ROM: System Bootstrap, Version 12.4(12.2r)T, RELEASE SOFTWARE (fc1)
BOOTLDR: Cisco IOS Software, 7200 Software (C7200P-BOOT-M), Version 15.2(4)S2, RELEASE SOFTWARE (fc1)

MURCIA uptime is 1 year, 8 weeks, 5 days, 17 hours, 20 minutes
System returned to ROM by reload at 09:33:10 METDST Fri Oct 30 2015
System restarted at 18:39:00 METDST Fri Oct 30 2015
System image file is "disk2:c7200p-advipservicesk9-mz.152-4.S2.bin"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 7206VXR (NPE-G2) processor (revision A) with 1966080K/65536K bytes of memory.
Processor board ID 36859447
MPC7448 CPU at 1666Mhz, Implementation 0, Rev 2.2
6 slot VXR midplane, Version 2.11

Last reset from power-on

PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points.
Current configuration on bus mb1 has a total of 0 bandwidth points.
This configuration is within the PCI bus capacity and is supported.

PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points.
Current configuration on bus mb2 has a total of 0 bandwidth points.
This configuration is within the PCI bus capacity and is supported.

Please refer to the following document "Cisco 7200 Series Port Adaptor
Hardware Configuration Guidelines" on Cisco.com <http://www.cisco.com>
for c7200 bandwidth points oversubscription and usage guidelines.

1 FastEthernet interface
3 Gigabit Ethernet interfaces
2045K bytes of NVRAM.

254464K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes).
65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

MURCIA#show int gi0/1
GigabitEthernet0/1 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is f4ac.c118.381b (bia f4ac.c118.381b)
MTU 1600 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 13/255, rxload 130/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is force-up, media type is ZX
output flow-control is XON, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 01:07:03
Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/1000/0 (size/max total/drops)
30 second input rate 511578000 bits/sec, 51088 packets/sec
30 second output rate 52534000 bits/sec, 30464 packets/sec
186939807 packets input, 735017821 bytes, 0 no buffer
Received 17933 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
32596 input errors, 0 CRC, 0 frame, 0 overrun, 32596 ignored
0 watchdog, 137121 multicast, 0 pause input
110015492 packets output, 2501314931 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
MURCIA#show buff
Buffer elements:
1108 in free list
870396829 hits, 0 misses, 617 created

Public buffer pools:
Small buffers, 104 bytes (total 101, permanent 50, peak 173 @ 7w0d):
89 in free list (20 min, 150 max allowed)
3273327119 hits, 56911 misses, 137802 trims, 137853 created
1543 failures (0 no memory)
Middle buffers, 600 bytes (total 59, permanent 25, peak 133 @ 7w0d):
57 in free list (10 min, 150 max allowed)
183032912 hits, 25901 misses, 63003 trims, 63037 created
1107 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50, peak 525 @ 7w0d):
49 in free list (5 min, 150 max allowed)
717169470 hits, 740508 misses, 10261 trims, 10261 created
726328 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 11, permanent 10, peak 116 @ 7w0d):
10 in free list (0 min, 100 max allowed)
21687 hits, 717412 misses, 679 trims, 680 created
717412 failures (0 no memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 52 @ 6w6d):
1 in free list (0 min, 10 max allowed)
2233 hits, 715179 misses, 4357 trims, 4358 created
715179 failures (0 no memory)
Huge buffers, 18024 bytes (total 1, permanent 0, peak 47 @ 7w0d):
1 in free list (0 min, 4 max allowed)
9374 hits, 714307 misses, 4297 trims, 4298 created
714302 failures (0 no memory)

Interface buffer pools:
Generic ED Pool buffers, 512 bytes (total 101, permanent 100, peak 101 @ 7w0d):
101 in free list (100 min, 100 max allowed)
32 hits, 0 misses
Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ 7w0d):
101 in free list (132 min, 132 max allowed)
20335080 hits, 0 misses
IPC buffers, 4096 bytes (total 2, permanent 2):
1 in free list (1 min, 8 max allowed)
1 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
IPC Medium buffers, 16384 bytes (total 2, permanent 2):
2 in free list (1 min, 8 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 7w0d):
17 in free list (16 min, 16 max allowed)
0 hits, 0 misses, 614480 trims, 614481 created
0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 512, permanent 256, peak 512 @ 7w0d):
256 in free list (256 min, 1024 max allowed)
171 hits, 85 misses, 0 trims, 256 created
0 failures (0 no memory)
256 max cache size, 256 in cache
156471514 hits in cache, 0 misses in cache

Particle Clones:
1024 clones, 19948762 hits, 0 misses

Public particle pools:
F/S buffers, 128 bytes (total 512, permanent 512):
0 in free list (0 min, 512 max allowed)
512 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
512 max cache size, 512 in cache
19948762 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 2048, permanent 2048):
2048 in free list (1024 min, 4096 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)

Private particle pools:
HQF Particle buffers, 0 bytes (total 2000, permanent 2000):
2000 in free list (500 min, 2000 max allowed)
764558687 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
GigabitEthernet0/1 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
1901880380 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
GigabitEthernet0/2 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
23171 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
GigabitEthernet0/3 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
3402839405 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
FastEthernet0/2 buffers, 768 bytes (total 529, permanent 400):
129 in free list (128 min, 1200 max allowed)
357 hits, 43 fallbacks, 7594 trims, 7723 created
0 failures (0 no memory)
400 max cache size, 272 in cache
128 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions

Router# show controll
Router# show controllers gi0/1
Interface GigabitEthernet0/1 (idb 0x693907C)
Hardware is MV64460 Internal MAC (Revision MV64460-Ethernet)
Network connection mode is FORCE
network link is up
Config is 1000Mbps, Full Duplex
Selected media-type is SFP
GBIC type is SFP
Ethernet Unit Global Registers:
PHY Address = 0x00000820 SMI (PHY Control) = 0x0F60BC03
Default Address (Err) = 0xFE200000 Default ID (Err) = 0x000001D1
Interrupt Cause = 0x00000A10 Interrupt Mask = 0x0000108E
Error Address = 0xD7FC6000 Internal Addr Error = 0x00000000
Port Pad Calibration = 0x0013000B Base Address 0 = 0xD7FC0002
Size (BA0) = 0x00030000 Base Address 1 = 0x78002D00
Size (BA1) = 0x03FF0000 Base Address 2 = 0x00000000
Size (BA2) = 0x00000000 Base Address 3 = 0x00000000
Size (BA3) = 0x00000000 Base Address 4 = 0x00000000
Size (BA4) = 0x00000000 Base Address 5 = 0x00000000
Size (BA5) = 0x00000000 Header Retarget Base = 0x00000000
Header Retarget Ctrl = 0x00000000 High Address Remap 0 = 0x00000000
High Address Remap 1 = 0x00000000 Base Address Enable = 0x0000003C
Port Access Protect 0 = 0x0000000F Port Access Protect 1 = 0x0000000F
Port Access Protect 2 = 0x0000000F

MAC Specific Registers:
Port Cfg PxC = 0x00000000 Port Cfg Extd PxCX = 0x00000000
MII Serial Params = 0x00218823 GMII Serial Params = 0x00000006
VLAN EtherType EVLANE = 0x00000000 MAC Addr Low MACAL = 0x0000381B
MAC Addr High MACAH = 0xF4ACC118 SDMA Config SDC = 0x01002005
PORT SERIAL CTRL PSC = 0x00EAA60D PORT STATUS PS = 0x0000041E
TX Queue Cmd TQC = 0x00000000 TxQ Fixed Prio TQFPC = 0x00000001
MTU (Token-Bucket) MTU = 0x00000000 INTR CAUSE IC = 0x80000005
INTR CAUSE EXT ICE = 0x00000000 Intr Mask PIM = 0x00080804
Extend Intr Mask PEIM = 0x00000101 RXQ Desc Ptr 0 CRDP0 = 0xD7FC06A0
RX Queue Cmd RQC = 0x0001FE01 TXQ Desc Ptr 0 TCQDP0 = 0xD7FC1060
TX Curr Desc TCSDP = 0xD7FC1060

MAC Serial Port is ENABLED
MAC Autoneg Capability: [Speed] [Duplex] [Flow Control]
MAC Status: Link is UP, Speed is 1000Mbps, Duplex is Full, Interface: GMII

RX Queue [0] is ENABLED

PHY Registers:
PHY is Marvell 88E1146C (Rev D0), address 0x0
Control = 0x1140 Status = 0x014D
PHY ID 1 = 0x0141 PHY ID 2 = 0x0CD4
Auto Neg Advertisement = 0x00A0 Link Partner Ability = 0x0000
Auto Neg Expansion = 0x0004 Next Page Tx = 0x2001
Link Partner Next Page = 0x0000 1000BaseT Control = 0x0000
1000BaseT Status = 0x0000 Extended Status = 0xC000
PHY Specific Control = 0x0008 PHY Specific Status = 0xA410
Interrupt Enable = 0x6800 Interrupt Status = 0x0000
Ext PHY Spec Control = 0x0CE2 Receive Error Counter = 0x0000
LED Control = 0x4101
Ext PHY Spec Control 2 = 0x000A Ext PHY Spec Status = 0xBC03
PHY says Link is UP, Speed 1000Mbps, Full-Duplex
Physical Interface - GBIC
AUTONEG - Our ability is 1000M/FD Pause Capable
AUTONEG - Partner ability is
GBIC registers:
Register 0x00: 03 04 07 00 00 00 42 80
Register 0x08: 00 01 01 01 0D 00 50 FF
Register 0x10: 00 00 00 00 43 49 53 43
Register 0x18: 4F 2D 46 49 4E 49 53 41
Register 0x20: 52 20 20 20 00 00 90 65
Register 0x28: 47 4C 43 2D 5A 58 2D 53
Register 0x30: 4D 20 20 20 20 20 20 20
Register 0x38: 41 30 20 20 06 0E 00 55
Register 0x40: 00 12 00 00 46 4E 53 31
Register 0x48: 37 30 37 31 48 36 55 20
Register 0x50: 20 20 20 20 00 00 00 00
Register 0x58: 00 00 20 20 00 00 00 AC
Register 0x60: 00 00 11 82 20 15 F8 DC
Register 0x68: 2C 6E 57 20 00 E8 A5 D0
Register 0x70: 86 14 79 00 00 00 00 00
Register 0x78: 00 00 00 00 39 49 B8 65
PartNumber: GLC-ZX-SM
PartRev: L
SerialNo: FNS17071H6U
Options: 0
Length(9um/50um/62.5um): 25500/000/000
Date Code:
Gigabit Ethernet Codes: 42

IDB Information:
lc_ip_turbo_fs = 0x3276FF4, ip_routecache = 0x11 (dfs = 0/mdfs = 0)
rx cache size = 1000, rx cache end = 868
max_mtu = 1628
Software MAC address filter(hash:length/addr/mask/hits):
need_af_check = 0
0x00: 0 ffff.ffff.ffff 0000.0000.0000 0
0xDA: 0 f4ac.c118.381b 0000.0000.0000 0

Internal Driver Information:
RX Ring base: 0xD7FC0000
TX Ring base: 0xD7FC1000
Software RX Head: 0xD7FC05E0
Hardware RX Head: 0xD7FC05A0
Software TX Head: 0xD7FC1060
Hardware TX Head: 0xD7FC26A0
ring sizes: RX = 128, TX = 256
rx_particle_size: 512
rx_pak = 0x068097D4
rx_head = 47
rx_discard = FALSE
tx_head = 3, tx_count = 0
chip_state = 2, ds->tx_limited = 1
throttled = 0, enabled = 0, disabled = 17
reset=7(init=1, restart=6), auto_restart=7
tx_underflow = 0, tx_overflow = 0, tx_end_count = 40335889
rx_nobuffer = 0, rx_overrun = 0
rx_no_descriptors = 18619, rx_interrupt_count = 135204678
rx_crc_error = 0, rx_too_big = 0, rx_resource_error = 32739
rx_sop_eop_error = 0
tqc = 0xF1002448, cause = 0xF1002460, cause_ext = 0xF1002464
Address Filter:
Promiscuous mode OFF
(All other entries are empty)

Statistics:

Receive and Transmit Statistics:
RX Good Octets 234951114490 TX Good Octets 24501693118
RX Good Packets 188311158 TX Good Packets 110376200
RX Bad Octets 0
RX Bad Packets 0
RX Broadcast Packets 18045 TX Broadcast Packets 0
RX Multicast Packets 137603 TX Multicast Packets 7

Error Statistics:
TX MAC Error 0
TX Excess Collisions 0
RX Bad MAC Ctrl Frame 0
RX Good Flow Control 0
RX Bad Flow Control 0
RX Undersize Paks 0
RX Oversize Paks 0
RX Fragments 0
RX Jabber Paks 0
RX MAC Error Events 0
RX Bad CRC Events 0
RX Collisions 0
RX Late Collisions 0
RX MAC Discard Frame 749363
RX MAC Overrun Frame 0

Total Packets (RX and TX), including errors:
0 -> 64 5191845
65 -> 127 101382963
128 -> 255 22422211
256 -> 511 4117410
512 -> 1023 4155543
1024 -> max 161417387
Router#

show processes cpu sorted
CPU utilization for five seconds: 35%/34%; one minute: 38%; five minutes: 38%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
89 2266224 1147537446 1 0.31% 0.34% 0.33% 0 IP ARP Retry Age
90 68565884 573767664 119 0.15% 0.13% 0.13% 0 IP Input
172 49774716 197175590 252 0.15% 0.12% 0.10% 0 ADJ resolve proc
155 6220756 54051095 115 0.15% 0.16% 0.15% 0 CEF: IPv4 proces
153 75528 580755459 0 0.15% 0.15% 0.15% 0 HQF Output Shape
86 29092 1147504700 0 0.07% 0.03% 0.02% 0 IPAM Manager
168 10432 3687044 2 0.07% 0.00% 0.00% 0 QoS stats proces
268 315396 36877953 8 0.07% 0.08% 0.07% 0 Per-Second Jobs
143 10700 6117 1749 0.07% 0.03% 0.01% 2 SSH Process
10 20439724 146930680 139 0.07% 0.02% 0.00% 0 ARP Input
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
11 1226884 38443155 31 0.07% 0.05% 0.07% 0 ARP Background
275 9586108 35323762 271 0.07% 0.02% 0.00% 0 SNMP ENGINE
273 9122560 70896527 128 0.07% 0.02% 0.00% 0 IP SNMP

[Leo Laohoo]

An input error value of "32596" vs a TOTAL packet input value of "186939807" is insignificant.  

[danimelocoton]

Thanks  Leo, 

You see insignificant cause you're viewing totals. When you monitor this in real time, ratio is 1 input error every 3000 input packets. The question is  which is the cause of this errors and how to solve it and if it's not a "real" error how to workaround this.

[Leo Laohoo]

You see insignificant cause you're viewing totals.

Valid point, but I disagree. 

1.  What is the uptime of the appliance?  "MURCIA uptime is 1 year, 8 weeks, 5 days, 17 hours, 20 minutes". 

2.  Next, how long has the counters for the interface been cleared?  Never.  

So how can one determine WHEN the line errors have been incrementing and how long or when has the line errors STOPPED incrementing?

[danimelocoton]

Thanks Leo for you reply.  As you can see router is uptime more than a year, true, but interface counters have been cleared yesterday and again this morning. You can see the output in my first post, but I issue the command again now.

I've been monitoring this since yesterday and I determine input errors continue incrementing. As long as load increases the number of input/ignored errors increment too. 

show int gi0/1
GigabitEthernet0/1 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is f4ac.c118.381b (bia f4ac.c118.381b)
MTU 1600 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 16/255, rxload 138/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is force-up, media type is ZX
output flow-control is XON, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 03:01:22
Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/1000/0 (size/max total/drops)
30 second input rate 543525000 bits/sec, 55930 packets/sec
30 second output rate 63329000 bits/sec, 34843 packets/sec
536850899 packets input, 3786575477 bytes, 0 no buffer
Received 49096 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
106524 input errors, 0 CRC, 0 frame, 0 overrun, 106524 ignored
0 watchdog, 371063 multicast, 0 pause input
325982440 packets output, 2914552272 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Leo Laohoo]

Last clearing of "show interface" counters 03:01:22

I completely missed this with your initial output.  My apologies. 

Total output drops: 0

106524 input errors, 0 CRC, 0 frame, 0 overrun, 106524 ignored

CRC and Overrun are 0.  Input errors are actually "Ignored".   Total output drops is also 0. 

No total output drops rules out link saturation.  

Ignored errors, as per Cisco documentation means:  

Description: Cisco IOS sh interfaces counter. The number of received packets ignored by the interface because the interface hardware ran low on internal buffers. Common Causes: Broadcast storms and bursts of noise can cause the ignored count to be increased.

[danimelocoton]

Thanks Leo for your reply. We`ll contact to our transport network provider to be sure is not related with noise. But I see this ignored errors, input errors increase at the same time as RX MAC Discard Frame counter

What about RX MAC Discard Frame  counter?. I saw this is related with the ignored errors too and in some cases seems to be a cosmetical bug. Do you know something about this counter?