INPUT errors on C8500L-8S4X interface connected to C4500-x switch

[Dmitriy Ivanov]

I have router, connected to the switch - x10G interfaces in Ethernet channel. Switch side interfaces is clean, router side interface constantly shows INPUT ERRORS incrementing.

Contacted Cisco TAC- they suspected that I hit the bug:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwe09298

I filtered BPDUs on the switch side, but counters are still increasing

!!!!!!!!!!!!Router side

C8k5-inet#sh clock

20:03:21.689 EDT Tue Jun 13 2023

c8k5-inet#sh int po19 | i input errors

     602197 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored

c8k5-inet#sh clock

20:03:32.063 EDT Tue Jun 13 2023

c8k5-inet#sh int po19 | i input errors

     602208 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored

c8k5-inet#sh clock

20:03:45.885 EDT Tue Jun 13 2023

c8k5-inet#sh int po19 | i input errors

     602235 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored

!!!!!!!! router side int config

interface Port-channel19
description link to c4k5-core-dc1
no ip address
!
interface Port-channel19.530
description UNDP-HQ-BB-EXT-CAMPUS
encapsulation dot1Q 530
ip address x.x.x.2 255.255.255.128
no ip redirects
standby 1 ip x.x.x..1
standby 1 timers 5 15
standby 1 priority 50
standby 1 preempt
standby 1 authentication HSRP-42


!!!!!!! Switch side!!!!!!!!!!!!!!!!!


interface Port-channel19

description c8k5-inet

switchport

switchport mode trunk

spanning-tree bpdufilter enable

end

We are clearly see that input errors are not BPDU related........

I also verified- the switch does not send any BPDUs into the interface

command:

c4k5-core-dc1.hq(config-if)#do sh spanning-tree int po19 detail | i BPDU

does not show BPDU counters increment- means that BPDU filter works as expected.



Router interfaces and "sh tec" provided to Cisco- I have ticket open with TAC- so far no solution.

sh int on the router side:

c8k5-inet#sh int te0/1/0
TenGigabitEthernet0/1/0 is up, line protocol is up
Hardware is 4xSFP+, address is 6c29.d25c.cbd2 (bia 6c29.d25c.cb88)
Description: c4k5-core Te1/1/19
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 253/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 10000Mbps, link type is auto, media type is 10GBase-SR
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:26, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:32:17
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 84000 bits/sec, 111 packets/sec
5 minute output rate 258000 bits/sec, 150 packets/sec
214549 packets input, 20882442 bytes, 0 no buffer
Received 50418 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
4922 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 152941 multicast, 0 pause input
304295 packets output, 62546336 bytes, 0 underruns
Output 22179 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 0 interface resets
102 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out


c8k5-inet#sh int te0/1/2
TenGigabitEthernet0/1/2 is up, line protocol is up
Hardware is 4xSFP+, address is 6c29.d25c.cbd2 (bia 6c29.d25c.cb8a)
Description: c4k5-core Te2/1/19
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 10000Mbps, link type is auto, media type is 10GBase-SR
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:32:35
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1545000 bits/sec, 798 packets/sec
5 minute output rate 1563000 bits/sec, 655 packets/sec
1859924 packets input, 571069825 bytes, 0 no buffer
Received 45410 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
588 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 85582 multicast, 0 pause input
1763840 packets output, 955899454 bytes, 0 underruns
Output 20910 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 0 interface resets
102 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out


Yes I tried to replace fibers, replace modules on both ends, tried another modules etc- same thing.

Any ideas? Thank you!

[Leo Laohoo]

---

@Dmitriy Ivanov wrote:

reliability 253/255​

---

Replace the optic and patch cable at the router side.

[Dmitriy Ivanov]

Done that twice. 100% sure optics and fibers are fine.

[Dmitriy Ivanov]

I tried to use these SFPs with fibers on another 1001-X router- no errors on 1001-X, but 8500 gives below. I also switched SFPs to slots 0/1/2 and 0/1/3 - nothing changes. 


sh int te0/1/2
TenGigabitEthernet0/1/2 is up, line protocol is up
Hardware is 4xSFP+, address is 6c29.d25c.cbd2 (bia 6c29.d25c.cb8a)
Description: c4k5-core Te2/1/19
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 10000Mbps, link type is auto, media type is 10GBase-SR
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:17, output 00:00:00, output hang never
Last clearing of "show interface" counters 3d21h
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4223000 bits/sec, 1577 packets/sec
5 minute output rate 16464000 bits/sec, 1951 packets/sec
1926561998 packets input, 1124541335646 bytes, 0 no buffer
Received 11033285 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
105582 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 20484758 multicast, 0 pause input
1511035897 packets output, 1265509982453 bytes, 0 underruns
Output 2559599 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 2 interface resets
17489 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
1 lost carrier, 0 no carrier, 12 pause output
0 output buffer failures, 0 output buffers swapped out


sh int te0/1/3
TenGigabitEthernet0/1/3 is up, line protocol is up
Hardware is 4xSFP+, address is 6c29.d25c.cbd2 (bia 6c29.d25c.cb8b)
Description: c4k5-core Te1/1/19
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 253/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 10000Mbps, link type is auto, media type is 10GBase-SR
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:00, output hang never
Last clearing of "show interface" counters 2d07h
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 137000 bits/sec, 174 packets/sec
5 minute output rate 7461000 bits/sec, 1355 packets/sec
68233647 packets input, 7853747137 bytes, 0 no buffer
Received 8238293 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
867170 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 19877693 multicast, 0 pause input
1627946935 packets output, 1473884439235 bytes, 0 underruns
Output 1073898 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 3 interface resets
10490 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
3 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Dmitriy Ivanov]

you also see that on 0/1/2 

reliability 255/255, txload 1/255, rxload 1/255

BUT

105582 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

[Leo Laohoo]

Ok, let's do something really wonky. 

Loop Te0/1/2 and Te0/1/3 together.  Let's see if the the reliability falls.  

[Dmitriy Ivanov]

It is production, and I manage it remotely. Also the service interruption is not an option at the moment. May be sometime on weekends. 

Interestingly, I have 10G ISP interface on the same router and no errors reported

TenGigabitEthernet0/1/1 is up, line protocol is up
Hardware is 4xSFP+, address is 6c29.d25c.cb89 (bia 6c29.d25c.cb89)
Description: Circ ID 4432
Internet address is x.x.x.x/30
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 10000Mbps, link type is auto, media type is SFP-LRM
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 94178000 bits/sec, 14939 packets/sec
5 minute output rate 44418000 bits/sec, 8842 packets/sec
3295302361 packets input, 2906114470917 bytes, 0 no buffer
Received 4 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
1888982673 packets output, 1128031428848 bytes, 0 underruns
Output 1 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
1 lost carrier, 0 no carrier, 12 pause output
0 output buffer failures, 0 output buffers swapped out

I also run MTR from "inside" - no packet loss reported and the customer also does not report any connectivity problems. Looks like it is "pure cosmetic" bug but I'm really puzzled that Etherchannel is not something "rare/new" I'm doing that site CatOs (20+ years) - and never had any problems.... if it is IOS bug- they supposed to do some "basic" tests before release.... And also I do not see any "internet noise" related to this issue- looks like I'm the only one "lucky" to hit this bug? I'm really puzzled. Is Etherchannel "obsolete" ? I need etherchannel for redundancy- I have pair c4500-X in VSS mode on another end of the cable... 

[Dmitriy Ivanov]

"I'm doing that site CatOs (20+ years) - and never had any problem"

I meant I used Etherchannel on Catos and later on IOS - for 20+ years it was the "standard" for redundancy. 

I'm thinking may be configure switch side PO as "switch port access plan XXXX" instead of trunk and remove sub-interface on the router side but have L3 PO instead... Did not try yet. For some reason I think that traffic via "trunk" from the switch affects counters on the router side due to some bug in IOS on the router side..... Just pure speculation- no any proof. 

[Dmitriy Ivanov]

Now I see these:

#sh int te0/1/2 | in reliability | error
reliability 255/255, txload 2/255, rxload 2/255
125999 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 output errors, 0 collisions, 2 interface resets


#sh int te0/1/3 | in reliability | error
reliability 255/255, txload 7/255, rxload 1/255
1122150 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 output errors, 0 collisions, 3 interface resets


#sh int po19 | in reliability | error
reliability 255/255, txload 4/255, rxload 1/255
1252830 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored
0 output errors, 0 collisions, 0 interface resets

[Leo Laohoo]

What firmware is the switch on?

[Dmitriy Ivanov]

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.06.07.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Wed 12-Jul-17 13:41 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: 15.0(1r)SG12
c4k5-core-dc1.hq uptime is 25 weeks, 6 days, 1 hour, 8 minutes
Uptime for this control processor is 25 weeks, 6 days, 1 hour, 10 minutes
System returned to ROM by power-on
System image file is "bootflash:cat4500e-universalk9.SPA.03.06.07.E.152-2.E7.bin"
Jawa Revision 3, Winter Revision 0x0.0x41

Last reload reason: power-on

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Information for 'WS-C4500X-32'
License Level: entservices Type: Permanent
Next reboot license Level: entservices

cisco WS-C4500X-32 (MPC8572) processor (revision 9) with 4194304K bytes of physical memory.
Processor board ID xxx
MPC8572 CPU at 1.5GHz, Cisco Catalyst 4500X
Last reset from PowerUp
20 Virtual Ethernet interfaces
64 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2102

#sh redundancy
Redundant System Information :

------------------------------
Available system uptime = 25 weeks, 6 days, 1 hour, 12 minutes
Switchovers system experienced = 0
Standby failures = 1
Last switchover reason = none

Hardware Mode = Duplex
Configured Redundancy Mode = Stateful Switchover
Operating Redundancy Mode = Stateful Switchover
Maintenance Mode = Disabled
Communications = Up

Current Processor Information :
------------------------------
Active Location = slot 1/1
Current Software state = ACTIVE
Uptime in current state = 25 weeks, 6 days, 1 hour, 9 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.06.07.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Wed 12-Jul-17 13:41 by prod
BOOT = bootflash:cat4500e-universalk9.SPA.03.06.07.E.152-2.E7.bin,1;
Configuration register = 0x2102

Peer Processor Information :
------------------------------
Standby Location = slot 2/1
Current Software state = STANDBY HOT
Uptime in current state = 25 weeks, 10 hours, 31 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.06.07.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Wed 12-Jul-17 13:41 by pr
BOOT = bootflash:cat4500e-universalk9.SPA.03.06.07.E.152-2.E7.bin,1;
Configuration register = 0x2102

[Leo Laohoo]

Right.  I just saw the Bug ID.  

So TAC is saying the BPDU traffic was incorrectly added into the "Input Error" counters?  But it does not answer the question as to why the "reliability" numerator is <255.

[Dmitriy Ivanov]

Well you see that bug (CSCwe09298) does not answer anything in my case. it says that SpanTree BPDU TCN from the switch increases error counter on the router. First, number of errors does not correlate with the number of TCNs, second, applying "spantree bpdufilter" does not stop error counters. While BUG says "no workaround" - IMHO why do not stop spantree totally on the router connected interface with "bpdufilter"? It is workaround and it should stop error counter. I do not believe Cisco "forgot" about bpdufilter.  Looks like there is some "inconsistency" in that bug description- possible it is more than BPDUs, and it keeps going from one IOS version to another and affects this specific model of the router only.

Cisco TAC also says that the scope of the bug is "not only" BPDUs. I do not know if it is his personal opinion or it is in their internal DB. He also says that "capture" on the router does not catch these errors. He tried to replicate my scenario on generic switch/router- with no success. I have ASR1001-X in exactly the same config, connected to the same switch- no errors. Of course I tried to swap cables/SFPs between these 2- as I said no errors on one router and INPUT ERRORS on another. I also tried to use UCS 10G cables (black ones with built-in SFPs)- same think, link UP, Input errors are counting with the rate about 2.25 error per second with traffic rate : 5 minute input rate 64843000 bits/sec, 12264 packets/sec, So we are talking about 1 error per ~2000 packets in pick of user's activity.

I do not see reliability issue anymore, it is 255/255 on both physical interfaces:

reliability 255/255, txload 5/255, rxload 1/255

I observe such behavior on only  this particular model of the router C8500L-8S4X. TAC requested their mothership to deliver such model in their lab so they can fiddle around.

Meanwhile what we agreed to do- these weekends I will de-configure PO, shut down 1 link and configure another link as "switch port access vlan XXXX" and another end as L3. It will take out of the picture "Trunk/Etherchannel". They also asked if I can convert it to L3-L3 p2p link , unfortunately I can't. 4500-X can not do that. Still missing BigCats(6500)- they were capable to do that.

We both TAC and me agree that it looks like "L2" packets which are absolutely legitimate from the Switch's point of view trigger error counter on the router side and only this particular model of the router affected. No other side-effects. It is purely "cosmetic" issue. Router does NOT give any details on errors it counts. TAC confirmed that.



Regarding Reliability- well I see it is 255/255 both "Te". interfaces and it stays that way already couple of days. Who knows- may be another bug.

Thanks for reading and replying!

[Dmitriy Ivanov]

Sorry miscalculated - 1 error per ~6000 packets in pick of user's activity.

[Dmitriy Ivanov]

All right- it is fixed.

Port-channel19 is up, line protocol is up
Hardware is 10GEChannel, address is 6c29.d25c.cbd2 (bia 6c29.d25c.cbd2)
Description: link to c4k5-core-dc1
Internet address is x.x.x.x/25
MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 04:00:00
No. of active members in this channel: 2
Member 0 : TenGigabitEthernet0/1/2 , Full-duplex, 10000Mb/s
Member 1 : TenGigabitEthernet0/1/3 , Full-duplex, 10000Mb/s
No. of PF_JUMBO supported members in this channel : 2
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:16:54
Input queue: 0/750/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/80 (size/max)
5 minute input rate 1545000 bits/sec, 876 packets/sec
5 minute output rate 1663000 bits/sec, 946 packets/sec
904406 packets input, 206161988 bytes, 0 no buffer
Received 524 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3298 multicast, 0 pause input
992885 packets output, 231477825 bytes, 0 underruns
Output 12125 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out


Fix was to configure "switch port mode access vlan xxx" on the switch side, remove sub-interface on the router side and configure PO as L3 interface:

interface Port-channel19
description link to switch
ip address x.x.x.x.2 255.255.255.128
no ip redirects
standby 1 ip x.x.x.x.1
standby 1 timers 5 15
standby 1 priority 50
standby 1 preempt
standby 1 authentication HSRP-42
end


Conclusion: as I said in my previous post, when the switch port configured as "trunk", it sends some packets to the router which counted as "INPUT ERRORS" on the router side. Looks like I discovered new Cisco bug.

Leo: Thanks a lot for reading and being with me! We need to get cup of beer together some day.