11-25-2018 02:44 PM - edited 03-05-2019 11:04 AM
Dear All,
Could anyone point me in the right direction to troubleshoot input drops on my ISR4331 Gi0 interface please? The drops are constantly incrementing and the interface stops responding to ping/ssh/snmp after around 1 or 2 hours. A shut/no shut makes it responsive for another while.
I have swapped the cable (it's a 2m cat6 to a nearby Nexus 2k FEX), tried auto neg, removed any access-policies on the line vty. The code version is 16.03.07, but I can't find any bugs. Packet capture is also not supported on the management interface.
4431#sho run inter gi0 ! interface GigabitEthernet0 description OOB-FEX Eth109/1/6 VL806 vrf forwarding Mgmt-intf ip address 10.0.0.1 255.255.255.192 no ip redirects no ip unreachables speed 1000 no negotiation auto end
Here are the errors:
4431#show inter gi0
GigabitEthernet0 is up, line protocol is up
Hardware is RP management port, address is 4001.7a35.c85f (bia 4001.7a35.c85f)
Description: GESOOBML03 Eth109/1/6 VL806
Internet address is 10.238.203.52/26
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 254/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is force-up, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output 00:00:05, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/6156/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2000 bits/sec, 2 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
442568 packets input, 51132110 bytes, 0 no buffer
Received 1735 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
6156 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
19497 packets output, 4748654 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
This also doesn't shed much light:
chges-d-svuvu-01#sho inter gi0 switching
GigabitEthernet0 GESOOBML03 Eth109/1/6 VL806
Throttle count 0
Drops RP 6165 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 71 3282 8487 866569
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2416 144960 1392 83520
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol CDP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 3073 875805 6900 3026964
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 12356 4606910
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Any suggestions would be greatly appreciated.
James.
11-27-2018 05:45 AM
Hi James.
Did you try to set negotiation auto on both sides at the same time?
BR
Gaston
12-11-2018 03:25 AM
Yes, we tried disabling auto neg at both sides to no avail.
In fact, we recently resolved this issue. I should mention that the upstream device is a Nexus 2k FEX and the HSRP gateway is running on a VPC pair. We noticed this within a debug interface gi0 on the ISR4K:
316292: Dec 3 18:19:31.673 GMT: IOSXE-INJECT: set pak datagramstart(from 0x7sho F77CBFB6B1A to 0x7F77CBFB6B28), datagramsize(from 98 to 84); 316293: Dec 3 18:19:31.674 GMT: IOSXE-INJECT: add L3 inject hdr 316300: Dec 3 18:19:31.678 GMT: IOSXE-INJECT: ARP/ND trigger failed, id 0x1, linktype ip, type incomplete, nexthop defined, reason adj not found 316301: Dec 3 18:19:31.678 GMT: IOSXE-INJECT: simulate cpp forward drop failed packet,: triggering arp/nd, v4 10.238.203.62, adj-id 0x0, intf Gi0 Output intf Gi0 matches debug filer
It looks like the ISR was failing to make an ARP request for the default gateway.
So I added a static entry corresponding to the virtual mac on the HSRP GW:
arp vrf Mgmt-intf 10.0.0.1 0000.0c9f.f001 ARPA
This worked around the issue of the interface going unresponsive every few hours.
We then upgraded from Denali to Everest (without upgrading ROMMON or the CPLD) and removed the static arp entry. The issue has gone away, including all the input errors.
My conclusion from all this is that there's an undocumented bug in Denali which prevents the Gi0 interface from working with Nexus HSRP.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide