Re: Nexus continously sends ICMP Unreachable without any apparent reasons

lpuri · ‎05-20-2021

Hi Everyone,

Does anyone have any idea why Nexus sends ICMP unreachable packets in every seconds? According to debug there is no other ICMP traffic before or after.

Debug shows this:

2021 May 20 12:58:16.340743 netstack: [27854] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_UNREACH, tos/dscp=0xc0/0x30, ip_len=56, id=7a2e, ttl=255
2021 May 20 12:58:17.257769 netstack: [27854] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_UNREACH, tos/dscp=0xc0/0x30, ip_len=56, id=3c71, ttl=255
2021 May 20 12:58:18.216792 netstack: [27854] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_UNREACH, tos/dscp=0xc0/0x30, ip_len=56, id=de5e, ttl=255
2021 May 20 12:58:19.071847 netstack: [27854] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_UNREACH, tos/dscp=0xc0/0x30, ip_len=56, id=f85a, ttl=255
2021 May 20 12:58:20.820830 netstack: [27854] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_UNREACH, tos/dscp=0xc0/0x30, ip_len=56, id=f42a, ttl=255
2021 May 20 12:58:21.606792 netstack: [27854] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_UNREACH, tos/dscp=0xc0/0x30, ip_len=56, id=39bd, ttl=255
2021 May 20 12:58:22.374802 netstack: [27854] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_UNREACH, tos/dscp=0xc0/0x30, ip_len=56, id=bdb8, ttl=255
...

When pinging from/to the device the debug shows correctly the echo/echo-reply packets, so the debug should be correct.

Nexus is connected to a firewall which shows this traffic as coming from source port 3, dest port 3, protocol 1.

Topology and screen shots from firewall attached.

The relevant part of Nexus config is here:

vpc domain 1
peer-switch
role priority 1
peer-keepalive destination 10.255.0.14 source 10.255.0.13
peer-gateway
layer3 peer-router

interface Vlan1
description Interface to ISP-1
no shutdown
ip access-group blacklist in
no ip redirects
ip address 195.111.103.115/29
no ipv6 redirects

interface Vlan2298
description -> FW-1
no shutdown
no ip redirects
ip address 192.168.250.1/24
no ipv6 redirects

interface port-channel1
description Interface to ISP-1
switchport
spanning-tree port type edge
vpc 1

interface port-channel11
description -> fw1
switchport
switchport access vlan 2298
vpc 11

interface port-channel100
description vPC peer link
switchport
switchport mode trunk
spanning-tree port type network
spanning-tree guard loop
vpc peer-link

interface Ethernet1/1
description -> fw1
switchport
switchport access vlan 2298
channel-group 11 mode active
no shutdown

interface Ethernet1/17
description Interface to ISP-1
switchport
spanning-tree port type edge
channel-group 1 mode active
no shutdown

interface Ethernet1/47
description vPC peer link
switchport
switchport mode trunk
channel-group 100 mode active
no shutdown

interface Ethernet1/48
description vPC peer link
switchport
switchport mode trunk
channel-group 100 mode active
no shutdown

Thanks for any help.

Laszlo

f00z · ‎05-22-2021

I would use ethanalyzer on the inband interface to see what mac addrs it is going to; or bash shell and tcpdump; or span the port/mirror.

Most likely another device is sending packets to the nexus (maybe to a udp port) or it has no route to some destination, which can be seen by the ethanalyzer/tcpdump.

lpuri · ‎05-25-2021

Hi,
the same as I also wanted to do, but the geographical distance and the pandemic situation prevented me to access the device onsite.

I suspected that it may related to bfd. Enabling the 'feature 'bfd' changed immediately the ICMP debug screen like this:

ICMP packet debugging is on
wdc-extgw1a#
2021 May 25 10:50:58.679585 netstack: [27889] (default) Rcvd packet on Vlan2298 (mbuf_prty 0): s=192.168.250.254, d=192.168.250.1, proto=1 (icmp), ICMP_ECHO, tos/dscp=0x0/0x0, ip_len=84, id=7839, ttl=63
2021 May 25 10:50:58.679658 netstack: [27889] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_ECHOREPLY, tos/dscp=0x0/0x0, ip_len=84, id=7839, ttl=255
2021 May 25 10:51:01.680718 netstack: [27889] (default) Rcvd packet on Vlan2298 (mbuf_prty 0): s=192.168.250.254, d=192.168.250.1, proto=1 (icmp), ICMP_ECHO, tos/dscp=0x0/0x0, ip_len=84, id=efad, ttl=63
2021 May 25 10:51:01.680788 netstack: [27889] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_ECHOREPLY, tos/dscp=0x0/0x0, ip_len=84, id=efad, ttl=255
2021 May 25 10:51:04.679555 netstack: [27889] (default) Rcvd packet on Vlan2298 (mbuf_prty 0): s=192.168.250.254, d=192.168.250.1, proto=1 (icmp), ICMP_ECHO, tos/dscp=0x0/0x0, ip_len=84, id=0c7a, ttl=63
2021 May 25 10:51:04.679621 netstack: [27889] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_ECHOREPLY, tos/dscp=0x0/0x0, ip_len=84, id=0c7a, ttl=255
2021 May 25 10:51:07.679031 netstack: [27889] (default) Rcvd packet on Vlan2298 (mbuf_prty 0): s=192.168.250.254, d=192.168.250.1, proto=1 (icmp), ICMP_ECHO, tos/dscp=0x0/0x0, ip_len=84, id=f8a0, ttl=63
2021 May 25 10:51:07.679088 netstack: [27889] (default) Send packet (mbuf_prty 0): s=192.168.250.1, d=192.168.250.254, proto=1 (icmp), ICMP_ECHOREPLY, tos/dscp=0x0/0x0, ip_len=84, id=f8a0, ttl=255

At the same time the warning of suspicious traffic on the firewall disappeared.
It is still unclear why incoming ICMP echo requests do not show up in the debug until bfd is not enabled. While other ICMP echo requests such as pinging from the firewall do appear among the debug lines.
Magic....

f00z · ‎05-25-2021

BFD is usually hardware offloaded so it doesn't show up hitting the CPU, although punted exception packets will. (like if it's an actual BFD packet) but it is odd why normal ICMP requests show up differently depending on BFD configuration.

Maybe CoPP?

Why can't you use ethanalyzer?

lpuri · ‎05-27-2021

Hi f00z,

thanks for the help.

In the firewall BFD was configured by mistake. Odd enough but icmp debug shows only echo request from ping only until feature bgp is off. After enabled the sending of unreachables disappeared from debug.

Anyway, BFD was switched off as it was in system doc, and the problem disappeared.

Thanks

(Why not ethanalyzer? It is a production environment, every action needs a change ticket and on-site assistance. The current one did not permit it)