Re: mfib debug output on C9K

andy!doesnt!like!uucp · ‎12-23-2019

Hi Everyone

after migration to new core on C9500 series i've discovered some multicast blackholing between VRFs connected with GRE Tunnels on the same box. 'debug ip mfib vrf vrf-red 239.255.255.250' hints with bodled text

Dec 23 19:45:19: MFIBv4(0x5): Pkt (10.201.208.3,239.255.255.250) from Tunnel2 (PS) accepted for forwarding
Dec 23 19:45:19: MFIBv4(0x5): Pkt (10.201.208.3,239.255.255.250) from Tunnel2 (PS) MFIB_PAK_NO_FORWARD is set in subblock
Dec 23 19:45:19: MFIBv4(0x5): Pkt (10.200.0.3,239.255.255.250) from Tunnel1 (PS) accepted for forwarding
Dec 23 19:45:19: MFIBv4(0x5): Pkt (10.200.0.3,239.255.255.250) from Tunnel1 (PS) MFIB_PAK_NO_FORWARD is set in subblock

any clues on problem?

P.S.in BUG search tool there is something similar can be found:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvn11429/?rfs=iqvred

Giuseppe Larosa · ‎12-23-2019

Hello @andy!doesnt!like!uucp ,

I would say it is an error in TCAM low level programming

Dec 23 19:45:19: MFIBv4(0x5): Pkt (10.201.208.3,239.255.255.250) from Tunnel2 (PS) accepted for forwarding
Dec 23 19:45:19: MFIBv4(0x5): Pkt (10.201.208.3,239.255.255.250) from Tunnel2 (PS) MFIB_PAK_NO_FORWARD is set in subblock

But, before opening a case to Cisco TAC I would like to ask you more info:

how are the GRE tunnels configured ?

You need to read the release notes and configuration guide for your IOS XE version to understand if it is a supported configuration.

The bug you listed at first sight looks like a different problem.

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/16-11/release_notes/ol-16-11-9500.html

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/16-12/release_notes/ol-16-12-9500.html#reference_usb_ylx_tjb

see the following one solved in 16.12.2

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvo33983

Missing/incorrect FED entries for IGMP Snooping on Cat9300/Cat3850/Cat3650

CSCvn31653

Edit:

In IOS XE FED should be responsible of low level TCAM programming on mutilayer switches

I would consider an upgrade to a newer IOS XE however can you tell if the C9500 is performing IGMP /MLD Snooping on the GRE Tunnels

my guess is that your tunnels are used to join the two VRFs multicast topologies (one side in VRF A one side in VRF B) using a logical interface disabling IGMP snooping on these p2p tunnels can solve your issue.

Hope to help

Giuseppe

andy!doesnt!like!uucp · ‎12-23-2019

Hi Juseppe

thanks for feedback.

we use Tunnels between VRFs to overcome problem of multicasting through FW separating VRFs in Q.

tunnel in default VRF:

interface Tunnel101
ip address 192.168.160.226 255.255.255.252
no ip proxy-arp
ip pim bsr-border
ip pim sparse-mode
tunnel source Loopback3

tunnel destination 192.168.160.251

vrf-red (RP-bidir) side:

interface Tunnel1
ip vrf forwarding vrf_red
ip address 192.168.160.225 255.255.255.252
no ip proxy-arp
ip pim bsr-border
ip pim sparse-mode

!below ACL prevents MC from being relayed between tunnels (there is another one similar pair of tunnels for vrf-red & vrf_guest)

!only MC from vrf-red subnets are allowed in reverse direction
ip access-group acl-BLOCK-UPnP-RELAY out
tunnel source Loopback162
tunnel destination 10.255.255.160
tunnel vrf vrf_red

#sho access-lists acl-BLOCK-UPnP-RELAY
Extended IP access list acl-BLOCK-UPnP-RELAY
10 permit ip 192.168.161.0 0.0.0.255 host 239.255.255.250
20 permit ip 192.168.162.0 0.0.0.255 host 239.255.255.250
30 permit ip 192.168.163.0 0.0.0.255 host 239.255.255.250
40 permit ip 192.168.164.0 0.0.0.255 host 239.255.255.250
50 permit pim any any
60 permit ip host 192.168.160.255 any

we dont run MLD & igmp snooping is by default enabled only on vlan 1

#sho ip igmp snooping
Global IGMP Snooping configuration:
-------------------------------------------
IGMP snooping : Enabled
Global PIM Snooping : Disabled
IGMPv3 snooping : Enabled
Report suppression : Enabled
TCN solicit query : Disabled
TCN flood query count : 2
Robustness variable : 2
Last member query count : 2
Last member query interval : 1000

Vlan 1:
--------
IGMP snooping : Enabled
Pim Snooping : Disabled
IGMPv2 immediate leave : Disabled
Explicit host tracking : Enabled
Multicast router learning mode : pim-dvmrp
CGMP interoperability mode : IGMP_ONLY
Robustness variable : 2
Last member query count : 2
Last member query interval : 1000

andy!doesnt!like!uucp · ‎12-23-2019

it's funny, just found my replies to my post require approval to be shown :0D

andy!doesnt!like!uucp · ‎12-23-2019

to be short:

interface Tunnel101
ip address 192.168.160.226 255.255.255.252
no ip proxy-arp
ip pim bsr-border
ip pim sparse-mode
tunnel source Loopback3
tunnel destination 192.168.160.251
end

!Below is RP side

#sho run int tu1
Building configuration...

Current configuration : 288 bytes
!
interface Tunnel1
ip vrf forwarding vrf_red
ip address 192.168.160.225 255.255.255.252
no ip proxy-arp
ip pim bsr-border
ip pim sparse-mode
ip access-group acl-BLOCK-UPnP-RELAY out
tunnel source Loopback162
tunnel destination 10.255.255.160
tunnel vrf vrf_red
end

!this ACL prevents from unwanted MC between Tu1 & Tu2 (toward another one VRF)
#sho access-lists acl-BLOCK-UPnP-RELAY
Extended IP access list acl-BLOCK-UPnP-RELAY
10 permit ip 192.168.161.0 0.0.0.255 host 239.255.255.250
20 permit ip 192.168.162.0 0.0.0.255 host 239.255.255.250
30 permit ip 192.168.163.0 0.0.0.255 host 239.255.255.250
40 permit ip 192.168.164.0 0.0.0.255 host 239.255.255.250
50 permit pim any any
60 permit ip host 192.168.160.255 any

we dont run MLD & IGMP snooping is enabled by default only on VLAN 1 i guess:

#sho ip igmp vrf vrf_red sno
Global IGMP Snooping configuration:
-------------------------------------------
IGMP snooping : Enabled
Global PIM Snooping : Disabled
IGMPv3 snooping : Enabled
Report suppression : Enabled
TCN solicit query : Disabled
TCN flood query count : 2
Robustness variable : 2
Last member query count : 2
Last member query interval : 1000

Vlan 1:
--------
IGMP snooping : Enabled
Pim Snooping : Disabled
IGMPv2 immediate leave : Disabled
Explicit host tracking : Enabled
Multicast router learning mode : pim-dvmrp
CGMP interoperability mode : IGMP_ONLY
Robustness variable : 2
Last member query count : 2
Last member query interval : 1000

andy!doesnt!like!uucp · ‎12-23-2019

Ok. in the Restrictions for MC routing over GRE Tunnels its's said:

Bidirectional PIM is not supported. one i exactly have for this deployment...

andy!doesnt!like!uucp · ‎12-24-2019

again thanks to u Juseppe. rebuilding deployment w/o bidir helped to resolve the issue.

but now i have different problem: MC originated from the VLAN on the distribution N7K from within non-default VRF gets dropped on the core switch within this vrf with "Acceptance check failed". I've broken all tools to find out why it's like this but no luck so far.
Incoming intf is pim operational, OIF-list on the core is populated, rpf-check passes Ok for both source & RP & so on.

Dec 24 10:58:39: MFIBv4(0x4): Pkt (10.201.208.3,239.255.255.250) from FortyGigabitEthernet1/0/5.2294 (PS) Acceptance check failed - dropping
swt-de-c-dc67-01#
Dec 24 10:58:42: MFIBv4(0x4): Pkt (10.201.208.3,239.255.255.250) from FortyGigabitEthernet1/0/5.2294 (PS) Acceptance check failed - dropping
#

andy!doesnt!like!uucp · ‎12-24-2019

after double verification: it doesnt work now for both default & guest vrf:
debug from red-vrf:

Dec 24 14:41:45: MFIBv4(0x5): Pkt (10.1.8.95,239.255.255.250) from Tunnel7 (PS) accepted for forwarding
Dec 24 14:41:45: MFIBv4(0x5): Pkt (10.1.8.95,239.255.255.250) from Tunnel7 (PS) MFIB_PAK_NO_FORWARD is set in subblock

Dec 24 14:42:31: MFIBv4(0x5): Pkt (10.60.98.105,239.255.255.250) from Tunnel1 (PS) Queued signalling for routing protocol
Dec 24 14:42:31: MFIBv4(0x5): Pkt (10.60.98.105,239.255.255.250) from Tunnel1 (PS) Packet is marked as not to be forwarded by MFIB. Hence it will not be preserved - dropping

debug from default vrf:

Dec 24 14:44:18: MFIBv4(0x0): Pkt (10.200.0.2,239.255.255.250) from FortyGigabitEthernet1/0/29 (PS) Acceptance check failed - dropping

debug fro guest vrf (most extravagant):

ec 24 14:45:08: MFIBv4(0x4): Pkt (192.168.163.218,239.255.255.250) from Tunnel201 (PS) Queued signalling for routing protocol
Dec 24 14:45:08: MFIBv4(0x4): Pkt (192.168.163.215,239.255.255.250) from Tunnel201 (PS) Queued signalling for routing protocol
Dec 24 14:45:08: MFIBv4(0x4): Pkt (192.168.163.215,239.255.255.250) from Tunnel201 (PS) Queued signalling for routing protocol
Dec 24 14:45:08: MFIBv4(0x4): Pkt (10.201.208.3,239.255.255.250) from FortyGigabitEthernet1/0/5.2294 (PS) Acceptance check failed - dropping
Dec 24 14:45:08: MFIBv4(0x4): Pkt (192.168.161.31,239.255.255.250) from Tunnel201 (PS) Queued signalling for routing protocol

=> Case in Cisco TAC

andy!doesnt!like!uucp · ‎12-30-2019

one interesting observation: if u dont define local receiver for the group G on the C9500 16.12.01c u wont able to see mpacket processing with "debug ip mfib pak G S"
another one BUG in NG Cisco IOS-XE? :0)