Nexus vrf routing issue?

[jiahmed]

Hello everyone, 

I'm having a weird issue with the below image topology. I have a Nexus 9k with an additional vrf created to isolate some LANs (represented by loopbacks for my testing). I'm not trying to route directly from one vrf to the other, but rather through another device and a firewall. The issue I'm running into is that traffic seems to die when it reaches from one side of the 9k to the other. As configured, I can ping Lo72 from the bottom left router just fine, I can ping it from the firewall as well, but I cannot ping it from the 9k vrf RED. However, if I make an exact copy of lo72 on the router, pings work just fine. 

***EDIT: Configs added

N9K vrf default

***************

ip route 10.0.10.0/24 10.1.1.1

ip route 10.0.20.0/24 10.1.1.1

!

interface Ethernet1/9

 description Connection to BORDER

 ip address 10.1.1.2/30

 no shut

!

interface loopback72

 ip address 192.168.21.1/24

***************************************************************************
ROUTER

************

hostname BORDER
!
boot-start-marker
boot-end-marker
!
crypto ikev2 proposal PROP
encryption aes-gcm-128
prf sha256
group 5
!
crypto ikev2 policy IKEV2_POLICY
proposal PROP
!
crypto ikev2 keyring KEYRING
peer ALL
address 0.0.0.0 0.0.0.0
pre-shared-key local TEST
pre-shared-key remote TEST
!
crypto ikev2 profile IKEV2_PROFILE
match identity remote address 10.0.0.2 255.255.255.255
identity local address 10.0.0.1
authentication remote pre-share
authentication local pre-share
keyring local KEYRING
!
crypto ipsec transform-set TSET esp-aes 192 esp-sha256-hmac
mode tunnel
!
crypto map CM 10 ipsec-isakmp
set peer 10.0.0.2
set transform-set TSET
set ikev2-profile IKEV2_PROFILE
match address AC_GRAY
!
interface GigabitEthernet0/1
no switchport
ip address 10.1.1.1 255.255.255.252
duplex auto
speed auto
!
interface GigabitEthernet0/8
no switchport
ip address 10.0.0.1 255.255.255.252
duplex auto
speed auto
crypto map CM

!
ip forward-protocol nd
!
ip route 192.168.20.0 255.255.255.0 10.1.1.2
ip route 192.168.21.0 255.255.255.0 10.1.1.2
ip route 10.0.10.0 255.255.255.0 10.0.0.2
ip route 10.0.20.0 255.255.255.0 10.0.0.2
!
ip access-list extended AC_GRAY
permit ip 192.168.20.0 0.0.0.255 10.0.10.0 0.0.0.255
permit ip 192.168.20.0 0.0.0.255 10.0.20.0 0.0.0.255
permit ip 192.168.21.0 0.0.0.255 10.0.10.0 0.0.0.255
permit ip 192.168.21.0 0.0.0.255 10.0.20.0 0.0.0.255
!

*******************************************************************************

FIREWALL

***********

interface GigabitEthernet1/1
nameif inside
security-level 100
ip address 10.2.2.1 255.255.255.0
!
interface GigabitEthernet1/8
nameif outside
security-level 0
ip address 10.0.0.2 255.255.255.252
!
interface Management1/1
management-only
nameif MGT
security-level 0
ip address 1.0.0.1 255.255.255.252
!
access-list RED extended permit ip 10.0.10.0 255.255.255.0 192.168.20.0 255.255.255.0
access-list RED extended permit ip 10.0.20.0 255.255.255.0 192.168.20.0 255.255.255.0
access-list RED extended permit ip 10.0.10.0 255.255.255.0 192.168.21.0 255.255.255.0
access-list RED extended permit ip 10.0.20.0 255.255.255.0 192.168.21.0 255.255.255.0
access-list TESTICMPIN extended permit icmp host 10.0.10.1 host 192.168.21.1
access-list TESTICMPOUT extended permit icmp host 192.168.21.1 host 10.0.10.1
access-group TESTICMPIN in interface inside
access-group TESTICMPOUT in interface outside
mtu inside 1500
mtu outside 1500
mtu MGT 1500
icmp unreachable rate-limit 1 burst-size 1
route outside 192.168.20.0 255.255.255.0 10.0.0.1 1
route outside 192.168.21.0 255.255.255.0 10.0.0.1 1
route inside 10.0.10.0 255.255.255.0 10.2.2.2 1
route inside 10.0.20.0 255.255.255.0 10.2.2.2 1
user-identity default-domain LOCAL
aaa authentication enable console LOCAL
aaa authentication login-history
http server enable
http 10.100.26.0 255.255.255.0 outside
no snmp-server location
no snmp-server contact
service sw-reset-button
crypto ipsec ikev2 ipsec-proposal TSET
protocol esp encryption aes-192
protocol esp integrity sha-256
crypto ipsec security-association pmtu-aging infinite
crypto map CM 10 match address RED
crypto map CM 10 set peer 10.0.0.1
crypto map CM 10 set ikev2 ipsec-proposal TSET
crypto map CM interface outside
crypto ca trustpool policy
crypto ikev2 policy 10
encryption aes-gcm
integrity null
group 5
prf sha256
lifetime seconds 86400
crypto ikev2 enable outside
group-policy DfltGrpPolicy attributes
vpn-tunnel-protocol ikev2
dynamic-access-policy-record DfltAccessPolicy
tunnel-group 10.0.0.1 type ipsec-l2l
tunnel-group 10.0.0.1 ipsec-attributes
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
!
class-map inspection_default
match default-inspection-traffic
!

*****************************************************************************

N9K vrf RED

*************

interface Ethernet1/11

 description Connection to FW Inside

 vrf member RED

 ip address 10.2.2.2/30

 no shutdown

interface loopback201

 vrf member RED

 ip address 10.0.10.1/24

vrf context RED

 ip route 192.168.20.0/24 10.2.2.1

 ip route 192.168.21.0/24 10.2.2.1

 address-family ipv4 unicast

[balaji.bandi]

Nature of the FW default deny, do you have any Access Policy to allow this ? (high level ?)

also what source you using to ping ?

[jiahmed]

Hi,

I do have access rules in the firewall permitting icmp from lo72 to lo201. From vrf RED I do ping 192.168.21.1 source-interface lo201

[balaji.bandi]

I do ping 192.168.21.1 source-interface lo201

what you see logs in FW ? is the FW do any NAT ? what FW is this ?

From FW are you able to ping 192.168.21.1 ? and 10,.0.10.1 ?

[jiahmed]

The traffic is permitted in the FW logs, I don't think the FW is the issue. I can ping from 192.168.21.1 to 10.0.10.1 just fine if the loopback is on my router, so the firewall isn't blocking it. But when I move 192.168.21.1 to the Nexus, pings stop working.

EDIT: Configs added to original post

[balaji.bandi]

When you say VRF default, is the VRF name  Default?

your Global Routing table do not aware of your VRF routing right ?

post-show IP route and vrf routing to understand.

[jiahmed]

Will do when I get in tomorrow. By Default I mean the global routing table.
There is only one additional vrf named Red

[jiahmed]

show vrf

VRF-Name                                                             VRF-ID State Reason

RED                                                                               5    Up    --

default                                                                           1    Up    --

management                                                                  2   Up     --

show ip route static

IP Route Table for VRF "default"

"*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denoted VRF

10.0.10.0/24, ubest/mbest: 1/0

     *via 10.1.1.1, [1/0], 1d23h, static

10.0.20.0/24, ubest/mbest: 1/0

     *via 10.1.1.1, [1/0], 1d23h, static

show ip route static vrf RED

IP Route Table for VRF "RED"

"*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denoted VRF

192.168.20.0/24, ubest/mbest: 1/0

     *via 10.2.2.1, [1/0], 1d03h, static

192.168.21.0/24, ubest/mbest: 1/0

     *via 10.2.2.1, [1/0], 1d03h, static

[balaji.bandi]

you need to issue show ip route with VRF to look, and GRT it not learned.