ISR 4431 Mgmt interface ARP issue

[Emmanuel Jason Israel]

Hi All,

We have a current design where in we use an ISR to be an OOB router we have 2 links connected to the WAN switch(production) one links is connected on a normal gig interface and the other link is via MGMT interface (gi0)

we use 2 sets of mgmt IP via SDwan(10.103.x.x/27) and another is via OOB (we use 1:1 NAT translation for backdoor access 10.84.3.x/27) we have 10.103.0.8 as the Inside global

Pro          Inside global            Inside local                  Outside local              Outside global
---          10.103.0.8              10.84.3.152                        ---                            ---

we are running on a problem where in the mgmt interface is the one being learned on the ARP table so our NAT ip doesnt work

rtmgt001#

Internet 10.103.0.6 169 c4b3.6abe.ad94 ARPA GigabitEthernet0/0/1
Internet 10.103.0.7 - c4b3.6abe.ad06 ARPA GigabitEthernet0/0/1
Internet 10.103.0.8 189 c4b3.6abe.ad94 ARPA GigabitEthernet0/0/1

GigabitEthernet0 is up, line protocol is up (LINK going to WAN switch)
Hardware is RP management port, address is c4b3.6abe.ad94 (bia c4b3.6abe.ad94)
Description: sgsin16swwan001-TW 2/0/6
Internet address is 10.103.0.6/27

rtmgt001#sho int gi0/0/01  (LINK going to WAN switch)
GigabitEthernet0/0/1 is up, line protocol is up
Hardware is ISR4331-3x1GE, address is c4b3.6abe.ad06 (bia c4b3.6abe.ad06)
Description: sgsin16swwan001-TW 1/0/6
Internet address is 10.103.0.7/27

anyone has experienced this issue? thank you

[M02@rt37]

Hello @Emmanuel Jason Israel 

Based on the information provided, it appears that the management interface (Gi0) is being learned in the ARP table instead of the NAT IP. This is likely because the router is receiving ARP requests for both the management interface and the NAT IP, and is responding with the management interface MAC address for both requests.

To resolve this issue, you could try configuring static ARP entries on the router for the NAT IP address, pointing to the MAC address of the WAN switch. This should ensure that the router responds to ARP requests for the NAT IP with the MAC address of the WAN switch, rather than the management interface mac address.

[Kanan Huseynli]

Hi,

topology is not  clear for me. You have 2 interfaces (g0/0/1 and mgmt port g0) which are connected to switch.

10.103.0.7 is g0/0/1 IP and 10.84.3.152 is oob port g0 IP address. Then you do NAT, on which device? Is it the same router? If, yes what is purpose of it? If you lose g0/0/1 ,you will now have access to g0 over NAT.

Please, give a bit details.

[Emmanuel Jason Israel]

Hi guys here is our topology below:

the OOB is an ISR 4431 connected to a c9300 the interface on WAN001 are just regular access ports configured with vlan100(mgmt vlan) on ISR they are a normal eth port (g0/0/1) and the mgmt interface G0

M02@rt37 

I tried configuring static ARP on the ISR with 10.103.0.8(NAT IP) to be to correspond to c4b3.6abe.ad06 (Gi0/0/1 mac) but did not work or am i missing something?

10.103.0.7 is gi0/0/1 on ISR and 10.103.0.6 is Gi0 on ISR, 10.84.3.152 is the nat inside local address on the DC its 1:1 nat translation we have a Tunnel connected from DC to OOB ISR router so incase 10.103.x.x went down which is SDwan we still have backdoor access on mgmt via 10.84.3.x IP the issue im encountering is the MGMT interface on ISR is answering to ARP which sends his mac and being binded instead to inside local IP of 10.103.0.8 since mgmt interface has its own separate vrf

[Kanan Huseynli]

Hi,

why is 10.84.3.152 is inside local? It should be inside global, could you share your NAT config, explain on which device it is and how inside/outside interfaces are configured with respect to ISR?

By the way, what is gateway address? Can you ping 10.103.0.6 from it?

To see arp in vrf, use show ip arp vrf {vrf_name} command.

[MHM Cisco World]

Not only mgmt have it vrf but I think it completely isolated from other data plane.

What is ios-xe you use ?

[paul driver]

Hello
Try disabling the rtr from creating an arp for the inside global address then create a static for the correct arp
ip nat inside source static x.x.x.x y.y.y.y no-alias 
arp y.y.y.y xxxx.xxxx.xxx arpa

[Emmanuel Jason Israel]

Hi guys

10.84.3.x is IP on the HQ thats why its inside local @Kanan Huseynli 

interface GigabitEthernet0/0/0
ip address 64.40.x.x 255.255.254.0
no ip redirects
no ip proxy-arp
ip nat inside
negotiation auto
no cdp enable
crypto map S2SVPN
ip virtual-reassembly

interface GigabitEthernet0/0/1
description sgsin16swwan001-TW 1/0/6
ip address 10.103.0.7 255.255.255.224
ip nat outside
media-type rj45
negotiation auto
ip virtual-reassembly

interface GigabitEthernet0
description sgsin16swwan001-TW 2/0/6
vrf forwarding Mgmt-intf
ip address 10.103.0.6 255.255.255.224
ip verify unicast source reachable-via rx
negotiation auto

sho ip arp vrf Mgmt-intf
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.103.0.1 0 000c.2951.1c63 ARPA GigabitEthernet0
Internet 10.103.0.5 25 e069.ba58.8a51 ARPA GigabitEthernet0
Internet 10.103.0.6 - c4b3.6abe.ad94 ARPA GigabitEthernet0
Internet 10.103.0.7 112 c4b3.6abe.ad06 ARPA GigabitEthernet0
Internet 10.103.0.8 - c4b3.6abe.ad94 ARPA GigabitEthernet0
Internet 10.103.0.10 33 5ced.8c32.4bd0 ARPA GigabitEthernet0
Internet 10.103.0.11 208 5ced.8c32.5b10 ARPA GigabitEthernet0
Internet 10.103.0.12 181 507c.6f2b.8526 ARPA GigabitEthernet0
Internet 10.103.0.13 127 507c.6f2b.7845 ARPA GigabitEthernet0
Internet 10.103.0.14 139 000c.2999.af45 ARPA GigabitEthernet0
Internet 10.103.0.15 24 000c.294a.a39a ARPA GigabitEthernet0

sho ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.103.0.1 0 000c.2951.1c63 ARPA GigabitEthernet0/0/1
Internet 10.103.0.5 37 e069.ba58.8a51 ARPA GigabitEthernet0/0/1
Internet 10.103.0.6 112 c4b3.6abe.ad94 ARPA GigabitEthernet0/0/1
Internet 10.103.0.7 - c4b3.6abe.ad06 ARPA GigabitEthernet0/0/1
Internet 10.103.0.10 239 5ced.8c32.4bd0 ARPA GigabitEthernet0/0/1
Internet 10.103.0.11 162 5ced.8c32.5b10 ARPA GigabitEthernet0/0/1
Internet 10.103.0.12 183 507c.6f2b.8526 ARPA GigabitEthernet0/0/1
Internet 10.103.0.13 216 507c.6f2b.7845 ARPA GigabitEthernet0/0/1
Internet 10.103.0.14 144 000c.2999.af45 ARPA GigabitEthernet0/0/1
Internet 10.103.0.15 121 000c.294a.a39a ARPA GigabitEthernet0/0/1

WAN Switch interface

interface TwentyFiveGigE1/0/6
description sgsin16rtmgt001 - Gi0/0/1
switchport access vlan 100
switchport mode access
switchport nonegotiate
spanning-tree portfast
end

Current configuration : 177 bytes
!
interface TwentyFiveGigE2/0/6
description sgsin16rtmgt001 - Gi0 mgmt
switchport access vlan 100
switchport mode access
switchport nonegotiate
spanning-tree portfast
end

@MHM Cisco World 
Cisco IOS XE Software, Version 17.06.05
Cisco IOS Software [Bengaluru], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 17.6.5, RELEASE SOFTWARE (fc2)

i will try this @paul driver 

ip nat inside source static x.x.x.x y.y.y.y no-alias
arp y.y.y.y xxxx.xxxx.xxx arpa

[Kanan Huseynli]

So, 10.84.3.x is IP/subnet from HQ and you do NAT on this ISR router to change IP from 10.84.3.152  to 10.103.0.8 which is over G0/0/1 interface. Through this NAT traffic should be sent to 10.103.0.6 which is OOB IP, right?

If yes, immediate question, what is purpose here? OOB depends on G0/0/1 which is another management interface. If you lose g0/0/1 then this NAT will not work. Btw, how routing is configured on global and vrf routing table for 10.84.3.x?

Or still I don't get topology and logic..

[Emmanuel Jason Israel]

Hi Kanan

the gateway for normal data traffic is via SDwan which the gateway is the WAN switch in case the SDwan is down we will have a backdoor access via OOB using s2s tunnel from HQ to this site from my laptop if i need to access lets say esx server with ip 10.x.x.x since SDwan is down i will access it via 10.84.3.x ip which has 1:1 nat on the fw in hq although if gi0/0/1 went down we still have console server connected to the WAN switch.

[Kanan Huseynli]

Hi,

still not clear )

You say NAT on FW , but you also have NAT on this ISR router or you have both? If below NAT is on the same ISR , then question: how did you configure routing in VRF ? Your default route should point 10.103.0.7 (the same router), not WAN switch to have symmetric traffic flow.

Pro Inside global Inside local Outside local Outside global
--- 10.103.0.8 10.84.3.152 --- ---

[paul driver]

Hello
what looks to the wan interface is on the inside nat domain

Are you trying to hairpin some local addressing to be reachable via the wan ip?

[jirka.novak]

Hello,

by chance I touched same issue...

Two lessons I learnt:

a) It looks the issue is in the way how ISR responds to ARP. I found that MGMT interface responds to any ARP request in IP range it belongs - it is probably caused by ip proxy-arp feature. It looks that MGMT responds to ARP requests sooner than ISR routing interface in same IP range therefore other party learns wrong MAC address and all NAT packets flows to MGMT interface If you configure no ip proxy-arp on MGMT, NAT starts to work.

b) It is useless to have IP interface of ISR and MGMT in same IP network. If you unconfigure IP address from MGMT or choose different range, issue is away too. You can manage ISR on IP interface address and you don't need MGMT at all.

Best regards,

Jirka Novak