ISP NAT / ARP Issue

[jonathanw84]

We have a 1941 that we use at a residence with a Frontier FIOS circuit. They provide us with 5 static IPs and we NAT different internal networks to each of the addresses - a pretty standard config. As of a week ago, this stopped working with the exception of the main IP. The ISP says there is nothing on their end that is wrong, but nothing has changed on our end and this has been working for around 6 months. We are also seeing entries in the arp table that show the ISP as the hardware address:

Internet X.X.X.1 -   204e.71c5.31c8 ARPA GigabitEthernet0/1
Internet X.X.X.37 - c47d.4f75.21e1 ARPA GigabitEthernet0/1
Internet X.X.X.38 - 204e.71c5.31c8 ARPA GigabitEthernet0/1
Internet X.X.X.39 - 204e.71c5.31c8 ARPA GigabitEthernet0/1

I have tried doing a static arp entry with no success as well. It's a very standard and simple configuration and while I think it's an ISP issue (perhaps they enabled proxy-arp or something like that), I wanted to check here as well. Thanks!

Relevant Configuration:

interface GigabitEthernet0/1
bandwidth 100000
ip address X.X.X.37 255.255.255.0
ip nat outside
ip virtual-reassembly in
zone-member security OUTSIDE
duplex auto
speed auto

ip route 0.0.0.0 0.0.0.0 X.X.X.1

ip nat pool GUEST X.X.X.39 X.X.X.39 netmask 255.255.255.0
ip nat pool CAMERAS X.X.X.38 X.X.X.38 netmask 255.255.255.0

ip nat inside source route-map CAMERAS pool CAMERAS overload
ip nat inside source route-map GUEST pool GUEST overload
ip nat inside source route-map INTERNAL interface GigabitEthernet0/1 overload

route-map INTERNAL permit 10
match ip address NETWORKS_INTERNAL
match interface GigabitEthernet0/1

!

route-map CAMERAS permit 10
match ip address NETWORKS_CAMERAS
match interface GigabitEthernet0/1
!
route-map GUEST permit 10
match ip address NETWORKS_GUEST
match interface GigabitEthernet0/1

[balaji.bandi]

As of a week ago, this stopped working with the exception of the main IP. 

what is stop working NAT ?   and Only the Connected interface and configurd working as expected(X.X.X.37) , rest all not working.

what is the Logs shows on the device ? have you rebooted the Router and tested. If the ISP side issue (for instance) the main IP should also not work right ?

This one works right

ip nat inside source route-map INTERNAL interface GigabitEthernet0/1 overload

For testing  Try same other Pool also with G0/1 ? is that works ?

Also enable debug and see what is wrong ? 

[paul driver]

Hello
Has anything be changed on the rtr regards the zbfw policys or access-lists relating to the NAT

[jonathanw84]

Hi Paul,

Nope. Nothing was changed. This has been working for months. What's odd is that if I clear out my NAT configuration and re-apply it, it will work again for a while and then the arp table will revert back to showing the ISP MAC as the hardware address. 

[balaji.bandi]

 if I clear out my NAT configuration and re-apply it, it will work again for a while and then the arp table will revert back to showing the ISP MAC as the hardware address. 

This Looks like having - may be thinking NAT  XLATE issue on the router  - how big is your network, what kind of userbase/bandwidth/load on Router?

worth reading :

https://www.cisco.com/c/en/us/support/docs/ip/network-address-translation-nat/8605-13.html#table

[jonathanw84]

Hi BB,

This is for a home user so it's a very small network. CPU on the router is low and there are only a handful of NAT translations. This was working fine until about two weeks ago. It's a standard and simple configuration.

Thanks!

[balaji.bandi]

Thanks for the input, we can understand. it not required to be a big network, sometimes even 1 user who also has attacks can fil NAT Table try to initiate many connections. Do you have any incoming port-forward? when you have an issue, worth capturing the information to identify the issue.(this was just a suggestion)

can you also post-show the version from the device? if not a major security concern posts the config also.

[paul driver]

Hello

Has the rtr been reloaded ?
Sh ip nat statistics

sh log 

Can you check the the licence for that rtr 

show license statistics 
show license status
show licence  detail

Can you post in a file the running configuration and the output from the above please

[jonathanw84]

Hi All,

Sorry for the delayed response. The router has been reloaded numerous times and this seems to fix things for a while. I have an IP SLA that is pinging 8.8.8.8 from one of the networks that is set up for NAT and it works for about an hour, and then it times out. Here is the status right now:

*2 icmp-echo 8.8.8.8 - Timeout 26 seconds ago

This was working up until 20 minutes ago. Prior to that, I had removed and re-added the NAT overload statement:

ip nat inside source route-map CAMERAS pool CAMERAS overload

Here is the "show ip nat statistics" output:

#sho ip nat statistics
Total active translations: 29 (0 static, 29 dynamic; 29 extended)
Peak translations: 704, occurred 2w2d ago
Outside interfaces:
GigabitEthernet0/1, Cellular0/1/0
Inside interfaces:
Vlan112, Vlan172, Vlan194, Vlan195
Hits: 46861122 Misses: 0
CEF Translated packets: 46250251, CEF Punted packets: 493203
Expired translations: 276579
Dynamic mappings:
-- Inside Source
[Id: 16] route-map CAMERAS pool CAMERAS refcount 3
pool CAMERAS: netmask 255.255.255.0
start X.X.X.38 end X.X.X.38
type generic, total addresses 1, allocated 1 (100%), misses 0
[Id: 14] route-map GUEST interface GigabitEthernet0/1 refcount 18
[Id: 8] route-map INTERNAL interface GigabitEthernet0/1 refcount 8
[Id: 4] route-map VZW interface Cellular0/1/0 refcount 0

And here is the "show ip arp" output:

#show ip arp

Internet X.X.X.1 0 204e.71c5.31c8 ARPA GigabitEthernet0/1
Internet X.X.X..37 - c47d.4f75.21e1 ARPA GigabitEthernet0/1
Internet X.X.X..38 30 204e.71c5.31c8 ARPA GigabitEthernet0/1

You can see that the .1 and .38 address have the same MAC. That MAC belongs to the gateway router of the ISP. The MAC address for the .37 is from my router. As soon as I clear the NAT overload configuration for the .38 address, the MAC changes to my router and works for about an hour and then stops. The fact that it works for a while and then fails is odd.

I've also attached the rest of the relevant config. 

Thanks!

[Giuseppe Larosa]

Hello Jonathan,

it is quite strange that MAC entry for X.X.X.38 is overriden by ISP GW MAC address after one hour.

In an attempt to protect your router from these unwanted ARP entries you can add arp statements

arp X.X.X.38 c47d.4f75.21e1 ARPA

arp X.X.X.39 c47d.4f75.21e1 ARPA

As actually these should be the correct value for the MAC address.

The question is who is on the pubic subnet making ARP requests for X.X.X.38 ?

Only X.X.X.1 should send ARP requests at the beginning and it should get answer from your router with gi0/1 MAC address

Hope to help

Giuseppe

[jonathanw84]

Thanks Giuseppe. Definitely strange. I did try a static arp entry and the issue came back. I just upgrade the IOS on the router so perhaps that will fix it. I should know in two hours.

[jonathanw84]

Kind of a strange development after no help from the ISP. We also have a 4G cellular connection on this router as a backup. This has been in place for years. Recently though, something must have changed because the cellular signal has really degraded. Anyway, I noticed that the cellular interface was flapping throughout the day, and though perhaps it could be related to the weird NAT / ARP issue we are facing on our main circuit, so I shut down the cellular interface and since then, the NAT / ARP issue on the main circuit has gone away. Any ideas as to why this would happen? I am stumped as this has been a working configuration for years. Thanks!

[Giuseppe Larosa]

Hello @jonathanw84 ,

thanks for your valuable feedaback.

There is no easy explanation for what was happening.

ARP is local to a LAN interface.

The cellular interface is a backup interface and it is also an async serial interface where ARP is not used.

Being the cellular interface a backup interface it shouldn't be causing issues with ARP on the main link, unless you have additional static NAT statements using the cullular interface involving the same hosts exposed with the static NAT on the main WAN interface.

But this is just a guess.

Hope to help

Giuseppe

[jonathanw84]

Hi Giuseppe. Thank you for the response.

Being that the cellular interface is a backup, we do have NAT statements using the cellular interface that do involve the same networks that are being NATd on the main circuit. This has been working for years without issue but not it breaks unless I shut down the cellular interface. I've provided some more specifics on the configuration below if you could take a look. Thanks! 

ip nat pool CAMERAS X.X.X.38 X.X.X.38 netmask 255.255.255.0 (main circuit)

ip nat pool GUEST X.X.X.39 X.X.X.39 netmask 255.255.255.0 (main circuit)

ip nat pool INTERNAL X.X.X.40 X.X.X.40 netmask 255.255.255.0 (main circuit)

!

ip nat inside source route-map CAMERAS pool CAMERAS overload (main circuit)

ip nat inside source route-map GUEST pool GUEST overload (main circuit)

ip nat inside source route-map INTERNAL pool INTERNAL overload (main circuit)

!

ip nat inside source list NETWORKS_NAT interface Cellular0/1/0 overload (used for cellular backup)

!

ip access-list extended NETWORKS_CAMERAS
permit ip 172.20.16.0 0.0.0.255 any

!
ip access-list extended NETWORKS_GUEST
permit ip 10.194.4.0 0.0.0.255 any

!
ip access-list extended NETWORKS_INTERNAL
permit ip 10.112.4.0 0.0.0.255 any
permit ip 10.195.4.0 0.0.0.255 any

!
ip access-list extended NETWORKS_NAT (used for cellular NAT but contains same networks)
permit ip 10.112.4.0 0.0.0.255 any
permit ip 10.195.4.0 0.0.0.255 any
permit ip 10.194.4.0 0.0.0.255 any
deny ip any any

!

route-map INTERNAL permit 10
match ip address NETWORKS_INTERNAL
match interface GigabitEthernet0/1
!
route-map CAMERAS permit 10
match ip address NETWORKS_CAMERAS
match interface GigabitEthernet0/1
!
route-map GUEST permit 10
match ip address NETWORKS_GUEST
match interface GigabitEthernet0/1

[paul driver]

Hello
Looking at your configuration your ZBFW and Nat its has duplication and missing access-lists also in your two previous posts your are showing different nat configuration, so which one is correct!

NAT
ip nat pool GUEST X.X.X.39 X.X.X.39 netmask 255.255.255.0
ip nat pool CAMERAS X.X.X.38 X.X.X.38 netmask 255.255.255.0
ip nat inside source route-map CAMERAS pool CAMERAS overload
ip nat inside source route-map GUEST interface GigabitEthernet0/1 overload
ip nat inside source route-map INTERNAL interface GigabitEthernet0/1 overload

or
ip nat pool CAMERAS X.X.X.38 X.X.X.38 netmask 255.255.255.0
ip nat pool GUEST X.X.X.39 X.X.X.39 netmask 255.255.255.0
ip nat pool INTERNAL X.X.X.40 X.X.X.40 netmask 255.255.255.0
ip nat inside source route-map CAMERAS pool CAMERAS overload
ip nat inside source route-map GUEST pool GUEST overload
ip nat inside source route-map INTERNAL pool INTERNAL overload
ip nat inside source list NETWORKS_NAT interface Cellular0/1/0 overload (used for cellular backup

ZBFW
INSIDE-OUTSIDE policy <--– non existing access-lists
OUTSIDE-SELF policy <--– missing  access-list INET_PROTECT_VZW, Also acl INET_PROTECT is very convoluted, you have many deny ace statements but as you are deny everything at the end anyway so you could amend this so you permit your specific traffic then deny the rest

DMZ-OUTSIDE  & CAMERAS-OUTSIDE policy’s <---are matching on the exact protocols in their class-maps so why not just use the one?