NAT inside through different SVIs and from multiple sources

[mury]

I have a 9407 that connects to two 2901s. OSPF and iBGP are running on them. The 2901s are redundant routers that terminate tunnels to a wireless carrier using eBGP.

Please note that I do not have permission to change the design (wishy-washy term, I know.) I just have to fix it. NAT from the outside (public) to the inside works. NAT coming from the inside does not. 
On the 9407 side, the ports to the 2901s go out a link that is in native vlan 790. Here is the relevant config:

interface GigabitEthernet1/0/7 and 2/0/7   <--- note there is an interface going out to each VPN router
description To VPN routers G0/0
switchport trunk native vlan 790

interface Vlan790
description VerizonVPN /29 Subnet
ip address 10.10.73.209 255.255.255.248 <- Note first two octets have been changed for privacy
ip nat inside  

interface Vlan440
description /28 BGP Core, LTE NAT 
ip address 10.10.74.158 255.255.255.224 secondary  <---- Public IPs for CPE to be natted
ip address 10.10.73.248 255.255.255.240
ip nat outside

Now on the VPN router side iBGP and OSPF are running. 

vpn0.2901#sho ip inter bri
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 10.10.73.211 YES NVRAM up up
Loopback0 10.10.72.18 YES NVRAM up up
Tunnel0 172.20.20.2 YES NVRAM up up

Abbreviated routing table on VPN routers:

Gateway of last resort is 10.10.73.7 to network 0.0.0.0 <- Note this is a loopback address on the 9407

B* 0.0.0.0/0 [200/0] via 10.10.73.7, 15:42:08
C 10.10.72.18 is directly connected, Loopback0
O 10.10.72.19 [110/2] via 208.84.73.212, 15:42:12, GigabitEthernet0/0  <-- other VPN router
10.10.73.0/24 is variably subnetted, 8 subnets, 4 masks

O 10.10.73.1/32

[110/3] via 10.10.73.209, 15:42:12, GigabitEthernet0/0
O 10.10.73.7/32
[110/2] via 10.10.73.209, 15:42:12, GigabitEthernet0/0
O 10.10.73.8/32
[110/3] via 10.10.73.209, 15:42:12, GigabitEthernet0/0
O 10.10.73.9/32
[110/3] via 10.10.73.209, 15:42:12, GigabitEthernet0/0
O 10.10.73.200/30
[110/3] via 10.10.73.209, 15:42:12, GigabitEthernet0/0
C 10.10.73.208/29 is directly connected, GigabitEthernet0/0
L 10.10.73.211/32 is directly connected, GigabitEthernet0/0
O 10.10.73.240/28 - Note this is the network with an address assigned to the nat inside interface on 9407
[110/2] via 10.10.73.209, 15:42:12, GigabitEthernet0/0

So looking at these IP address/blocks on the 9407:

C 10.10.73.7/32 is directly connected, Loopback0

L 10.10.73.209/32 is directly connected, Vlan790

C 10.10.73.240/28 is directly connected, Vlan440

Then back on the VPN routers:
vpn0.2901#sho ip route 10.10.73.7
Routing entry for 10.10.73.7/32
Known via "ospf 1", distance 110, metric 2, type intra area
Last update from 10.10.73.209 on GigabitEthernet0/0, 15:53:11 ago
Routing Descriptor Blocks:
* 10.10.73.209, from 10.10.73.7, 15:53:11 ago, via GigabitEthernet0/0
Route metric is 2, traffic share count is 1

So the default route is to the Loopback on the 9407, and it is being learned from OSPF from a neighbor address that is in vlan 790.

So I either want to change the default routes on the VPN routers to point to an address on the 9407 that is in vlan 440, or I want to change where the ip NAT outside statement resides.

Am I on the right path? According to Cisco when the path is Inside-to-Outside routing happens before NAT. So is that traffic coming in on vlan 790 then hitting the default route which does happen to be in vlan 440, but because of the SVI routing that has to happen first the NAT rule never gets applied? Can this all be fixed by changing the default route on the VPN routers to hit the SVI interface in vlan 440 on the 9407, which the VPN routers know about via OSPF - 10.10.73.248?

Or, do I have to look at changing where the ip NAT outside is configured on the 9407?

As mentioned before I do not have much latitude here. The person that set all this up no longer works for the company. I was brought in mostly to document the network, but as problems have popped up I'm getting pulled into some troubleshooting too. However, I don't have permission to change much and I need to be "100%" sure I don't break anything. There is no room for experimenting with solutions.

Note, I have very minimal experience with NAT on Ciscos. Just basic stuff where you put the NAT inside and outside config on physical interfaces. In this case I'm dealing with the NAT statements on SVIs and I'm just not completely sure how it all flows.

Oh, not sure it makes a difference but here is an example of the NAT config:

ip nat inside source static 172.30.1.2 10.10.74.129
ip nat inside source static 172.30.1.4 10.10.74.130


Thanks in advance!

[MHM Cisco World]

can you draw the topology

[paul driver]

Hello

"Interface Vlan440
description /28 BGP Core, LTE NAT 
ip address 10.10.74.158 255.255.255.224 secondary  <---- Public IPs for CPE to be natted"

Can you elaborate why do you have primary  and secondary addressing on this L3 vlan and only the secondary is to be natted? 

[mury]

I can try. I can't fully explain the original intent, but the primary IP block for Vlan440 is for what the previous person called the BGP network. The devices on this network are the two border routers to the carriers, this switch (9407), and I believe a device at a different location connected via layer 2. Anyway, the primary usage is to advertise iBGP routes (mostly just default routes) from the border routers to other internal devices that he set up. There are a few routes being advertised back from the other devices as well.  It is this secondary block that he set up as public <-> private NAT for devices location at customers. So this inside static NAT allows customers access from their remote sites to come over 4G into the carrier's LTE network, and then over a tunnel to us to what I am calling the VPN routers. I hope that makes more sense. I still need to provide a diagram for "MHM Cisco World" as well. I cannot provide the actual diagram I made up for the company, so I am just trying to find time to get the relevant pieces into a separate diagram. 

I can try to type it out as well, but not sure how well this will turn out:

Border Router0 ------                     ----- VPN Router0 ----- Tunnel to LTE
        |                        --- 9407 ---
Border Router1 ------                     ----- VPN Router1 ----- Tunnel to LTE

One thing to note is the tunnels obviously come in over GRE through the Border Routers, through the 9407, before terminating in the VPN Routers.

[paul driver]

Hello
Its hard to follow your topology, but it seems:
vpn routers are on the inside of your network via (vl790)
Border routers are on the outside of you network via (vl440
cat9407 is the L3 in between performing the nat

if im correct then the below statement doesn't make sense, unless this default is being received over the GRE tunnel of the vpn router then re advertised back to the cat9407 via ospf?

"So the default route is to the Loopback on the 9407, and it is being learned from OSPF from a neighbour address that is in vlan 790."

Regards the secondary addressing, it seems at this point its not really applicable as you can basically nat on any ip address as long as it reachable externally and your nat device it aware of it.

Can you post some outputs ?
sh ip int brief
sh ip protocols

[mury]

"Its hard to follow your topology, but it seems:
vpn routers are on the inside of your network via (vl790)
Border routers are on the outside of you network via (vl440
cat9407 is the L3 in between performing the nat"

Yes, the topology *is* hard to follow and I even have the advantage of looking at it. But what you state is correct.

"Regards the secondary addressing, it seems at this point its not really applicable as you can basically nat on any ip address as long as it reachable externally and your nat device it aware of it."

That is something I've been confused about from the beginning. I am assuming he had to put the addresses to be NAT'ed somewhere and since their access on the outside had to be through VLAN440 to the border routers he just added the block as secondary on the Vlan440 interface. There is evidence of him trying to configure NAT on the VPN routers which I'm guessing failed because the packets could come out either tunnel to either VPN router from the LTE carrier. 

9407R#sho ip protocols
*** IP Routing is NSF aware ***

Routing Protocol is "ospf 1"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is (prefix-list) OSPFDenyDefault   <--- prevents default route from iBGP from being overwritten by default routes coming in from other parts of the network over OSPF

Router ID .73.7
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
.73.7 0.0.0.0 area 0      <---- Loopback
.73.208 0.0.0.7 area 0     <---- .209 IP in vlan 790
.73.240 0.0.0.15 area 0      <---- .248 IP in vlan 440
Routing Information Sources:
Gateway Distance Last Update
.73.9 110 3w6d <-- border 1
.73.8 110 3w6d  < -- border 0
.73.6 110 46w1d  <--- I think this device is removed. 
.72.17 110 6w6d  <---- IP of a 3750X at a different location connected via layer 2
.72.18 110 2d10h  <--- vpn 0
.72.19 110 1y5w  <--- vpn 1
Distance: (default is 110)

Routing Protocol is "bgp XXXXX"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Route Reflector for address family IPv4 Unicast, 2 clients
IGP synchronization is disabled
Automatic route summarization is disabled
Redistributing: static
Neighbor(s):
Address FiltIn FiltOut DistIn DistOut Weight RouteMap
.72.18 mycompany-in     <---- VPN router 0
.72.19 mycompany-in      <----- VPN router 1
.73.8 mycompany-in         <----- border router 0
.73.9 mycompany-in        <------- border router 1
Maximum path: 2
Routing Information Sources:
Gateway Distance Last Update
.73.9 200 3w6d
.73.8 200 3w6d
.72.18 200 1d11h
.72.19 200 2d10h
Distance: external 20 internal 200 local 200

Note I've removed the first two octets for privacy but they are all the same... they are all in the /22. In my first post I replaced them with 10.10, but could be any publicly routable IPs you want them to be.


9407R#sho ip inter bri
Interface IP-Address OK? Method Status Protocol
Vlan1 unassigned YES NVRAM administratively down down
Vlan103 .72.81 YES manual up up
Vlan104 .72.97 YES manual up up
Vlan105 .73.161 YES manual up up
Vlan106 .73.49 YES manual up up
Vlan107 .72.105 YES manual up up
Vlan109 72.153 YES manual up up
Vlan110 .72.129 YES manual up up
Vlan410 172.16.30.3 YES manual up up
Vlan411 .72.177 YES manual up up
Vlan420 10.99.2.21 YES manual up up
Vlan430 10.51.30.72 YES manual up up
Vlan431 10.51.31.72 YES manual up up
Vlan440 .73.248 YES NVRAM up up
Vlan460 192.168.85.11 YES manual up up
Vlan511 .73.225 YES manual up up
Vlan682 172.20.22.250 YES manual up up
Vlan700 .72.25 YES manual up up
Vlan701 10.5.1.1 YES manual up up
Vlan703 10.105.2.40 YES manual up up
Vlan704 10.105.4.1 YES manual up up
Vlan712 .72.33 YES manual up up
Vlan721 .72.1 YES manual up up
Vlan729 .72.9 YES manual up up
Vlan730 .73.41 YES manual up up
Vlan790 .73.209 YES manual up up
Vlan800 10.8.2.1 YES manual up up
Vlan901 .73.113 YES manual up up

GigabitEthernet1/0/7 unassigned YES unset up up

Inserting interface config ---
interface GigabitEthernet1/0/7
description vpn0.2901 G0/0
switchport trunk native vlan 790
switchport mode trunk
spanning-tree portfast trunk
---

GigabitEthernet1/0/8 unassigned YES unset up up

---

interface GigabitEthernet1/0/8
description vpn0.2901 G0/1
switchport trunk native vlan 440
switchport mode trunk
spanning-tree portfast trunk

---




GigabitEthernet2/0/7 unassigned YES unset up up
GigabitEthernet2/0/8 unassigned YES unset up up

Te5/0/1 unassigned YES unset up up
Te5/0/2 unassigned YES unset up up
Te5/0/3 unassigned YES unset up up

Te6/0/1 unassigned YES unset up up
Te6/0/2 unassigned YES unset up up
Te6/0/3 unassigned YES unset up up
Te6/0/4 unassigned YES unset up up

Port-channel4 unassigned YES unset up up
Port-channel48 unassigned YES unset down down
Port-channel51 unassigned YES unset up up
Port-channel52 unassigned YES unset up up
Loopback0 .73.7 YES NVRAM up up

[paul driver]

Hello

---

@mury wrote:

interface Vlan440
description /28 BGP Core, LTE NAT 
ip address 10.10.74.158 255.255.255.224 secondary  <---- Public IPs for CPE to be natted
ip address 10.10.73.248 255.255.255.240

9407R#sho ip inter bri
Vlan440 .73.248 YES NVRAM up up

NAT from the outside (public) to the inside works. NAT coming from the inside does not. 

---

can you add the following please and test:
ip route 10.10.74.128 255.255.255.224 vlan 440

[MHM Cisco World]

description /28 BGP Core, LTE NAT 
ip address 10.10.74.158 255.255.255.224 secondary  <---- Public IPs for CPE to be natted
ip address 10.10.73.248 255.255.255.240

ip nat inside source static 172.30.1.2 10.10.74.129 extendable 
ip nat inside source static 172.30.1.4 10.10.74.130 extendable 

you use secondary IP in ip nat command, can you try
clear NAT entry 
add extendable keyword to NAT 

[mury]

"ip nat inside source static 172.30.1.2 10.10.74.129 extendable 
ip nat inside source static 172.30.1.4 10.10.74.130 extendable 

you use secondary IP in ip nat command, can you try
clear NAT entry 
add extendable keyword to NAT "

Isn't extendable when I need to map two different public IPs to a single private IP? My understanding of extendable could be wrong. We have an ASN with a single /22 tied to it that we announce out to both ISPs. So both border routers announce our /22 and they both receive full routing tables. iBGP runs between the border routers and the 9407. The VPN routers are iBGP route reflector clients to the 9407.  However, even if I'm right this does bring up a thought. If the timestamps are correct, this cannot be the problem, since the NAT broke before this change. However, if the timestamps are wrong iBGP multipath was configured shortly after the problems started happening.