Cisco 1921 Routing/NAT Issue

[ewellsie07]

I've recently inherited a new client that currently sits on a flat network (192.168.2.0/24) spanning two physically separate buildings, connected by fiber.

In order to reach compliance, I need to separate those two buildings logically, and be able to control access between the two.

Since we already had a 1921 in place at one of the buildings for a separate purpose, the idea was to add in a EHWIC that would accept the fiber connection and split the networks using the router to control access, at least until we can properly redesign the network.

The ISP service comes in at the primary location where it connects to the ASA5505, and the 1921 router is located at the secondary location. There needs to be static incoming traffic to both the primary and secondary site, and there are VPN clients that also need to reach the secondary location. The other inside interface on the 1921 is a separate extranet that will only need to accept traffic from the 192.168.2.0/24 and the VPN clients and includes some routes for the 65.xx.xx.xx networks.

As the configs will show, the plan was to create a 192.168.6.0/24 subnet at the primary location, and keep the existing 192.168.2.0/24 subnet at the secondary location. Configured as below, the primary location is working just fine, and all traffic flows in and out. We can also ping from a host at the primary location to a host within the secondary location. However, we cannot ping from the secondary to the primary location, and the secondary location hosts do not have internet access.

The issue appears to be the 1921 not sending the traffic from 192.168.2.0/24 to 192.168.6.0/24 and I can't figure out why. I've tried various NAT statements but just can't seem to figure it out.

Pasted below are the relevant snippets of the ASA config, and the entire 1921 config.

Can anyone assist?

***********ASA**************

interface Vlan1
 nameif inside
 security-level 100
 ip address 192.168.6.1 255.255.255.0
!
interface Vlan2
 nameif outside
 security-level 0
 ip address 1xx.xx.xx.xx 255.255.255.224

ip local pool VPN 192.168.4.2-192.168.4.100 mask 255.255.255.0

access-list VPN-Access extended permit ip any 192.168.4.0 255.255.255.0
access-list VPNTunnelList standard permit 192.168.2.0 255.255.255.0
access-list VPNTunnelList standard permit 192.168.6.0 255.255.255.0
access-list VPNTunnelList standard permit host 65.xx.xx.xx
access-list VPNTunnelList standard permit host 65.xx.xx.xx  
access-list VPNTunnelList standard permit host 65.xx.xx.xx  
access-list VPNTunnelList standard permit host 65.xx.xx.xx  

icmp permit any inside

global (inside) 1 interface
global (outside) 1 interface
nat (inside) 0 access-list VPN-Access
nat (inside) 1 0.0.0.0 0.0.0.0
nat (outside) 1 192.168.4.0 255.255.255.0

static (inside,inside) 192.168.2.0 192.168.2.0 netmask 255.255.255.0
static (inside,inside) 192.168.6.0 192.168.6.0 netmask 255.255.255.0

route inside 65.xx.xx.xx 255.255.255.255 192.168.6.254 1
route inside 65.xx.xx.xx 255.255.255.255 192.168.6.254 1
route inside 65.xx.xx.xx 255.255.255.255 192.168.6.254 1
route inside 65.xx.xx.xx 255.255.255.255 192.168.6.254 1
route inside 1xx.xxx.0.0 255.255.0.0 192.168.6.254 1
route inside 192.168.2.0 255.255.255.0 192.168.6.254 1

********1921**********

version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname *********
!
boot-start-marker
boot-end-marker
!
!
logging buffered 51200 warnings
!
no aaa new-model
clock timezone EST -5 0
!
ip cef
!
!
!
!
!
!
ip domain name *******.local
ip name-server 192.168.6.20
no ipv6 cef
multilink bundle-name authenticated
!
!
crypto pki trustpoint TP-self-signed-4183443168
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-4183443168
 revocation-check none
 rsakeypair TP-self-signed-4183443168
!
!
crypto pki certificate chain TP-self-signed-4183443168
 certificate self-signed 01
  3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
  31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
  69666963 6174652D 34313833 34343331 3638301E 170D3134 30353036 32313032
  33335A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
  4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D34 31383334
  34333136 3830819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
  8100CD4C 75896018 68857109 B9ACBC96 0229831F 2F316A1B DA7EB9EA 5EFD488C
  E61EF6FE 49890039 CA5F6CE1 947826FA 949424D7 C6CC19C7 BA0F5E0D 4DAB0E5F
  A308668D 47130371 352225FB 77C23430 37110F37 FEC6C065 0791A1B0 DE3650DE
  398799F6 54E75454 C308320D 40B59B7C EA2560CA E78D8357 4EAB68BA FE73F549
  F6190203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
  551D2304 18301680 14C8C326 4B33918D 08A20CDE 0142E8F2 FE60648F 81301D06
  03551D0E 04160414 C8C3264B 33918D08 A20CDE01 42E8F2FE 60648F81 300D0609
  2A864886 F70D0101 05050003 8181000F F87EDB80 9BAA5943 61E07975 C34BEBB6
  C8343A10 A2A3CEC4 518260FD E5DABC89 4364118F A915A2B6 A994D156 0C58C555
  D6751F82 4D4E62BF 00B082FF 0115F2C9 7A0C0DC2 66DE8F9E 3478DCE7 713E2992
  5412BC1D EE44C152 8D8E3425 15BED73A 299B8D38 6CDB6667 955A3875 43E28416
  FC9BBBC8 396D826C A875E42B 43FDA4
      quit
license udi pid CISCO1921/K9 sn FGL181921L5
!
!
username admin privilege 15 secret 5 **********************
username ****** privilege 15 secret 5 ********************
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
interface GigabitEthernet0/0
 description ****** Network Interface
 ip address 16x.xx.xx.xx 255.255.255.240
 ip access-group 100 in
 ip access-group 100 out
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip flow ingress
 ip nat outside
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 description ******** LAN Interface
 ip address 192.168.2.254 255.255.255.0
 ip access-group 100 in
 ip access-group 100 out
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip flow ingress
 ip nat inside
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface GigabitEthernet0/1/0
 description Fiber Interface to *********
 ip address 192.168.6.254 255.255.255.0
 ip access-group 100 in
 ip access-group 100 out
 ip nat inside
 ip virtual-reassembly in
 duplex auto
 speed auto
 media-type sfp
!
ip default-gateway 192.168.6.1
ip forward-protocol nd
!
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip nat pool POOLNAME 1xx.xx.xx.xx 1xx.xx.xx.xx netmask 255.255.255.240
ip nat inside source list 102 pool POOLNAME overload
ip route 0.0.0.0 0.0.0.0 192.168.6.1
ip route 65.xx.xx.xx 255.255.255.255 1xx.xx.xx.xx
ip route 65.xx.xx.xx 255.255.255.255 1xx.xx.xx.xx
ip route 65.xx.xx.xx 255.255.255.255 1xx.xx.xx.xx
ip route 65.xx.xx.xx 255.255.255.255 1xx.xx.xx.xx
ip route 1xx.xxx.0.0 255.255.0.0 1xx.xx.xx.xx
ip route 192.168.4.0 255.255.255.0 192.168.6.1
!
access-list 100 permit ip any any
access-list 102 permit ip 192.168.2.0 0.0.0.255 any
access-list 102 permit ip 192.168.4.0 0.0.0.255 any
no cdp advertise-v2
!
!
!
control-plane
!
!
!
line con 0
 login local
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 access-class 23 in
 privilege level 15
 login local
 transport input telnet ssh
line vty 5 15
 access-class 23 in
 privilege level 15
 login local
 transport input telnet ssh
!
scheduler allocate 20000 1000
!
end

[Jon Marshall]

So any 192.168.6.x client uses the ASA as it's default gateway ?

So lets work out which device is stopping it working.

Can you pick a client on the 192.168.6.x subnet and set it's default gateway to be 192.168.6.254 and then try pinging both ways.

Also why do you have "ip nat inside" on that interface on the router ?

Jon

[ewellsie07]

Yes, the 192.168.6.0/24 clients would use the ASA inside as their default gateway, the 192.168.2.0/24 clients would use the LAN interface of the 1921, which would then have the ASA as the default gateway.

Unfortunately, since this is a live network and the secondary location is critical for operations, for the time being I've reverted it back to the prior configs and the flat network.

Any changes I make would have to be after hours and involve some downtime.

The nat inside is just something I added at one point, didn't seem to affect it either way, and I must have left it in there. 

I've also involved a Cisco TAC engineer who said the ASA should be working as it needs to be, and packet tracers confirm that.

[Jon Marshall]

Okay, I was just wondering where the issue actually was.

few things -

1) if I understand what your topology is why do you have "ip nat inside" on the connection back to the other site ?

2) from a 192.168.2.x client can you ping 192.168.6.254. Obviously not now as you have reverted back.

3) I don't think you need the  "ip default-gateway 192.168.6.1" command.

Jon

[ewellsie07]

1. Added it at some point to mirror the LAN side, just never removed it.

2. No, the 192.168.2.0/24 hosts cannot ping the 192.168.6.254 interface when it's configured as posted. 

3. Agreed, just another redundancy I need to remove, since I'm already using the default route statement.

[Jon Marshall · Accepted solution]

Okay I think the issue is your ASA and the stateful flow.

I know you are testing with ping but lets use a TCP connection as an example -

The packet flow from a 192.168.6.x client to a 192.168.2.x client is -

192.168.6.x -> ASA -> router -> 192.168.2.x  and return traffic is -

192.168.2.x -> router -> direct to 192.168.6.x client because the router interface is in the same subnet which means your default route is not used.

This works because the initial packet TCP SYN goes via the ASA. The fact that the ASA doesn't see the return packet doesn't matter.

Now the other way -

192.168.2.x client -> router -> direct to 192.168.6.x client but return traffic is -

192.168.6.x client -> ASA -> router -> 192.168.2.x client

this won't work because the ASA sees a TCP SYN-ACK packet but it has no record  of ever seeing the initial SYN packet so it won't pass the traffic.

What you need is to use a separate subnet for the link between the ASA and the router so that all traffic both ways has to follow the same path ie. go via the ASA.

Edit - this does not explain why there is no internet access though as that should go direct to and from the ASA.

Jon

[ewellsie07]

To clarify, the ASA inside is plugged into a switch that the fiber is connected to, not directly to the router.

If I made the link between the two of them into a separate subnet, that would require I use an additional interface on the ASA to handle the communication between the two devices for routing.

Edit: What if I just used the 192.168.6.254 interface as the default gateway for the primary LAN, and the 192.168.2.254 interface as the gateway for the secondary LAN. Would this fix the flow issue as you described, which does make sense. 

[Jon Marshall]

Yes you would need an additional interface.

What I wrote does not explain internet access.

It might explain ping if you have ICMP inspection configured on your ASA.

Do you ?

Jon

[ewellsie07]

It is configured in the global policy, yes.

Would it make sense to make the gateway 192.168.6.254 for the 192.168.6.0/24 clients? Would this also fix the flow issue?

Eventually all of the servers and resources will be located at the secondary location. 

[Jon Marshall]

Okay then it does make sense.

See my last post as to what you may be able to do.

Don't know whether TCP state bypass would allow ping to work but you could always disable ICMP inspection.

The only thing unaccounted for is why internet access does not work which is not explained by the issue of traffic flow.

Jon

[ewellsie07]

I suspect the Internet access is more of a DNS issue with the primary DNS server being at the primary location.

Once I get traffic flowing between the two networks, I'm sure I can manage to figure that one out. 

So as it stands now, given what we've discussed, I can plug all the configuration changes back in, and try the following in order:

1. Disable ICMP inspection and test if pings now work across the network.

2. Set the default gateway for the 192.168.6.0/24 clients to 192.168.6.254 to see if this allows all traffic to pass, even though it's less than ideal and does not allow for a loss of physical connectivity between the two sites as I'm depending on the router at the secondary location.

3. Set up another ASA interface on another subnet, probably 192.168.5.0/24 to handle the routing between the ASA and the 1921, leaving the 192.168.6.0 and 192.168.2.0 as completely separate networks.

[Jon Marshall]

The DNS at the primary explains the internet.

Yes, first disable ICMP inspection and try ping. It should work.

No TCP will work until you do something else.

Personally if it was me I would readdress and use another interface assuming you have one.

Not only does this mean the primary site only goes to the secondary site for the secondary site subnets but as you point out if you change the gateway the router is now a single point of failure for both sites which is not a good design.

And also worth bearing in mind the link between sites would have a lot of extra internet traffic on there that doesn't really need to be there.

I also wouldn't use TCP state bypass. It is designed to work around asymmetric routing which is what you have but your routing could be made to be symmetric quite easily whereas in some situations it can't.

Jon

[ewellsie07]

Yea, I like the additional subnet idea, just hadn't thought of that, but it makes sense.

Am I going to have any NAT issues translating incoming traffic from outside to the 192.168.2.0/24 subnet if I put it behind the ASA/1921 subnet, say 192.168.5.0/24?

As long as my routes are good, that should work correct? 

Edit: Yes, I have the interface permit statements configured already, would just need to add the new internal subnet NAT.

[Jon Marshall]

There should be no NAT issues if you use a different IP subnet for the connection between the router and the ASA.

The ASA does not need the networks to be directly connected to perform NAT on the IPs.

As you say you just need to modify the default route on your router to point to the new IP address and the ASA would need a route for 192.168.2.0/24 pointing to the new router IP.

Jon

[ewellsie07]

That's what I thought. 

Thanks is for the help, I will give that a shot, seems like it should be fairly easy, just need to re-plan my config changes.