PAT to the internet does not work

[Dimitri_Toronto]

Good day,

   I seem to have an issue with the routing/NAT. In my virtual test lab I have traffic that is not able to go out to the internet.

From the sw06 it goes to a Palo Alto in vWire mode to the edge router R10. This node has a dhcp interface towards the internet and I did add a default route to dhcp.

I configured a subinterface on the vlan192 allowing all tagging to be allow. I added static routes pointing towards the SW06 for the internal subnets

From sw06 i can source ping the router gateway IP (192.168.10.1) and to the dhcp addresses (192.168.140.144) but not the 192.168.140.1 on the other side which is a meraki switch. The output of debugging NAT and IP packet on R10 shows up as route failed.

Switch#sh ip access-lists 7
Standard IP access list 7
10 permit 192.168.10.0, wildcard bits 0.0.0.255 log (29 matches)
20 permit 172.16.0.0, wildcard bits 0.0.0.255 log
30 permit 172.16.10.0, wildcard bits 0.0.0.255 log (25 matches)
40 permit 172.16.32.0, wildcard bits 0.0.0.255 log

hostname R10

!
interface Ethernet0/0
no switchport
ip address dhcp
ip nat outside
!
interface Ethernet0/1.192
encapsulation dot1Q 192
ip address 192.168.10.1 255.255.255.0
ip nat inside

ip nat inside source list 7 interface Ethernet0/0 overload
ip forward-protocol nd
!
ip http server
ip http secure-server
!
ip route 172.16.0.0 255.255.255.0 192.168.10.252
ip route 172.16.10.0 255.255.255.0 192.168.10.252
ip route 172.16.32.0 255.255.255.0 192.168.10.252
!
access-list 7 permit 192.168.10.0 0.0.0.255 log
access-list 7 permit 172.16.0.0 0.0.0.255 log
access-list 7 permit 172.16.10.0 0.0.0.255 log
access-list 7 permit 172.16.32.0 0.0.0.255 log
!

ip forward-protocol nd
!
ip http server
!
ip route 0.0.0.0 0.0.0.0 192.168.10.1

What is wrong

[Georg Pauwen]

Hello,

the 'log' keyword at the end of your access list effectively kills your NAT. Remove that keyword:

no access-list 7 permit 192.168.10.0 0.0.0.255 log
no access-list 7 permit 172.16.0.0 0.0.0.255 log
no access-list 7 permit 172.16.10.0 0.0.0.255 log
no access-list 7 permit 172.16.32.0 0.0.0.255 log

access-list 7 permit 192.168.10.0 0.0.0.255
access-list 7 permit 172.16.0.0 0.0.0.255
access-list 7 permit 172.16.10.0 0.0.0.255

access-list 7 permit 172.16.32.0 0.0.0.255

Here is the relevant Q&A:

Q. Does Cisco IOS NAT support ACLs with a "log" keyword?
A. When you configure Cisco IOS NAT for dynamic NAT translation, an ACL is used to identify packets that can be translated. The current NAT architecture does not support ACLs with a "log" keyword.

https://www.cisco.com/c/en/us/support/docs/ip/network-address-translation-nat/26704-nat-faq-00.html

[paul driver]

Hello

Looking at the above OP I would say amend the nat access-list to NOT include the log keyword and also change the default static route on R10

no  ip route 0.0.0.0 0.0.0.0 192.168.10.1

ip route 0.0.0.0 0.0.0.0 eth0/0 dhcp

However your topology diagram shows a different story especially the physically setup as I would say here the meraki MX is providing all L3 and NAT/Access-list services, with the Meraki MS providing the L2 for the network?

Could you please elaborate if applicable.

[balaji.bandi]

ip route 0.0.0.0 0.0.0.0 192.168.10.1   ( i believe this should be as below )

ip route 0.0.0.0 0.0.0.0 Ethernet0/0 dhcp (@paul - thank you heads up  - typo issue here)

remove  in ACL logs and test

[paul driver]

Hello

---

@balaji.bandi wrote:

ip route 0.0.0.0 0.0.0.0 Ethernet0/0

---

Specifying default route to a physical interface of a rtr would incur unnecessary arp on each and every egress request- Wouldn't you agree?

[Dimitri_Toronto]

Hi,

   @paul driver You are correct, there is a L2 Meraki switch and L3 Meraki firewall before it goes out to the internet. I did add routing to the firewall for the 172.16.x.x subnets on the meraki firewall and R10.

  @Georg Pauwen @balaji.bandi  I have tried to add and remove the default routing on both the r6(switch) and r10(router). The logging on R10 looks better as there is no more error messages of "routing failed"

I can ping the dhcp IP address on R10 (.144) but nothing past it. Pings to .140.1 fail as well as to any other Host I have on that subnet.

R6>

ip route 0.0.0.0 0.0.0.0 192.168.10.1

ip route 0.0.0.0 0.0.0.0 vlan192 192.168.10.1

ip route 0.0.0.0 0.0.0.0 vlan192

r10>

ip route 0.0.0.0 0.0.0.0 dhcp  <<< current default route + default gateway and network configured as well

ip route 0.0.0.0 0.0.0.0 ethernet0/0 dhcp

ip route 0.0.0.0 0.0.0.0 192.168.140.1

ip route 0.0.0.0 0.0.0.0 ethernet0/0

error message is:

*Dec 28 18:14:28.039: IP: s=172.16.10.100 (Ethernet0/1.192), d=8.8.8.8, len 67, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 28 18:14:28.039: IP: tableid=0, s=172.16.10.100 (Ethernet0/1.192), d=8.8.8.8 (Ethernet0/0), routed via RIB
*Dec 28 18:14:28.040: NAT: UDP s=59043->4503, d=53
*Dec 28 18:14:28.040: NAT: s=172.16.10.100->192.168.140.144, d=8.8.8.8 [13770]
*Dec 28 18:14:28.040: IP: s=192.168.140.144 (Ethernet0/1.192), d=8.8.8.8 (Ethernet0/0), len 67, output feature, Post-routing NAT Outside(26), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwd
chk FALSE
Switch(config)#
*Dec 28 18:14:28.040: IP: s=192.168.140.144 (Ethernet0/1.192), d=8.8.8.8 (Ethernet0/0), g=8.8.8.8, len 67, forward
*Dec 28 18:14:28.040: IP: s=192.168.140.144 (Ethernet0/1.192), d=8.8.8.8 (Ethernet0/0), len 67, encapsulation failed
Switch(config)#
*Dec 28 18:14:33.045: IP: s=172.16.10.100 (Ethernet0/1.192), d=1.1.1.1, len 67, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 28 18:14:33.045: IP: tableid=0, s=172.16.10.100 (Ethernet0/1.192), d=1.1.1.1 (Ethernet0/0), routed via RIB
*Dec 28 18:14:33.045: NAT: UDP s=46634->4502, d=53
*Dec 28 18:14:33.045: NAT: s=172.16.10.100->192.168.140.144, d=1.1.1.1 [20283]
*Dec 28 18:14:33.045: IP: s=192.168.140.144 (Ethernet0/1.192), d=1.1.1.1 (Ethernet0/0), len 67, output feature, Post-routing NAT Outside(26), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwd
chk FALSE
*Dec 28 18:14:33.045: IP: s=192.168.140.144 (Ethernet0/1.192), d=1.1.1.1 (Ethernet0/0), g=1.1.1.1, len 67, forward
*Dec 28 18:14:33.045: IP: s=192.168.140.144 (Ethernet0/1.192), d=1.1.1.1 (Ethernet0/0), len 67, encapsulation failed
*Dec 28 18:14:33.066: NAT: expiring 192.168.140.144 (172.16.10.100) udp 4504 (40874)
Switch(config)#
*Dec 28 18:14:38.050: IP: s=172.16.10.100 (Ethernet0/1.192), d=8.8.8.8, len 67, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 28 18:14:38.050: IP: tableid=0, s=172.16.10.100 (Ethernet0/1.192), d=8.8.8.8 (Ethernet0/0), routed via RIB
*Dec 28 18:14:38.050: NAT: UDP s=59043->4503, d=53
*Dec 28 18:14:38.050: NAT: s=172.16.10.100->192.168.140.144, d=8.8.8.8 [14550]
*Dec 28 18:14:38.050: IP: s=192.168.140.144 (Ethernet0/1.192), d=8.8.8.8 (Ethernet0/0), len 67, output feature, Post-routing NAT Outside(26), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwd
chk FALSE
Switch(config)#
*Dec 28 18:14:38.050: IP: s=192.168.140.144 (Ethernet0/1.192), d=8.8.8.8 (Ethernet0/0), g=8.8.8.8, len 67, forward
*Dec 28 18:14:38.050: IP: s=192.168.140.144 (Ethernet0/1.192), d=8.8.8.8 (Ethernet0/0), len 67, encapsulation failed
*Dec 28 18:14:38.190: NAT: expiring 192.168.140.144 (172.16.10.100) udp 4505 (51672)
Switch(config)#

[paul driver]

Hello

Still dont have a complete understanding of your network frommthe addressing point of view but From r10 remove the following:

ip route 0.0.0.0 0.0.0.0 ethernet0/0

ip route 0.0.0.0 0.0.0.0 192.168.140.1

Add ONLY specific route towards 192.168.140.1 if you need to reach certain subnets via that next-hop

[Dimitri_Toronto]

Hi @paul driver 

 I have tried both ways with the default route going to the next hop (.140.1), ethernet0/0, and dhcp which is supposed to inject a default route

[paul driver]

Hello

You network topology isnt clear enough - the Meraki MX = ASA, Merkai MS =R14, 3750 =R13, then you have two Palto Alto fws interconnecting R6 and R10?

Now where does it show the addressing interconnecting all these devices

[Dimitri_Toronto]

Hi @paul driver,

  there is a bug on eve-ng that even if you rename the nodes appropriately they do not often reflect those changes, that is why I had to insert all the extra texts. The LAB example is the main one with 2 palo alto firewall with fw02 is just sitting idling on vlan32,

• fw01 is in vWire mode that allows traffic to pass in from R6 to R10. That is vlan192 192.168.10.0/24 or the outbound interface for the simple HQ network.
• Vlan 10 is 172.16.10.0/24 which is for management traffic,
• vlan32 is 172.16.32.0/24 is reserved for the server LAN
• on the node r6 is a layer2 switch which terminate all the VLANs (router on a stick)
• the icon is a CLOUD0 interface that provides a break out from the virtual lab to connect to the real world
• all red numbers you see near the interface is the IP of the address within the VLAN subnet.

I've updated the network diagram with more details. Keep in mind the virtual lab is hosted on the EVE-NG

[Richard Burts]

I am looking at this line of output

*Dec 28 18:14:38.050: IP: s=192.168.140.144 (Ethernet0/1.192), d=8.8.8.8 (Ethernet0/0), len 67, encapsulation failed

which reveals at least 2 issues

1) encapsulation failed most likely indicates that the router arped for a next hop and did not receive a response. Why would the router be sending an arp request? Several versions of a default route that have been in the discussion would cause an arp request and several would avoid an arp request. What is the current default route (and is there more than one default route in the router config)?

2) the source address is 192.168.140.144. This address does not match the version of the access list given in a previous post.

You should do something to fix these issues. If you are not sure how to fix them then provide more details of the configuration.

[Dimitri_Toronto]

hi,

 I originally started with the default route to dhcp (ip route 0.0.0.0 0.0.0.0 dhcp) and I had played around with it a few times to see if it will work. currently on my R10 router i have it set to 'ip route 0.0.0.0 0.0.0.0 ethernet0/0 dhcp. I believe this is the proper way of not generating an ARP request. configurations of both devices are listed at the bottom.

PING TEST #1 from R6 ping the DHCP address assigned to ethernet0/0 on R10

R6#ping ip 192.168.140.144 source 172.16.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.140.144, timeout is 2 seconds:
Packet sent with a source address of 172.16.0.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/3 ms
R6#

debug output on R10:

R10#
*Dec 29 10:34:07.328: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, NAT(2), rtype 0, forus F
ALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 29 10:34:07.328: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, Auth Proxy(16), rtype 0,
forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 29 10:34:07.330: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, NAT(2), rtype 0, forus F
ALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 29 10:34:07.330: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, Auth Proxy(16), rtype 0,
forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 29 10:34:07.331: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, NAT(2), rtype 0, forus F
ALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 29 10:34:07.332: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, Auth Proxy(16), rtype 0,
forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 29 10:34:07.333: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, NAT(2), rtype 0, forus F
ALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 29 10:34:07.333: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, Auth Proxy(16), rtype 0,
forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
R10#
*Dec 29 10:34:07.335: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, NAT(2), rtype 0, forus F
ALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 29 10:34:07.335: IP: s=192.168.140.144 (local), d=172.16.0.1, len 100, local feature, Auth Proxy(16), rtype 0,
forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
R10#
a - application route

PING TEST #2 ping the remote gateway of 192.168.140.1 to breakout of the lab into the live internet

R6#
R6#ping ip 192.168.140.1 source 172.16.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.140.1, timeout is 2 seconds:
Packet sent with a source address of 172.16.0.1
.....
Success rate is 0 percent (0/5)
R6#

R10 had 0 output on the debug and nat translation table.

R10#sh ip int brie
Interface IP-Address OK? Method Status Protocol
Ethernet0/0 192.168.140.144 YES DHCP up up
Ethernet0/1 unassigned YES unset up up
Ethernet0/1.192 192.168.10.1 YES NVRAM up up
Ethernet0/2 unassigned YES unset down down
Ethernet0/3 unassigned YES unset down down
R10#
R10#
R10#sh run
Building configuration...

Current configuration : 1467 bytes
!
! Last configuration change at 05:33:10 EST Tue Dec 29 2020
!
version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
service compress-config
!
hostname R10
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
clock timezone EST -5 0
!
ip cef
no ipv6 cef
!
spanning-tree mode pvst
spanning-tree extend system-id
!
interface Ethernet0/0
no switchport
ip address dhcp
ip nat outside
!
interface Ethernet0/1
no switchport
no ip address
duplex full
!
interface Ethernet0/1.192
encapsulation dot1Q 192
ip address 192.168.10.1 255.255.255.0
ip nat inside
!
interface Ethernet0/2
!
interface Ethernet0/3
!
ip nat inside source list 7 interface Ethernet0/0 overload
ip forward-protocol nd
!

ip http server
ip http secure-server
!
ip route 172.16.0.0 255.255.255.0 192.168.10.252
ip route 172.16.10.0 255.255.255.0 192.168.10.252
ip route 172.16.32.0 255.255.255.0 192.168.10.252
ip route 0.0.0.0 0.0.0.0 Ethernet0/0 dhcp
ip ssh server algorithm encryption aes128-ctr aes192-ctr aes256-ctr
ip ssh client algorithm encryption aes128-ctr aes192-ctr aes256-ctr
!
!
control-plane
!
!
line con 0
logging synchronous
line aux 0
line vty 0 4
login
!
!
!
end

R10#

-------------------------------------------------------------------------------------------------

R6#sh ip int brie
Interface IP-Address OK? Method Status Protocol
Ethernet0/0 unassigned YES unset up up
Ethernet0/1 unassigned YES unset up up
Ethernet0/2 unassigned YES unset up up
Ethernet0/3 unassigned YES unset up up
Ethernet1/0 unassigned YES unset up up
Ethernet1/1 unassigned YES unset down down
Ethernet1/2 unassigned YES unset down down
Ethernet1/3 unassigned YES unset down down
Loopback0 172.16.0.1 YES NVRAM up up
Vlan1 unassigned YES unset administratively down down
Vlan10 172.16.10.1 YES NVRAM up up
Vlan32 172.16.32.1 YES NVRAM up up
Vlan192 192.168.10.252 YES NVRAM up up
R6#sh run
Building configuration...

Current configuration : 1774 bytes
!
! Last configuration change at 05:33:38 EST Tue Dec 29 2020
!
version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
service compress-config
!

hostname R6
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
clock timezone EST -5 0
!
no ip domain-lookup
ip cef
no ipv6 cef
!
spanning-tree mode pvst
spanning-tree extend system-id

!
interface Loopback0
ip address 172.16.0.1 255.255.255.0
!
interface Ethernet0/0
description ** TO FW VLAN 192 **
switchport trunk encapsulation dot1q
switchport mode trunk
spanning-tree portfast network
!
interface Ethernet0/1
switchport access vlan 10
switchport mode access
spanning-tree portfast edge
!
interface Ethernet0/2
switchport access vlan 32
switchport mode access
spanning-tree portfast edge
!
interface Ethernet0/3
switchport access vlan 32
switchport mode access
spanning-tree portfast edge
!
interface Ethernet1/0
switchport access vlan 10
switchport mode access
spanning-tree portfast edge
!
interface Ethernet1/1
!
interface Ethernet1/2
!
interface Ethernet1/3
!
interface Vlan1
no ip address
shutdown
!

interface Vlan10
description MANAGEMENT-VLAN
ip address 172.16.10.1 255.255.255.0
!
interface Vlan32
description SERVER VLAN
ip address 172.16.32.1 255.255.255.0
!
interface Vlan192
description ** UPLINK TO R10 **
ip address 192.168.10.252 255.255.255.0
!
ip forward-protocol nd
!
ip http server
!
ip route 0.0.0.0 0.0.0.0 192.168.10.1
ip ssh server algorithm encryption aes128-ctr aes192-ctr aes256-ctr
ip ssh client algorithm encryption aes128-ctr aes192-ctr aes256-ctr
!
!
control-plane
!
!
line con 0
logging synchronous
line aux 0
line vty 0 4
login
!
!
!
end

R6#

[Georg Pauwen]

Hello,

this thread is getting really long and, I think, overcomplicated. The main reason being that we still don't fully know what you have configured. Update the topology drawing to at least show all the full IP address used to interconnect the devices.

[Dimitri_Toronto]

Hello Guys,

   I cannot thank you all enough for sticking with me on this issue. If I was miscommunicating with any of you please feel free to drop some feedback on improving my communication to the community. I have uploaded another diagram with more details on it, keep in mind that the routing lab is hosted on the Ubuntu-EVE-NG

[Richard Burts]

Thanks for posting the router configs. The main thing that I notice is that R10 references access list 7 for nat but I do not see any configuration of access list 7.