NVI NAT overload

[0kami]

Hi!

I would like to ask members of this community for a help. Currently, I am making small office lab with two 3560's, 2811 and laptop that is running vWLC on top of esxi. 

My reserverd port from office switch is configured as trunk and I have first 3560 connected to it.

Here is the configuration:

SW-AP-1#sh run
Building configuration...

Current configuration : 2143 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname SW-AP-1
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
system mtu routing 1500
ip subnet-zero
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
interface FastEthernet0/1
!
interface FastEthernet0/2
!
interface FastEthernet0/3
!
interface FastEthernet0/4
!
interface FastEthernet0/5
!
interface FastEthernet0/6
description R1
switchport access vlan 920
switchport mode access
mls qos trust dscp
!
interface FastEthernet0/7
description PC
switchport access vlan 920
switchport mode access
mls qos trust dscp
spanning-tree portfast
!
interface FastEthernet0/8
description PHONE
switchport access vlan 908
switchport mode access
mls qos trust dscp
spanning-tree portfast
!
interface GigabitEthernet0/1
description UPLINK
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 908,920
switchport mode trunk
ip arp inspection trust
ip dhcp snooping trust
!
interface Vlan1
no ip address
!

All connections here are working fine.

Next, 2811 is connected to this first switch and acts as default gateway and ROAS for second switch.

2811 configuration:

R1#sh run
Building configuration...

Current configuration : 2392 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname R1
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
!
resource policy
!
ip subnet-zero
!
!
ip cef
no ip dhcp use vrf connected
ip dhcp excluded-address 192.168.1.1 192.168.1.5
ip dhcp excluded-address 192.168.10.1 192.168.10.5
ip dhcp excluded-address 192.168.20.1 192.168.20.5
ip dhcp excluded-address 192.168.30.1 192.168.30.5
!
ip dhcp pool vlan1
network 192.168.1.0 255.255.255.0
default-router 192.168.1.1
!
ip dhcp pool vlan10
network 192.168.10.0 255.255.255.0
default-router 192.168.10.1
option 43 hex c0a8.0a05
!
ip dhcp pool vlan20
network 192.168.20.0 255.255.255.0
default-router 192.168.20.1
!
ip dhcp pool vlan30
network 192.168.30.0 255.255.255.0
default-router 192.168.30.1
!
!
ip name-server 193.198.184.130
!
!
!
!
interface FastEthernet0/0
description ### WAN ###
ip address 161.53.12.219 255.255.254.0
ip nat enable
duplex auto
speed auto
!
interface FastEthernet0/1
no ip address
duplex auto
speed auto
!
interface FastEthernet0/1.1
encapsulation dot1Q 1 native
ip address 192.168.1.1 255.255.255.0
ip nat enable
no snmp trap link-status
!
interface FastEthernet0/1.10
encapsulation dot1Q 10
ip address 192.168.10.1 255.255.255.0
ip nat enable
no snmp trap link-status
!
interface FastEthernet0/1.20
encapsulation dot1Q 20
ip address 192.168.20.1 255.255.255.0
ip nat enable
no snmp trap link-status
!
interface FastEthernet0/1.30
encapsulation dot1Q 30
ip address 192.168.30.1 255.255.255.0
ip nat enable
no snmp trap link-status
!
interface Serial0/0/0
no ip address
shutdown
clock rate 2000000
!
ip classless
!
no ip http server
ip nat source list client-list interface FastEthernet0/0 overload
ip nat source static tcp 192.168.1.5 443 161.53.12.219 100 extendable
ip nat source static tcp 192.168.10.5 443 161.53.12.219 443 extendable
ip nat source static tcp 192.168.1.5 902 161.53.12.219 902 extendable
ip nat source static tcp 192.168.20.5 443 161.53.12.219 2000 extendable
!
ip access-list standard client-list
permit 192.168.1.0 0.0.0.255
permit 192.168.10.0 0.0.0.255
permit 192.168.20.0 0.0.0.255
permit 192.168.30.0 0.0.0.255
!

What I tried here is to use NVI with both static PAT and NAT overload configured at same time. Static PAT was needed so I can access hypervisor, vWLC management port and vWLC service port which are all in different VLAN's. 

SW-AP-2#sh run
Building configuration...

Current configuration : 1774 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname SW-AP-2
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
system mtu routing 1500
ip subnet-zero
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
interface FastEthernet0/1
!
interface FastEthernet0/2
!
interface FastEthernet0/3
!
interface FastEthernet0/4
!
interface FastEthernet0/5
description AP
switchport access vlan 30
switchport trunk encapsulation dot1q
switchport mode access
mls qos trust dscp
spanning-tree portfast
!
interface FastEthernet0/6
description WLC
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,10,20,30
switchport mode trunk
switchport nonegotiate
ip arp inspection trust
mls qos trust dscp
spanning-tree portfast trunk
ip dhcp snooping trust
!
interface FastEthernet0/7
description AP
switchport access vlan 10
switchport trunk encapsulation dot1q
switchport mode access
mls qos trust dscp
spanning-tree portfast
!
interface FastEthernet0/8
description AP
switchport access vlan 10
switchport trunk encapsulation dot1q
switchport mode access
mls qos trust dscp
spanning-tree portfast
!
interface GigabitEthernet0/1
description 2router
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,10,20,30
switchport mode trunk
ip arp inspection trust
ip dhcp snooping trust
!
interface Vlan1
no ip address
shutdown
!
ip classless
ip http server
!

While static PAT works beautifully, I can't get NAT overload to work. With PC or any other client, I can ping Fa0/0 address (which is in VLAN 920) on router and that is all. Can anyone show me what I am missing here?

[Peter Paluch]

Hi Antonio,

I have quickly recreated the main parts of your router's configuration on a virtual setup running a 15.5(2)XB IOS, but for me, it worked. I was unable to reproduce the problem you describe. As far as your NAT configuration is concerned, it looks okay to me.

Quickly checking your configuration, however, I do not see a default route configured on your R1. Normally, it would be included immediately beneath the "ip nat source" commands, but I do not see it there. Are you truly missing the default route from R1, or did you just remove that command when pasting the configuration here?

Best regards,
Peter

[0kami]

Hi Peter,

Thank you for quick answer. 

Yes, I removed portions of configs when I pasted them here. But, you did point me in right direction, maybe I did messed up routes. I will look that up. I will try to recreate similar config in my home lab and report back.

Thank you

[0kami]

Peter, I configured my home lab (minus first switch since I don't need it) but result is the same. Clients can't open any websites but they can ping my home router. I don't think this is route problem.

[Peter Paluch]

Antonio,

I've tested it again in Dynamips for you, both with 12.4(15)T13 and 15.5(3)XB, but I still get it working on the first attempt.

Let's try comparing the configurations!

My testing topology consists of three routers connected in a row.

This is the configuration of the client router:

hostname R-Client
!
interface FastEthernet0/0
 description => Toward NAT box <=
 no shutdown
!
interface FastEthernet0/0.1
 encapsulation dot1Q 1 native
 ip address 192.168.1.2 255.255.255.0
 ip address 192.168.1.5 255.255.255.0 secondary
!
interface FastEthernet0/0.10
 encapsulation dot1Q 10
 ip address 192.168.10.2 255.255.255.0
 ip address 192.168.10.5 255.255.255.0 secondary
!
interface FastEthernet0/0.20
 encapsulation dot1Q 20
 ip address 192.168.20.2 255.255.255.0
 ip address 192.168.20.5 255.255.255.0 secondary
!
interface FastEthernet0/0.30
 encapsulation dot1Q 30
 ip address 192.168.30.2 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 192.168.1.1
ip route 0.0.0.0 0.0.0.0 192.168.10.1
ip route 0.0.0.0 0.0.0.0 192.168.20.1
ip route 0.0.0.0 0.0.0.0 192.168.30.1

The four default routes above are just to test the connectivity - I always had only one subinterface up, so only one of the default routes would be used anyway, and I tested the ping to 1.1.1.1 emulated on the Internet router.

This is the NAT box:

hostname R-NAT
!
interface FastEthernet0/0
 description => Toward INTERNET <=
 ip address 161.53.12.219 255.255.254.0
 ip nat enable
 no shutdown
!
interface FastEthernet0/1
 description => Toward CLIENT <=
 no shut
!
interface FastEthernet0/1.1
 encapsulation dot1Q 1 native
 ip address 192.168.1.1 255.255.255.0
 ip nat enable
!
interface FastEthernet0/1.10
 encapsulation dot1Q 10
 ip address 192.168.10.1 255.255.255.0
 ip nat enable
!
interface FastEthernet0/1.20
 encapsulation dot1Q 20
 ip address 192.168.20.1 255.255.255.0
 ip nat enable
!
interface FastEthernet0/1.30
 encapsulation dot1Q 30
 ip address 192.168.30.1 255.255.255.0
 ip nat enable

ip access-list standard client-list
 permit 192.168.1.0 0.0.0.255
 permit 192.168.10.0 0.0.0.255
 permit 192.168.20.0 0.0.0.255
 permit 192.168.30.0 0.0.0.255
!
ip nat source list client-list interface FastEthernet0/0 overload
ip nat source static tcp 192.168.1.5 443 161.53.12.219 100 extendable
ip nat source static tcp 192.168.10.5 443 161.53.12.219 443 extendable
ip nat source static tcp 192.168.1.5 902 161.53.12.219 902 extendable
ip nat source static tcp 192.168.20.5 443 161.53.12.219 2000 extendable
!
ip route 0.0.0.0 0.0.0.0 161.53.12.220

And this is the box emulating internet - the address 1.1.1.1 on its loopback must be reachable from the client:

hostname R-INET
!
interface FastEthernet0/1
 ip address 161.53.12.220 255.255.254.0
 no shutdown
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255

This configuration, when pasted as-is to the routers, works for me right out of the box - from the client, I can ping 1.1.1.1, and I can get appropriate responses when trying to access the static PAT-exported ports.

Can you try this please and let me know if you see different results?

Best regards,
Peter

[0kami · Accepted solution]

I found out what causing issues-my 2811 crashed each time when client tried to load some webpage.

*Mar 5 14:48:19.159: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Mar 5 14:48:19.195: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up
*Mar 5 14:48:19.295: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
*Mar 5 14:48:20.211: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
*Mar 5 14:48:20.211: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Mar 5 14:48:20.391: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to down
*Mar 5 14:48:21.347: %LINEPROTO-5-UPDOWN: Line protocol on Interface NVI0, changed state to up
*Mar 5 14:48:22.147: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up
*Mar 5 14:48:22.147: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
*Mar 5 14:48:23.039: %SYS-5-CONFIG_I: Configured from memory by console
*Mar 5 14:48:23.347: %SYS-5-RESTART: System restarted --
Cisco IOS Software, 2800 Software (C2800NM-IPBASE-M), Version 12.4(3d), RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Wed 19-Apr-06 09:18 by alnguyen
*Mar 5 14:48:23.351: %SNMP-5-COLDSTART: SNMP agent on host R1 is undergoing a cold start
*Mar 5 14:48:23.551: %SYS-6-BOOTTIME: Time taken to reboot after reload = 48 seconds
*Mar 5 14:48:24.943: %LINK-5-CHANGED: Interface Serial0/0/0, changed state to administratively down

14:49:03 UTC Mon Mar 5 2018: Address Error (load or instruction fetch) exception, CPU signal 10, PC = 0x40F0A254

-Traceback= 0x40F0A254 0x40F0A9FC 0x40EDD974 0x40EDE4C8 0x40F0C724 0x400ED3E8 0x400664B0 0x4151D5C0 0x4151AD84 0x40043C40 0x40010428 0x400117C8
$0 : 00000000, AT : 42700000, v0 : 0B0D0B0D, v1 : 0B0D0B0D
a0 : 43A03E90, a1 : 0000F900, a2 : 40F0BE78, a3 : 00000000
t0 : 00000018, t1 : 3400F901, t2 : 3400E100, t3 : FFFF00FF
t4 : 4008EC10, t5 : 43A03EA0, t6 : 43A03E9C, t7 : 43A03E98
s0 : 43A03E90, s1 : 000000C0, s2 : 43817838, s3 : 4337A0BC
s4 : 43A03E90, s5 : 43A03ED0, s6 : 43A03EB8, s7 : 43A03F30
t8 : 432469FC, t9 : 00000000, k0 : 00000000, k1 : 00000000
gp : 4270FB00, sp : 43379E40, s8 : 42860000, ra : 40F0A9FC
EPC : 40F0A254, ErrorEPC : BFC0900C, SREG : 3400F903
MDLO : 00000000, MDHI : 0000003E, BadVaddr : 0B0D0B0D
Cause 00000410 (Code 0x4): Address Error (load or instruction fetch) exception

Writing crashinfo to flash:crashinfo_20180305-144903

14:49:03 UTC Mon Mar 5 2018: Address Error (load or instruction fetch) exception, CPU signal 10, PC = 0x40F0A254

-Traceback= 0x40F0A254 0x40F0A9FC 0x40EDD974 0x40EDE4C8 0x40F0C724 0x400ED3E8 0x400664B0 0x4151D5C0 0x4151AD84 0x40043C40 0x40010428 0x400117C8
$0 : 00000000, AT : 42700000, v0 : 0B0D0B0D, v1 : 0B0D0B0D
a0 : 43A03E90, a1 : 0000F900, a2 : 40F0BE78, a3 : 00000000
t0 : 00000018, t1 : 3400F901, t2 : 3400E100, t3 : FFFF00FF
t4 : 4008EC10, t5 : 43A03EA0, t6 : 43A03E9C, t7 : 43A03E98
s0 : 43A03E90, s1 : 000000C0, s2 : 43817838, s3 : 4337A0BC
s4 : 43A03E90, s5 : 43A03ED0, s6 : 43A03EB8, s7 : 43A03F30
t8 : 432469FC, t9 : 00000000, k0 : 00000000, k1 : 00000000
gp : 4270FB00, sp : 43379E40, s8 : 42860000, ra : 40F0A9FC
EPC : 40F0A254, ErrorEPC : BFC0900C, SREG : 3400F903
MDLO : 00000000, MDHI : 0000003E, BadVaddr : 0B0D0B0D
Cause 00000410 (Code 0x4): Address Error (load or instruction fetch) exception

-Traceback= 0x40F0A254 0x40F0A9FC 0x40EDD974 0x40EDE4C8 0x40F0C724 0x400ED3E8 0x400664B0 0x4151D5C0 0x4151AD84 0x40043C40 0x40010428 0x400117C8

=== Flushing messages (14:49:03 UTC Mon Mar 5 2018) ===

Queued messages:
No warm reboot Storage
*** System received a Bus Error exception ***
signal= 0xa, code= 0x10, context= 0x4285b7d4
PC = 0x40094afc, Cause = 0x20, Status Reg = 0x34008002

System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2006 by cisco Systems, Inc.

Initializing memory for ECC
.
c2811 platform with 262144 Kbytes of main memory
Main memory is configured to 64 bit mode with ECC enabled

Readonly ROMMON initialized
program load complete, entry point: 0x8000f000, size: 0xcb80
program load complete, entry point: 0x8000f000, size: 0xcb80

program load complete, entry point: 0x8000f000, size: 0xe35f8c
Self decompressing the image : ##################################################################################################################################################### [OK]

Quick IOS upgrade solved the issue. I am sorry that I didn't noticed this before.