Problems with Static NAT, of all things...

[jim_r]

Hello. This is making me bonkers but here it goes -- I have 3 ISR 4331 and 2 3650s set up in a line, and I'm trying to use Static NAT on the first router (R1) to reach in to the 3650 (D1) via telnet or SSH. But it doesn't work. Secondary to this issue -- what happened to the debug ip nat translations command??

(host)--(d1)--(r1)--(r2)--(r3)--(d2)--(host) is the topology.

R1's LAN is the 10.67.5.0/24 network. R3's LAN is the 201.55.3.0/24 network.

Inbetween the routers are the 12.0.0.0/29 and 23.0.0.0/29 networks, and before I removed the static route from R2 to 10.67.5.0/24, there was full reachability.

On R1, I configured static nat:

ip nat inside source static 10.67.5.2 12.0.0.3
interface g0/0/1
ip nat inside
int g0/0/0
ip nat outside

From R2, pings to 12.0.0.3 work, but attempts to telnet or ssh to 12.0.0.3 fail (timeout).

There is not debug ip nat translations on R1, so I cannot watch the translation process 

At D1, I can run debug ip packet and see the ping incoming and outgoing, but no traffic comes in during the SSH or Telnet attempts. I can telnet and SSH from the host on R1's LAN to D1. Prior to putting NAT in place, I could also Telnet and SSH to D1 from R2, R3, D2, and the distant host on the 201.55.3.0/24 network.

(I built this same network in Packet Tracer, and it works...copied and pasted the config commands. Even have the debug ip nat translations command)

So... what am I missing? (I removed extraneous stuff from the config)

R1-NAT# show ver | i Soft
Cisco IOS XE Software, Version 16.12.08
Cisco IOS Software [Gibraltar], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.12.8, RELEASE SOFTWARE (fc1)
R1-NAT# sho run brief | ex !
Building configuration...


Current configuration : 3277 bytes
version 16.12
service timestamps debug datetime msec
service timestamps log datetime msec
service call-home
platform qfp utilization monitor load 80
platform punt-keepalive disable-kernel-core
hostname R1-NAT
boot-start-marker
boot-end-marker
vrf definition Mgmt-intf
address-family ipv4
exit-address-family
address-family ipv6
exit-address-family
enable secret 9 @@@@@@@@@@@@
no aaa new-model
XXXXXX
ip domain name XXXXXXX
ip dhcp excluded-address 10.67.5.1 10.67.5.49
ip dhcp excluded-address 10.67.5.52 10.67.5.254
ip dhcp pool USERS
network 10.67.5.0 255.255.255.0
default-router 10.67.5.1
login on-success log
subscriber templating
multilink bundle-name authenticated
@@@ CRYPTO STUFF HERE @@
voice-card 0/1
no watchdog
voice-card 0/4
no watchdog
no license feature hseck9
license udi pid ISR4331/K9 sn @@@@@@@@@
memory free low-watermark processor 67065
diagnostic bootup level minimal
spanning-tree extend system-id
username admin secret 9 @@@@@@@@@
redundancy
mode none
interface GigabitEthernet0/0/0
ip address 12.0.0.1 255.255.255.248
ip nat outside
negotiation auto
interface GigabitEthernet0/0/1
ip address 10.67.5.1 255.255.255.0
ip nat inside
negotiation auto
@@@@@@@@@
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
ip tftp source-interface GigabitEthernet0
ip nat inside source static 10.67.5.2 12.0.0.3
ip route 0.0.0.0 0.0.0.0 12.0.0.2
control-plane
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
mgcp profile default
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
login local
transport input telnet ssh
end

R1-NAT#

R1-NAT# show cdp nei
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID Local Intrfce Holdtme Capability Platform Port ID
R2-NAT.XX Gig 0/0/0     165     R S I      ISR4331/K Gig 0/0/1
D1-NAT.XX Gig 0/0/1     128     S I        WS-C3650- Gig 1/0/1

Total cdp entries displayed : 2
R1-NAT#

[paul driver]

Hello
As a test can you try NVI nat instead. Let me know how you get on

int gig0/0/0
no ip nat outside
ip nat enable

int gig0/0/1
no ip nat inside
ip nat enable

no ip nat inside source static 10.67.5.2 12.0.0.3
ip nat source static 10.67.5.2 12.0.0.3

[jim_r]

Paul,

Thank you for your suggestion. Sadly, it did not work. The indications are the same -- pings work and show the correct addresses in the output of debug ip packet on D1, but an attempt to telnet to 12.0.0.3 times out and shows no packet activity on D1. The log below is from D1 seeing the ping.

D1-NAT#debug ip packet
IP packet debugging is on
D1-NAT#
*Feb 26 12:50:31.121: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Feb 26 12:50:31.122: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, rcvd 1
*Feb 26 12:50:31.122: IP: tableid=0, s=12.0.0.2 (Vlan2), d=10.67.5.2 (Vlan2) nexthop=10.67.5.2, routed via RIB
*Feb 26 12:50:31.123: IP: tableid=0, s=10.67.5.2 (local), d=12.0.0.2 (Vlan2) nexthop=10.67.5.1, routed via FIB
*Feb 26 12:50:31.123: IP: s=10.67.5.2 (local), d=12.0.0.2 (Vlan2), len 100, sending
*Feb 26 12:50:31.125: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Feb 26 12:50:31.126: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, rcvd 1
*Feb 26 12:50:31.126: IP: tableid=0, s=12.0.0.2 (Vlan2), d=10.67.5.2 (Vlan2) nexthop=10.67.5.2, routed via RIB
*Feb 26 12:50:31.126: IP: tableid=0, s=10.67.5.2 (local), d=12.0.0.2 (Vlan2) nexthop=10.67.5.1, routed via FIB
*Feb 26 12:50:31.127: IP: s=10.67.5.2 (local), d=12.0.0.2 (Vlan2), len 100, sending
*Feb 26 12:50:31.129: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Feb 26 12:50:31.129: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, rcvd 1
*Feb 26 12:50:31.129: IP: tableid=0, s=12.0.0.2 (Vlan2), d=10.67.5.2 (Vlan2) nexthop=10.67.5.2, routed via RIB
*Feb 26 12:50:31.130: IP: tableid=0, s=10.67.5.2 (local), d=12.0.0.2 (Vlan2) nexthop=10.67.5.1, routed via FIB
*Feb 26 12:50:31.130: IP: s=10.67.5.2 (local), d=12.0.0.2 (Vlan2), len 100, sending
*Feb 26 12:50:31.132: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Feb 26 12:50:31.133: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, rcvd 1
*Feb 26 12:50:31.133: IP: tableid=0, s=12.0.0.2 (Vlan2), d=10.67.5.2 (Vlan2) nexthop=10.67.5.2, routed via RIB
*Feb 26 12:50:31.133: IP: tableid=0, s=10.67.5.2 (local), d=12.0.0.2 (Vlan2) nexthop=10.67.5.1, routed via FIB
*Feb 26 12:50:31.133: IP: s=10.67.5.2 (local), d=12.0.0.2 (Vlan2), len 100, sending
*Feb 26 12:50:31.135: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Feb 26 12:50:31.136: IP: s=12.0.0.2 (Vlan2), d=10.67.5.2 (nil), len 100, rcvd 1
*Feb 26 12:50:31.136: IP: tableid=0, s=12.0.0.2 (Vlan2), d=10.67.5.2 (Vlan2) nexthop=10.67.5.2, routed via RIB
*Feb 26 12:50:31.137: IP: tableid=0, s=10.67.5.2 (local), d=12.0.0.2 (Vlan2) nexthop=10.67.5.1, routed via FIB
*Feb 26 12:50:31.137: IP: s=10.67.5.2 (local), d=12.0.0.2 (Vlan2), len 100, sending

[Torbjørn]

I think a FIA trace could be a good aid in troubleshooting this: https://www.cisco.com/c/en/us/support/docs/content-networking/adaptive-session-redundancy-asr/117858-technote-asr-00.html 

[jim_r]

Thanks - that's an interesting tool. It appears that R1 has taken ownership of the address I've assigned for static nat... Any revelations from this output? I also noticed that R1 assigned an L route to 12.0.0.3/32, which I never noticed happen before with static nat.

R1-NAT# show platform packet-trace packet 5
Packet: 5 CBUG ID: 6
Summary
Input : GigabitEthernet0/0/0
Output : GigabitEthernet0/0/0
State : DROP 55 (ForUs)
Timestamp
Start : 674480226414 ns (02/26/2025 12:41:45.270305 UTC)
Stop : 674480259221 ns (02/26/2025 12:41:45.270338 UTC)
Path Trace
Feature: IPV4(Input)
Input : GigabitEthernet0/0/0
Output : <unknown>
Source : 12.0.0.2
Destination : 12.0.0.3
Protocol : 6 (TCP)
SrcPort : 64233
DstPort : 23
Feature: DEBUG_COND_INPUT_PKT
Entry : Input - 0x8154e1a0
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 5600 ns
Feature: IPV4_INPUT_DST_LOOKUP_CONSUME
Entry : Input - 0x8154e1c4
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 3520 ns
Feature: IPV4_INPUT_FOR_US_MARTIAN
Entry : Input - 0x8154e1c8
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 7680 ns
Feature: DEBUG_COND_APPLICATION_IN
Entry : Input - 0x8154e1b0
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 3040 ns
Feature: DEBUG_COND_APPLICATION_IN_CLR_TXT
Entry : Input - 0x8154e1b4
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 320 ns
Feature: IPV4_INPUT_VFR
Entry : Input - 0x8154e2f8
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 3680 ns
Feature: NAT
Direction : OUT to IN
Action : FWD
FWD-POINT : LOOKUP_FAIL
VRF : 0
Feature: IPV4_NAT_INPUT_FIA
Entry : Input - 0x8159247c
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 161280 ns
Feature: IPV4_INPUT_SBC
Entry : Input - 0x815993ec
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 21760 ns
Feature: STILE_LEGACY_DROP_EXT
Entry : Input - 0x8159bb90
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 6720 ns
Feature: INGRESS_MMA_LOOKUP_DROP_EXT
Entry : Input - 0x81591a0c
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 5760 ns
Feature: INPUT_DROP_FNF_AOR_EXT
Entry : Input - 0x8157f100
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 17920 ns
Feature: INPUT_FNF_DROP_EXT
Entry : Input - 0x8157e1f4
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 12160 ns
Feature: INPUT_DROP_FNF_AOR_RELEASE_EXT
Entry : Input - 0x8157ea90
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 5440 ns
Feature: INPUT_DROP_EXT
Entry : Input - 0x815530a8
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 1920 ns
Feature: IPV4_INPUT_LOOKUP_PROCESS
Entry : Input - 0x8154e1cc
Input : GigabitEthernet0/0/0
Output : <unknown>
Lapsed time : 170080 ns

R1-NAT#
R1-NAT# show ip route | in L
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
o - ODR, P - periodic downloaded static route, l - LISP
L 10.67.5.1/32 is directly connected, GigabitEthernet0/0/1
L 12.0.0.1/32 is directly connected, GigabitEthernet0/0/0
L 12.0.0.3/32 is directly connected, GigabitEthernet0/0/0
R1-NAT#

[paul driver]

Hello
D2 <> rtr 1 12.0.0.1
rtr1 12.0.0.1 <> D2

Does the above work?

Edited- where are you trying to telnet from - i assume its from outside the nat domain and NOT from inside -correct?

[jim_r]

Hi Paul,

Yes to all. D2 can ping R1 and vice versa. I've been trying to do the telnet from R2, which is sending the telnet from 12.0.0.2... but I have also tried from a host on the "distant" 201.55.3.0/24 network. Both gave me the same indicator, which is to say the connection timed out and debug ip packet on D1 did not show any traffic inbound.

Thanks,

Jim

[paul driver]

Hello
The inside nat domain off R1 gig0/0/1 is a switch correct , is ip routing disabled on that switch?

[jim_r]

Hello Paul,

Yes the switch is in layer 2 mode - ip routing is not enabled.

Thanks,

Jim