Adding PAT within a VRF kills vty SSH/Telnet

[Drew T.]

I'm managing several devices that are using VRF lite. All works fine for VTY access, until I add a PAT statement within the VRF. Sanitised config as follows:

router-aart#sh run
Building configuration...


version 15.2
service nagle
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname router-aart
!
boot-start-marker
boot-end-marker
!
!
vrf definition InVRF
 rd 192.168.0.0:666
 !
 address-family ipv4
 exit-address-family
!
!
logging buffered 51200
enable secret 5 <removed>
!
aaa new-model
!
!
aaa authentication login default local
aaa authentication login vpnuser local
aaa authentication ppp default if-needed
aaa authorization exec default local 
!
!
!
!
!
aaa session-id common
clock timezone 6 0
!
ip cef
!
!
!
ip dhcp pool site-InVRF
 vrf InVRF
 network 192.168.0.0 255.255.255.0
 dns-server 196.168.1.1 8.8.8.8 
 default-router 192.168.0.254 
 lease 7
!
!
!
ip domain name InVRF.com
ip name-server 8.8.8.8
ip name-server 8.8.4.4
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
hw-module pvdm 0/0
!
!
!
!
redundancy
!
!
!
!
!
ip ssh time-out 60
ip ssh authentication-retries 2
ip ssh version 2
! 
!
crypto isakmp policy 1
!
crypto isakmp policy 2
!
crypto isakmp policy 10
crypto isakmp key x address y
crypto isakmp key a address b 
crypto isakmp invalid-spi-recovery
!
!
crypto ipsec transform-set blah <removed> 
 mode tunnel
!
!
!
crypto map crypto-vpn 10 ipsec-isakmp 
 description VPN to blah
 set peer w.x.y.z
 set transform-set blah 
 match address xxx
crypto map crypto-vpn 20 ipsec-isakmp 
 description VPN to feh
 set peer a.b.c.d
 set transform-set blah
 match address yyy
!
!
!
!
!
interface GigabitEthernet0/0
 description LAN
 no ip address
 load-interval 30
 duplex auto
 speed auto
!
interface GigabitEthernet0/0.10
 description InVRF LAN
 encapsulation dot1Q 10
 vrf forwarding InVRF
 ip address 192.168.0.254 255.255.255.0
 no ip redirects
 ip accounting output-packets
 ip nat inside
 no ip virtual-reassembly in
!
interface GigabitEthernet0/0.20
 description Local
 encapsulation dot1Q 20
 ip address 192.168.2.254 255.255.254.0
 no ip redirects
 no ip proxy-arp
 ip accounting output-packets
 ip nat inside
 no ip virtual-reassembly in
!

!
interface GigabitEthernet0/1
desc link for InVRF
 vrf forwarding InVRF
 ip address 1.2.3.1 255.255.255.252
 ip nat outside
 no ip virtual-reassembly in
 ip tcp adjust-mss 1452
 load-interval 30
 duplex auto
 speed auto
 crypto map crypto-vpn
!
!
interface GigabitEthernet0/0/0
 description Other Link
 ip address dhcp
 ip nat outside
 ip virtual-reassembly in
 load-interval 30
 duplex auto
 speed auto
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
no ip nat service sip udp port 5060
ip nat inside source list local-net interface GigabitEthernet0/0/0 overload
ip nat inside source list InVRF-NAT interface GigabitEthernet0/1 vrf InVRF overload
ip nat inside source static tcp 192.168.0.20 554 1.2.3.1 554 vrf InVRF extendable
!
ip route 0.0.0.0 0.0.0.0 dhcp
!

ip access-list extended InVRF-NAT
 permit ip 192.168.0.0 0.0.0.255 any
ip access-list extended local-net
  permit ip 192.168.2.0 0.0.0.255 any
!
access-list 110 permit ip host 4.4.4.4 any
! access-list 110 has all management hosts permitted to access this device
 !
!
!
!
mgcp profile default
!
!
!
!
!
gatekeeper
 shutdown
!
!
banner motd ^C

^C
!
line con 0
 logging synchronous
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport input all
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 access-class 110 in vrf-also
 login authentication local
 transport input telnet ssh
line vty 5 15
 access-class 110 in vrf-also
 login authentication local
 transport input telnet ssh

Everything worked perfectly until I put in 

ip nat inside source static tcp 192.168.0.20 554 1.2.3.1 554 vrf InVRF extendable

This is not the first time i've seen it, but as i'm using VRF Lite with increasing frequency to manage devices with multiple WAN links, i'd like to fix it. This has been on all versions of 15.1 on 29xx and 39xx's. 

Any advice appreciated. 

[Georg Pauwen]

Hello,

can you try a standard access list instead of an extended ? So, instead of:

access-list 110 permit ip host 4.4.4.4 any

use

access-list 1 permit 4.4.4.4 0.0.0.0

[Drew T.]

This makes no difference. The ACL 110 isn't even seeing its hit counter increment when a telnet or ssh session comes in. I tried a regular ACL as well, same problem.

[Georg Pauwen]

Hello,

since this appears to be a lab, post the full configs of both sides including they keys and IP addresses, so we can lab this.

Which RSA key modulus are you using (1024/2048) ?

[Drew T.]

This isn’t a lab so I can’t post the full config. This is a sanitised version (and I cant telnet so ssh won’t make any difference).

I have many sites with a similar problem all with similar configs.

[Georg Pauwen]

Hello,

understood.

1. Try 'transport input all' on all VTY lines instead of 'ssh telnet'. Does it make a difference if you use the 'true' SSH port (443) in your NAT statement ?

[Drew T.]

transport in all does nothing either. 

I'm not attempting to static NAT any port that SSH would use (not 22/23 or 443, but 443 isn't used for ssh; 443 is SSL/HTTPS default port). 

This happens the second I put any PAT on in the VRF on any device. If I remove the PAT/NAT statement, it's fine. 

[Richard Burts]

I agree with my colleague Georg that using a standard access list with access-class is better than using an extended access list (but I doubt that this relates to the problem described in the original post). I believe that you can try changing the transport input to all but I doubt that it will change the symptoms.

When I have done address translation using two outside interfaces I have usually used route maps to control the translations, so that you can match on both the interface and the ACL. But since the translations are for different address ranges it probably is ok to configure the translation as you are doing.

I do wonder about the static address translation you are adding. Is Georg correct that the 554 is part of the sanitization and that the port is really 443? Can you tell us what address you are attempting SSH (or telnet) to access?

HTH

Rick

[Drew T.]

Hi Richard,

ip nat inside source static tcp 192.168.0.20 554 1.2.3.1 554 vrf InVRF extendable

Is the statement that's being added that breaks telnet to the router itself (or SSH for that matter).

This router has 3 WAN links in total, and route-maps are widely used in production (most sites only have 2 WAN links). Unfortunately BGP is not an option here, so i'm having to use vrf lite to segregate traffic (route-maps are messy and unreliable at times i've found).

The port is not sanitised, it's 554. There is no port forwarding for 22, 23 or 443 at any of these sites. The first thing I checked was this, but the fact it kills telnet to the external interface that's referenced in the VRF ruled that out in my head.

I can confirm that if I don't have the vrf-also statment on the vty acl, that it will not permit telnet *before* I apply the NAT statements, so that part is working (applying it pre-NAT statement enables telnet/ssh to the WAN IP in that VRF). It's when I apply the NAT statement that telnet/ssh to the WAN IP from any host on ACL110 stops working.

Unfortunately the IP addresses are real-world so I can't publish them, but you can ping them fine. 

[Georg Pauwen]

Hello,

sorry my bad, with true SSH I meant 22 of course and not 443.

Either way, what if you configure a loopback interface, make that part of the VRF, and then use that for telnet ?

[Drew T.]

Thanks Georg,
Loopback is not possible as it's a /32 point to point.

I can access the devices by the other WAN IP's that aren't in the VRF, but I need accessiblity from all interfaces.

[Richard Burts]

Thanks for the additional information and for clarifying that the port in the translation statement is not sanitized. It is very curious that a static translate for port 554 breaks remote access. It suggests that the static translate is operating on more than the stated port. That suggests that it may be a bug in the software. I am disappointed to see that you have found this behavior in multiple versions of the software. But I continue to suspect that it is a software defect. If you have a support contract for this equipment I would suggest opening a case with Cisco TAC. But to continue the discussion here I would suggest that you enable debug for address translation, attempt SSH (or telnet) and post any debug output.

HTH

Rick

[Drew T.]

Thanks Rick. I can't enable NAT debug on these are they're heavily utilised. I enabled debug for NAT on the VRF but it showed up nothing (even when I tried to use the NAT in the VRF that is working).

I'll collate a list of the IOS versions, but they are all 15.x (mainly 1 and 2 I believe).

[Georg Pauwen]

Hello,

I recreated your setup in GNS3, both with IOSv 15.6 and IOS 15.2, in both cases it works with and without the static NAT. So it might as well be, as Richard suggested, either IOS or platform (or a combination of both) dependent...

[Richard Burts]

What device is at 192.168.0.20? Is there any way to determine whether it is receiving attempts to SSH and/or telnet when you are attempting to access the router but it is not working?

It is interesting that ping is successful when the static nat is in place. I wonder whether the ping response is generated from the router itself or is perhaps generated from the device at 192.168.0.20.

HTH

Rick