In case anyone stumbles across this, the solution ended up being to remove the public IP that the traffic is being NATed to from the interface. The IOS XE platform doesn't support using an IP address assigned to an interface in a NAT pool. Once the IPs on the vasi interfaces were separated from the IP address being used in the NAT pool traffic flowed properly across all protocols.
... View more
UDP is also being NATed properly.
ip-10-241-1-84#traceroute vrf vpn-transit 100.64.1.1 Type escape sequence to abort. Tracing the route to 100.64.1.1 VRF info: (vrf in name/id, vrf out name/id) 1 XX.XX.XX.XX 1 msec 1 msec 1 msec 2 100.64.1.1 1 msec 1 msec 1 msec
When I attempt the same traceroute from the 'vpn-external' VRF, the traffic seems to be going back to the vasileft101 interface. This is confusing since nothing in the routing table for 'vpn-external' should pass the traffic back from vasiright to vasileft.
ip-10-241-1-84#traceroute vrf vpn-external 100.64.1.1 Type escape sequence to abort. Tracing the route to 100.64.1.1 VRF info: (vrf in name/id, vrf out name/id) 1 XX.XX.XX.XY 0 msec 0 msec 1 msec 2 XX.XX.XX.XX 1 msec 1 msec 1 msec 3 100.64.1.1 1 msec 1 msec 1 msec
ip-10-241-1-84#sh ip ro vrf vpn-external
Routing Table: vpn-external Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override, p - overrides from PfR
Gateway of last resort is not set
188.8.131.52/24 is subnetted, 1 subnets S 100.64.1.0 [1/0] via ZZ.ZZ.ZZ.ZZ
Where ZZ.ZZ.ZZ.ZZ is the public IP of our remote IPsec endpoint.
I have a suspicion this routing loop is related to our issues with TCP NAT.
... View more
I have a CSR1000v running in AWS. Using the transit VPC template (https://github.com/awslabs/aws-transit-vpc) we have built an environment with routing between our VPCs passing through the CSRs in a transit VPC configuration. This deployment model heavily leverages VRFs; each VPC has a VRF associated with it.
I am extending this model by bringing up some transport IPsec links from the CSR to third party partners. I am unable to control the configuration of these IPsec links; so I am not able to utilize a VTI model for IPsec. I have created a 'vpn-external' VRF which all of my IPsec crypto-map definitions specify, and reverse-route from the crypto map into the 'vpn-external' VRF. I have created a second 'vpn-transit' VRF. I have linked these two VRFs using the VASI virtual interface. I have placed interface vasileft101 into the 'vpn-transit' VRF, and have placed the vasiright101 interface into the 'vpn-external' VRF. I have acquired a /30 of public IPs, and assigned one to each of the vasi*101 interfaces. I require a public IP on this interface as my VPN partners will not route private IP space. I have setup routes from the VPN endpoints across the vasi link statically, and redistribute these into the BGP space. My VPCs are able to all see these routes and communicate as needed.
As mentioned, I need the traffic passing over the transport IPsec links to come from a public IP address as my partners do not route private IP traffic. I have set a nat inside rule on the vasiright101 interface, and per the standard CSR deployment my GigabitEthernet1 interface has nat outside specified.
This architecture works for ICMP traffic. I am able to ping from anywhere in my AWS network, route it to the proper IPsec link, NAT the traffic, have it cross the transport IPsec link and appear on the other side (and indeed the ICMP responses make it back to the hosts).
I am unable to have a TCP connection successfully establish, however. It appears that the CSR is sending a TCP RESET packet as soon as a SYN is sent out the IPsec link, which effectively prevents TCP from working.
Let me show some examples and some configs. For testing purposes the remote end of the link is a Vyatta router, with the 100.64.1.0/24 subnet being routed (and the Vyatta responding to 100.64.1.1). The XX.XX.XX.XX is the public IP I am NATing all traffic to (assigned to vasiright101), and XX.XX.XX.XY is the other IP in the /30 which is assigned to vasileft101.
Pinging from the VRF with the vasiright101 interface works:
ip-10-241-1-84#ping vrf vpn-external 100.64.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 100.64.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Pinging from the VRF with the vasileft101 interface works:
ip-10-241-1-84#ping vrf vpn-transit 100.64.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 100.64.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
I see a nat translation is created:
ip-10-241-1-84#sh ip nat translations Pro Inside global Inside local Outside local Outside global icmp XX.XX.XX.XX:16 XX.XX.XX.XY:16 100.64.1.1:16 100.64.1.1:16
When I telnet from the vasiright101 interface, it works:
ip-10-241-1-84#telnet 100.64.1.1 22 /source-interface vasiright101 Trying 100.64.1.1, 22 ... Open SSH-2.0-OpenSSH_5.5p1 Debian-6+squeeze3
However, when I telnet from the vasileft101 interface, it does not connect:
ip-10-241-1-84#telnet 100.64.1.1 22 /source-interface vasileft101 Trying 100.64.1.1, 22 ... % Connection timed out; remote host not responding
A nat translation was created:
ip-10-241-1-84#sh ip nat translations Pro Inside global Inside local Outside local Outside global tcp XX.XX.XX.XX:47105 XX.XX.XX.XY:47105 100.64.1.1:22 100.64.1.1:22 Total number of translations: 1
Looking at the tcp debug log:
*Jul 27 01:28:08.573: tcp0: I LISTEN 100.64.1.1:22 XX.XX.XX.XX:47105 seq 3095664780 OPTS 8 ACK 336248651 SYN WIN 17922 *Jul 27 01:28:08.574: TCP: sent RST to 100.64.1.1:22 from XX.XX.XX.XX:47105 *Jul 27 01:28:10.574: tcp0: I LISTEN 100.64.1.1:22 XX.XX.XX.XX:47105 seq 3126920503 OPTS 8 ACK 336248651 SYN WIN 17922 *Jul 27 01:28:10.574: TCP: sent RST to 100.64.1.1:22 from XX.XX.XX.XX:47105 *Jul 27 01:28:14.575: tcp0: I LISTEN 100.64.1.1:22 XX.XX.XX.XX:47105 seq 3189437577 OPTS 8 ACK 336248651 SYN WIN 17922 *Jul 27 01:28:14.575: TCP: sent RST to 100.64.1.1:22 from XX.XX.XX.XX:47105 *Jul 27 01:28:22.574: tcp0: I LISTEN 100.64.1.1:22 XX.XX.XX.XX:47105 seq 3314429385 OPTS 8 ACK 336248651 SYN WIN 17922 *Jul 27 01:28:22.574: TCP: sent RST to 100.64.1.1:22 from XX.XX.XX.XX:47105
My configs should allow the traffic:
ip access-list extended al-allow-external-nat permit ip host XX.XX.XX.XY 100.64.1.0 0.0.0.255 ...<plus others>...
ip nat pool pool-vpn-external-nat XX.XX.XX.XX XX.XX.XX.XX prefix-length 30 ip nat inside source list al-allow-external-nat pool pool-vpn-external-nat vrf vpn-external match-in-vrf overload
I would greatly appreciate any thoughts as to what I am missing. I feel like things should be working, and the ICMP traffic traversing the link with NAT makes me think this configuration is almost correct, so any insights as to things I have missed would be greatly appreciated.
... View more