4451 router with BDI as NAT outside interface is losing TCP replies (but ping works)

[Mark Cantrell]

I'm looking for ideas on where my TCP handshakes might be going in this lab topology:

[Web client] --> [3650 switch & VLAN11] --> [4451 NAT router/dot1q 11 bridge domain/BDI 11] --> [Web server]

The 4451-x is running IOS XE Version 16.06.05 (IP Base).

The web client connects to a VLAN11 access port on the 3650 switch.  The switch is trunked to the router via PortChannel1. The Po1.11 subinterface is assigned to Bridge Domain 11 at the router, as is Gig0/1/0 (a NIM-2GE-CU-SFP). And the web server is connected to Gig0/1/0. (My apologies for not attaching a nice diagram. I'm swamped, and this problem is not helping.)

Pings & tracert from web client to server look good:

C:\Documents and Settings\Mark.OBS>ping 128.244.210.43

Pinging 128.244.210.43 with 32 bytes of data:

Reply from 128.244.210.43: bytes=32 time=4ms TTL=127
Reply from 128.244.210.43: bytes=32 time<1ms TTL=127
Reply from 128.244.210.43: bytes=32 time<1ms TTL=127
Reply from 128.244.210.43: bytes=32 time=1ms TTL=127

Ping statistics for 128.244.210.43:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 4ms, Average = 1ms

C:\Documents and Settings\Mark.OBS>tracert 128.244.210.43

Tracing route to 128.244.210.43 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.120.2
2 <1 ms <1 ms <1 ms 128.244.210.43

Trace complete.

C:\Documents and Settings\Mark.OBS>

Wireshark sessions on client and server show TCP SYN packet reaching server, and server sending SYN/ACK back towards client (via BDI 11's MAC and NAT global IP).  Embedded packet capture on router also shows router receiving SYN/ACK from server at BDI address, but the router immediately (same timestamp) responds with a TCP RST.  Server SYN/ACK replies never make it back to the client.  UDP from client to server also works.

Here is a dump from the NAT translation table on the router, showing both the ping and TCP translations, as expected:

router01#
router01#sh ip nat trans
Pro Inside global Inside local Outside local Outside global
--- 10.0.0.35 172.18.0.17 --- ---
tcp 10.0.0.32:50001 192.168.120.11:50001 --- ---
udp 10.0.0.32:1581 192.168.120.11:1581 --- ---
tcp 10.0.0.32:1581 192.168.120.11:1581 --- ---
tcp 10.0.0.32:80 192.168.120.4:80 --- ---
udp 10.0.0.32:53 192.168.120.4:53 --- ---
tcp 128.244.210.252:515 192.168.120.12:8 128.244.210.43 128.244.210.43
tcp 128.244.210.252:1385 192.168.120.12:1385 128.244.210.43:80 128.244.210.43:80
tcp 128.244.210.252:1387 192.168.120.12:1387 128.244.210.43:80 128.244.210.43:80
icmp 128.244.210.252:1280 192.168.120.12:1280 128.244.210.43:1280 128.244.210.43:1280
tcp 128.244.210.252:1386 192.168.120.12:1386 128.244.210.43:80 128.244.210.43:80
Total number of translations: 11

router01#

Relevant router configuration extracts are attached, along with a PCAP file showing successful ICMP and failed TCP as captured at the physical router interface by EPC (i.e. where server packets ingress on the way to the NAT outside BDI).

The zone-based firewall is not logging any denies, and I have opened up the relevant ACLs to minimize that risk.

Does anyone have a guess at what might be going wrong here?

Thanks,

  Mark

[Georg Pauwen]

Hello,

where is your zone security ZONE_INTERNAL ? It is hard to tell what you are missing without seeing the full config, better post that...

[Mark Cantrell]

Georg:

Thanks for taking a look.  ZONE_INTERNAL is VLAN3, where the web client resides.  A separate route map governs NAT for VLAN3 traffic to all other WAN destinations (i.e. another NAT outside interface). 

The rest of the config is sensitive, and relatively lengthy.  So I'm not at liberty to post the whole thing, and it would take a long time to sanitize it enough to make it releasable.  In any case, the rest of the router's NAT and ZBFW configuration, including bridge domain 11, has been in use for some time without issues.  The only thing new is that we assigned a new IP address to the existing BDI 11, added a second NAT route map, added a new firewall zone (BDI11), added zone pairs between BDI11 & INT, and added ACLs for the route map and zone pairs. 

NAT support was only recently added to IOS-XE for BDIs.  So it is tempting to conclude that this is a bug, especially given the fact that pings work and the firewall isn't logging drops.  But I have learned not to be too quick to assume that IOS is at fault for seemingly strange behavior.

  Mark

[Mark Cantrell]

I now have a workaround for this issue.  I simply replaced the NAT pool in the "ip nat inside" command with the outside interface name.  Specifically:

OLD non-working commands:

ip nat pool BDI11_NAT_POOL 128.244.210.252 128.244.210.252 netmask 255.255.255.0
ip nat inside source list BDI11_NAT_LIST pool BDI11_NAT_POOL overload

NEW working command:

ip nat inside source list BDI11_NAT_LIST interface BD I11 overload

That is cleaner anyway, since the BDI 11 interface address was the only entry in the old pool.

Apparently, NAT was working for all outbound traffic (ICMP, UDP, and TCP SYN from internal client).  But for two-way TCP traffic, when the outside server's SYN/ACK reply reached the router outside interface (BDI 11) the router failed to recognize that reply as part of an active NAT session originated by an inside client.  Since the router itself didn't initiate the three-way handshake, it sent a RST in response to the server's SYN/ACK.

Thanks again to Georg, and to anyone else who took the time to look this over and chew on it.

  Mark