Traffic drops for AWS GWLB + Cisco ASAv + VPN

[dethmix]

Hi all,

I have the following configuration on AWS Cloud using Gateway load balancer:

GWLB endpoint -> GWLB (geneve protocol) -> inside interface Cisco ASAv (version 9.22) -> outside interface Cisco ASAv -> Internet

NVE 1 configuration:

nve 1
encapsulation geneve
source-interface inside

VNI configuration :

interface vni1
proxy dual-arm
nameif ge
security-level 100
vtep-nve 1

Additional configuration 

mtu inside 1826
!
jumbo-frame reservation
!
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
!
nat (ge,outside) source dynamic any interface

This setup works properly and sends traffic from GWLB endpoint through GWLB via Cisco ASAv to the Internet.

I want to extend this configuration and send some traffic from GWLB to VPN IPSec tunnel configuration on the Cisco ASAv.

GWLB endpoint -> GWLB (geneve protocol) -> inside interface Cisco ASAv (version 9.22) -> VPN IPSec tunnel on outside interface Cisco ASAv

The main goal is to use multiple Cisco ASAv behind the GWLB. Unfortunately, as soon as I send traffic from GWLB to the CIDR that is covered by VPN it is being dropped with the message:

Non-IP packet received in routed mode (non-ip-pkt-in-routed-mode)

I tried to capture these packets via 

capture cap_drop type asp-drop all

and got something like this 

22:48:47.658321       4d84.3b64.52cf 9045.f5f6.314b 0x8b83 Length: 112

			 4f2d 4451 9c29 15dd bf09 52c0 32e8 e3e0
			 db49 d5df 9255 7bae fe55 2707 c258 c6f0
			 47c4 8c23 2d7b 0239 45ee a409 1d82 41c5
			 e64c f7cf 710b eb28 9160 74c1 8f90 d770
			 4897 839e 8ec4 3eb9 caf2 2abb 378a db0b
			 803d 3ca1 26f9 e1d0 de23 74c9 be40 0d97
			 8f98 Drop-reason: (non-ip-pkt-in-routed-mode) Non-IP packet received in routed mode, Drop-location: frame snp_sp_non_ip_flow:306 flow (NA)/NA

This is my test ping packets.

VPN's crypto map is being assigned to outside interface, so limitation "You cannot configure the VTEP source interface for VPN or use it as a VTI." should not be applied.

The same VPN connection and configuration works fine if I send traffic directly to the inside interface without GWLB.

It looks like that due to some reason Cisco ASAv does not decapsulate geneve traffic from GWLB IF its destination is VPN.

Has anybody seen this or is there any kind of limitation in using Cisco ASAv dual-arm mode and VPN?

Thanks

[jonesandrew]

Just curious if you got this resolved as I am considering deploying a similar design and would like to know if there are any issues with this approach. (Daul ASAv with glbp to provide remote access vpn and point to point vpn services) 

[dethmix]

Hello @jonesandrew ,

Nope, I haven't resolved this issue. I still believe that there is some limitation from the ASAv side, but didn't find any confirmation about this. Give it a try and maybe you will be more lucky and find some misconfiguration on my side. At some point I believed that issue was with MTU (I mean I thought that I didn't set it properly), so I even have re-created the whole setup from scratch, but no success. Just confirmed the same behavior. Also I did try to use the dedicated Interface for the geneve traffic instead of the inside interface. Same behavior