Solved: Re: FLEXVPN failing through NAT - TS_UNACCEPTABL

keibler09 · ‎01-18-2022

Hey folks,

I am running into an issue with getting my FLEXVPN working through NAT. I am running two 3925s with switch modules. One is the edge router and the other is where the VPN terminates as the hub. The remote side is a CSRv.

If reconfigure and directly connect the Remote client to the hub the tunnel comes up with no issue. I am running Version 15.7(3)M8 on the 3925s.

Can anyone come up with any ideas why this is not working since FLEXVPN should support NAT-T natively after version 12?

Edge NAT config

interface GigabitEthernet0/0
 ip address x.y.89.94 255.255.255.240
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 ip virtual-reassembly in
 duplex auto
 speed auto
 no cdp enable
 no lldp transmit

!
interface GigabitEthernet1/0.4
 encapsulation dot1Q 4
 ip address 192.168.3.17 255.255.255.248
 ip nat inside
 ip virtual-reassembly in

ip nat inside source static esp 192.168.3.19 interface GigabitEthernet1/0.4
ip nat inside source static udp 192.168.3.19 500 x.y.89.94 500 extendable
ip nat inside source static udp 192.168.3.19 4500 x.y.89.94 4500 extendable

ip route 0.0.0.0 0.0.0.0 x.y.89.81

Debug from hub:

*Jan 18 15:52:31: IKEv2-INTERNAL:Allocated addr 172.16.0.93 from local pool LOCAL_IP_POOL
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_VERIFY_AUTH Event: EV_CHK_GKM
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_VERIFY_AUTH Event: EV_CHK_DIKE
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_VERIFY_AUTH Event: EV_PROC_SA_TS
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_VERIFY_AUTH Event: EV_NO_EVENT
*Jan 18 15:52:31: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down
*Jan 18 15:52:31: IKEv2-ERROR:(SESSION ID = 90,SA ID = 1):: There was no IPSEC policy found for received TS
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_VERIFY_AUTH Event: EV_TS_UNACCEPT
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_BLD_AUTH Event: EV_MY_AUTH_METHOD
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_BLD_AUTH Event: EV_GET_PRESHR_KEY
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_BLD_AUTH Event: EV_GEN_AUTH
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_BLD_AUTH Event: EV_CHK4_SIGN
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_BLD_AUTH Event: EV_OK_AUTH_GEN
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: R_BLD_AUTH Event: EV_SEND_AUTH
*Jan 18 15:52:31: IKEv2-INTERNAL:Construct Vendor Specific Payload: CISCO-GRANITE
*Jan 18 15:52:31: IKEv2-INTERNAL:Construct Notify Payload: TS_UNACCEPTABLE
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: AUTH_DONE Event: EV_OK
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Action: Action_Null
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: AUTH_DONE Event: EV_CHK_GKM_RETRANS
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: AUTH_DONE Event: EV_PKI_SESH_CLOSE
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Closing the PKI session
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: AUTH_DONE Event: EV_UPDATE_CAC_STATS
*Jan 18 15:52:31: IKEv2-INTERNAL:New ikev2 sa request activated
*Jan 18 15:52:31: IKEv2-INTERNAL:Decrement count for incoming negotiating
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: AUTH_DONE Event: EV_INSERT_IKE
*Jan 18 15:52:31: %IKEV2-5-SA_UP: SA UP

*Jan 18 15:52:31: %CRYPTO-5-IKEV2_SESSION_STATUS: Crypto tunnel v2 is UP.  Peer 216.54.89.81:4500 f_vrf:  BLACK i_vrf:  BLACK   Id: TMA01-RTRGU001.gray.csfc.tma01.test
*Jan 18 15:52:31: IKEv2-INTERNAL:Store mib index ikev2 1, platform 90
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: AUTH_DONE Event: EV_CHK_COOP
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: AUTH_DONE Event: EV_CHECK_DUPE
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: AUTH_DONE Event: EV_CHK4_ROLE
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: READY Event: EV_R_OK
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 1 CurState: READY Event: EV_NO_EVENT
*Jan 18 15:52:31: IKEv2-INTERNAL:Got a packet from dispatcher

*Jan 18 15:52:31: IKEv2-INTERNAL:Processing an item off the pak queue

*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Request has mess_id 2; expected 2 through 2

*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 2 CurState: READY Event: EV_RECV_INFO_REQ
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Action: Action_Null
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 2 CurState: INFO_R Event: EV_RECV_INFO_REQ
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 2 CurState: INFO_R Event: EV_CHK_INFO_TYPE
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 2 CurState: INFO_R Event: EV_RECV_DEL
*Jan 18 15:52:31: IKEv2-INTERNAL:Removing child SA with spi 7B677367
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 2 CurState: INFO_R Event: EV_CHK4_ACTIVE_SA
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 2 CurState: INFO_R Event: EV_START_DEL_NEG_TMR
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Action: Action_Null
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 2 CurState: EXIT Event: EV_CHK_PENDING
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Sent response with message id 2, Requests can be accepted from range 3 to 3
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 2 CurState: EXIT Event: EV_NO_EVENT
*Jan 18 15:52:31: IKEv2-INTERNAL:Got a packet from dispatcher

*Jan 18 15:52:31: IKEv2-INTERNAL:Processing an item off the pak queue

*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Request has mess_id 3; expected 3 through 3

*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: READY Event: EV_RECV_INFO_REQ
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Action: Action_Null
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: INFO_R Event: EV_RECV_INFO_REQ
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: INFO_R Event: EV_CHK_INFO_TYPE
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: INFO_R Event: EV_RECV_DEL
*Jan 18 15:52:31: IKEv2-INTERNAL:Returned v4 config addr 172.16.0.93 to local pool
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: INFO_R Event: EV_CHK4_ACTIVE_SA
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: INFO_R Event: EV_STOP_ACCT
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: INFO_R Event: EV_IPSEC_DEL
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: INFO_R Event: EV_START_DEL_NEG_TMR
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Action: Action_Null
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: EXIT Event: EV_CHK_PENDING
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Sent response with message id 3, Requests can be accepted from range 4 to 4
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 3 CurState: EXIT Event: EV_NO_EVENT
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (R) MsgID = 2 CurState: EXIT Event: EV_FREE_NEG
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Deleting negotiation context for peer message ID: 0x2
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (I) MsgID = 1 CurState: READY Event: EV_RECV_DEL
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):Action: Action_Null
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (I) MsgID = 1 CurState: DELETE Event: EV_FREE_SA
*Jan 18 15:52:31: IKEv2-INTERNAL:(SESSION ID = 90,SA ID = 1):SM Trace-> SA: I_SPI=216AE4ED2C350011 R_SPI=F6D9739DEB433912 (I) MsgID = 1 CurState: DELETE Event: EV_FREE_CHKD_SA
*Jan 18 15:52:31: %IKEV2-5-SA_DOWN: SA DOWN

*Jan 18 15:52:31: IKEv2-INTERNAL:IKEv2 tunnel 1 stop, platform index 90 reason 4

*Jan 18 15:52:31: %CRYPTO-5-IKEV2_SESSION_STATUS: Crypto tunnel v2 is DOWN.

HUB config:

no crypto ikev2 authorization policy default
!
crypto ikev2 authorization policy AUTH-POLICY
 pool LOCAL_IP_POOL
 netmask 255.255.255.0
 route set access-list LOCAL_SUBNETS
 route accept any tag 100 distance 2
!
crypto ikev2 proposal IKEV2-PROP
 encryption aes-gcm-256
 prf sha384
 group 20
!
crypto ikev2 policy IKEV2-POLICY
 match fvrf BLACK
 proposal IKEV2-PROP
!
crypto ikev2 keyring IKEV2-KEYRING
 peer TMA00
  description FLEXPVN-SPOKES
  address 0.0.0.0 0.0.0.0
  identity fqdn domain mydomain.com
  pre-shared-key local cisco123
  pre-shared-key remote cisco123
 !
!
!



crypto ikev2 profile IKEV2-PROFILE
 match fvrf BLACK
 match identity remote fqdn domain mydomain.com
 identity local fqdn RTRGU001.mydomain.com
 authentication remote pre-share
 authentication local pre-share
 keyring local IKEV2-KEYRING
 aaa authorization group psk list default AUTH-POLICY
 virtual-template 101
!
!
!
ip tcp synwait-time 10
!
crypto logging session
!
!
crypto ipsec transform-set IPSEC-TSET esp-gcm 256
 mode tunnel
no crypto ipsec transform-set default
!
crypto ipsec profile IPSEC-PROFILE
 set transform-set IPSEC-TSET
 set pfs group20
 set ikev2-profile IKEV2-PROFILE
!
no crypto ipsec profile default

Client(remote) config:

no crypto ikev2 authorization policy default
!
crypto ikev2 authorization policy AUTH-POLICY
 netmask 255.255.255.0
 route set access-list LOCAL_SUBNETS
 route accept any tag 102 distance 2
!
crypto ikev2 proposal IKEV2-PROP
 encryption aes-gcm-256
 prf sha384
 group 20
!
crypto ikev2 policy IKEV2-POLICY
 match fvrf BLACK
 proposal IKEV2-PROP
!
crypto ikev2 keyring IKEV2-KEYRING
 peer TMA00
  description TMA FLEXVPN HUB
  address 0.0.0.0 0.0.0.0
  identity fqdn domain mydomain.com
  pre-shared-key local cisco123
  pre-shared-key remote cisco123
 !
 peer R2
  description FLEXVPN SPOKE(REPEATE FOR 2ND)
 !
!
!
crypto ikev2 profile IKEV2-PROFILE
 match fvrf BLACK
 match identity remote fqdn domain mydomain.com
 identity local fqdn RTRGU001.mydomain.com
 authentication remote pre-share
 authentication local pre-share
 keyring local IKEV2-KEYRING
 aaa authorization group psk list default AUTH-POLICY
 virtual-template 101
!
no crypto ikev2 http-url cert
!
!
!
!
crypto logging session
!
!
!
!
!
!
!
no crypto ipsec transform-set default
crypto ipsec transform-set IPSEC-TSET esp-gcm 256
 mode tunnel
!
crypto ipsec profile IPSEC-PROFILE
 set transform-set IPSEC-TSET
 set pfs group20
 set ikev2-profile IKEV2-PROFILE
!
no crypto ipsec profile default

Show IP nat translations:

RTRTX001#sho ip nat translations
Pro Inside global         Inside local          Outside local         Outside global
esp 192.168.3.17:0        192.168.3.19:0        ---                   ---
udp x.y.89.94:500      192.168.3.19:500      x.y.89.81:500      x.y.89.81:500
udp x.y.89.94:500      192.168.3.19:500      x.y.89.81:500      x.y.89.81:500
udp x.y.89.94:500      192.168.3.19:500      x.y.89.81:500      x.y.89.81:500
udp x.y.89.94:500      192.168.3.19:500      x.y.89.81:500      x.y.89.81:500
udp x.y.89.94:500      192.168.3.19:500      x.y.89.81:500      x.y.89.81:500
udp x.y.89.94:500      192.168.3.19:500      x.y.89.81:500      x.y.89.81:500
udp x.y.89.94:500      192.168.3.19:500      x.y.89.81:500      x.y.89.81:500
udp x.y.89.94:500      192.168.3.19:500      x.y.89.81:500      x.y.89.81:500
udp x.y.89.94:500      192.168.3.19:500      ---                   ---
udp x.y.89.94:4500     192.168.3.19:4500     x.y.89.81:4500     x.y.89.81:4500
udp x.y.89.94:4500     192.168.3.19:4500     ---                   ---

keibler09 · ‎02-04-2022

Hey Folks,

To follow up I switched the crypto ipsec transform-set to transport vs tunnel. This allowed the connection to work through NAT. So digging a little further I added the "tunnel mode ipsec ipv4" command under the tunnel interface on the Remote site and again on the virtual template and changed the ipsec transform-set back to tunnel. Now I am up and running...

interface Tunnel172
 vrf forwarding GRAY
 ip address negotiated
 ip nhrp network-id 1
 ip nhrp shortcut virtual-template 101
 tunnel source GigabitEthernet1
 tunnel mode ipsec ipv4
 tunnel destination x.x.x.x
 tunnel vrf BLACK
 tunnel protection ipsec profile IPSEC-PROFILE

View solution in original post

Rob Ingram · ‎01-19-2022

@keibler09 are the interfaces (physical, virtual-template and tunnel) configured in the correct vrf?

keibler09 · ‎01-19-2022

Yes, all the VRFs are configured correctly. There are no VRFs other than the connection for the VPN on the FVRF & IVRF. When I reconfigure and connect the FLEXVPN remote client directly to the hub the FLEXVPN connects. At that point I can pass traffic from one IVRF to the other. Only when running through NAT I have this issue. NAT-T is enabled and is being detected according to the debug. If I disable NAT-T it recognizes it and tells me that NAT-T is disabled via CLI like it is supposed to. I have also tried extending the " crypto isakmp nat keepalive" to 20 seconds with no luck. Any suggestions? NAT has always been my nemesis.

crypto ipsec nat-transparency udp-encapsulation ( is enabled)

MHM Cisco World · ‎01-20-2022

follow

keibler09 · ‎01-20-2022

I have opened a case with TAC. We spent some time on it today with no resolution. Once we resolve the issue I will post the solution.. l

keibler09 · ‎02-04-2022

Hey Folks,

To follow up I switched the crypto ipsec transform-set to transport vs tunnel. This allowed the connection to work through NAT. So digging a little further I added the "tunnel mode ipsec ipv4" command under the tunnel interface on the Remote site and again on the virtual template and changed the ipsec transform-set back to tunnel. Now I am up and running...

interface Tunnel172
 vrf forwarding GRAY
 ip address negotiated
 ip nhrp network-id 1
 ip nhrp shortcut virtual-template 101
 tunnel source GigabitEthernet1
 tunnel mode ipsec ipv4
 tunnel destination x.x.x.x
 tunnel vrf BLACK
 tunnel protection ipsec profile IPSEC-PROFILE