Can only ping across VTI or DMVPN as long as no crypto is applied

[westj]

I have a network where the client router cannot pass encrypted traffic. I see the SAs form, the DMVPN tunnel is up and I am getting encrypt on one side. And here is where it is confusing, if I remove the transform-set everything is fine.

Home side

crypto isakmp policy 1
 encr aes 256
 authentication pre-share
crypto isakmp key password address 0.0.0.0        
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 30 20 periodic
crypto isakmp diagnose error 10

!
crypto ipsec transform-set LTE ah-sha-hmac esp-aes 256
 mode transport
!
crypto ipsec profile LTE
 set transform-set LTE
!
interface Tunnel910
 ip address 99.0.0.1 255.255.255.0
 no ip redirects
 ip nbar protocol-discovery
 ip flow ingress
 ip nhrp authentication TTPS
 ip nhrp map multicast dynamic
 ip nhrp network-id 2
 ip nhrp holdtime 360
 ip virtual-reassembly in
 load-interval 30
 qos pre-classify
 tunnel source GigabitEthernet0/0
 tunnel mode gre multipoint
 tunnel key 1
 tunnel protection ipsec profile LTE

Remote Side

crypto isakmp policy 1
 encr aes 256
 authentication pre-share
crypto isakmp key password address 0.0.0.0        
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 30 20 periodic
crypto isakmp diagnose error 10
!
!
crypto ipsec transform-set LTE ah-sha-hmac esp-aes 256
 mode transport
!
crypto ipsec profile LTE
 set transform-set LTE
!
interface Tunnel910
 ip address 99.0.0.2 255.255.255.0
 ip nhrp authentication TTPS
 ip nhrp map multicast 172.30.1.1
 ip nhrp map 99.0.0.1 172.30.1.1
 ip nhrp network-id 2
 ip nhrp holdtime 360
 ip nhrp nhs 99.0.0.1
 ip tcp adjust-mss 1250
 qos pre-classify
 tunnel source Ethernet1/0
 tunnel destination 172.30.1.1
 tunnel key 1
 tunnel protection ipsec profile LTE

The tunnels are up/up on both sides, and when I ping the home side of the tunnel from the remote I get encrypts:

Lynx-2E#ca
Crypto Engine Connections

   ID  Type    Algorithm           Encrypt  Decrypt LastSeqN IP-Address
   31  IPsec   SHA+AES256                0        0        0 192.168.0.1
   32  IPsec   SHA+AES256                1        0        0 192.168.0.1
 1005  IKE     SHA+AES256                0        0        0 192.168.0.1

Lynx-2E#p 99.0.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 99.0.0.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Lynx-2E#ca        
Crypto Engine Connections

   ID  Type    Algorithm           Encrypt  Decrypt LastSeqN IP-Address
   31  IPsec   SHA+AES256                0        0        0 192.168.0.1
   32  IPsec   SHA+AES256                6        0        0 192.168.0.1
 1005  IKE     SHA+AES256                0        0        0 192.168.0.1

But the home side gets an encapsulation failed when I try to ping the remote tunnel IP:

home #p 99.0.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 99.0.0.2, timeout is 2 seconds:

IP: s=99.0.0.1 (local), d=99.0.0.2 (Tunnel910), len 100, sending
IP: s=99.0.0.1 (local), d=99.0.0.2 (Tunnel910), len 100, output feature, Common Flow Table(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: s=99.0.0.1 (local), d=99.0.0.2 (Tunnel910), len 100, output feature, Stateful Inspection(28), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: s=99.0.0.1 (local), d=99.0.0.2 (Tunnel910), len 100, output feature, QoS Preclassification(71), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: s=99.0.0.1 (local), d=99.0.0.2 (Tunnel910), len 100, output feature, Post-Ingress-NetFlow(74), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: s=99.0.0.1 (local), d=99.0.0.2 (Tunnel910), len 100, encapsulation failed.
IP: s=99.0.0.1 (local), d=99.0.0.2 (Tunnel910), len 100, sending

No encrypts:

ca
Crypto Engine Connections

   ID  Type    Algorithm           Encrypt  Decrypt LastSeqN IP-Address
   39  IPsec   SHA+AES256                0        0        0 172.30.1.1
   40  IPsec   SHA+AES256                0        0        0 172.30.1.1
 9005  IKE     SHA+AES256                0        0        0 172.30.1.1

And with no transform set all is well:

B12_3925(config)#in tu910
B12_3925(config-if)#no tunnel protection ipsec profile LTE
B12_3925(config-if)#
*Sep  2 17:09:26: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

B12_3925#p 99.0.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 99.0.0.2, timeout is 2 seconds:
!!!!!

If I put the transform set back on it looks like the home is actually seeing the packets from the remote side:

remote: ping 99.0.0.1

home:

 s=192.168.102.130 (GigabitEthernet0/0 [remote tunnel outside IP), d=172.30.1.1 (GigabitEthernet0/0), len 124, rcvd 3
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1, len 124, stop process pak for forus packet
IP: s=172.30.1.1 (local), d=192.168.102.130 (GigabitEthernet0/0), len 124, se
B12_3925#nding
IP: s=172.30.1.1 (local), d=192.168.102.130 (GigabitEthernet0/0), len 124, output feature, Post-Ingress-NetFlow(74), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: s=172.30.1.1 (local), d=192.168.102.130 (GigabitEthernet0/0), len 124, sending full packet
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1, len 188, input feature, MCI Check(94), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: tableid=0, s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1 (GigabitEthernet0/0), routed via RIB
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1 (GigabitEthernet0/0), len 188, rcvd 3
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1, len 188, stop process pak for forus packet

B12_3925#
B12_3925#ca
Crypto Engine Connections

   ID  Type    Algorithm           Encrypt  Decrypt LastSeqN IP-Address
   41  IPsec   SHA+AES256                0        0        0 172.30.1.1
   42  IPsec   SHA+AES256                0        0        0 172.30.1.1
 9007  IKE     SHA+AES256                0        0        0 172.30.1.1

Any idea why the IPSec does not work?

Full configs, crypto sa and network topo are attached.

Thank you

[Marcin Latosiewicz]

Not sure about the platforms you're running on, but not all platforms support nesting AH and ESP. 

Might want to try with just ESP and/or AES-GCM, vide :

http://www.cisco.com/web/about/security/intelligence/nextgen_crypto.html 

[westj]

Thanks for the suggestion, I am running the 5921 software router and an ISR G2 3925. The crypto worked across wired, it is just across the cellular link that is causing me issues.

[Marcin Latosiewicz]

Could be some weird padding/fragmentation problem in transit. 

" if I remove the transform-set everything is fine" <--- you mean when you remove crypto from DMVPN or use a different transform-set?

[westj]

Correct, if I remove the crypto from the DMVPN.

You are right about the weird padding/frag. If I put my DMVPN inside another tunnel the crypto works! I know that a tunnel in a tunnel is a horrible idea, I was testing a theory about an issue with the underlying infrastructure. I control the eNB and LTE EPC, so it is all my set up.

[Marcin Latosiewicz]

Do you mind trying using pure ESP for test? 

You also might also use tunnel path mtu discovery ... to see what happens rather than alleviating anything (do we start receiving ICMP unreachables?) 

You can also try to see if you're able to get some specific packet sizes to pass through:

You can calculate from here https://cway.cisco.com/tools/ipsec-overhead-calc/ipsec-overhead-calc.html

Taking a step back I'd like to see, if we're getting encap here and decap on remote end. 

The encap failed you indicate in debugs is typically happening when resolution did not work (ARP, NHRP etc), among other scenarios.

[westj]

No joy with esp only:

crypto ipsec transform-set esp-only esp-aes 256
 mode transport

!

interface Tunnel800
 description VTI in a GRE tunnel (ugly)
 ip address 8.0.0.2 255.255.255.0
 tunnel source 192.168.0.1
 tunnel destination 172.30.1.1
 tunnel path-mtu-discovery
 tunnel protection ipsec profile esp-only

Lynx-2E#ca
Crypto Engine Connections

   ID  Type    Algorithm           Encrypt  Decrypt LastSeqN IP-Address
    1  IPsec   AES256                    0        9        0 192.168.0.1
    2  IPsec   AES256                   19        0        0 192.168.0.1
 1003  IKE     SHA+AES256                0        0        0 192.168.0.1

Lynx-2E#ping 8.0.0.1      
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.0.0.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Lynx-2E#ca
Crypto Engine Connections

   ID  Type    Algorithm           Encrypt  Decrypt LastSeqN IP-Address
    1  IPsec   AES256                    0       11        0 192.168.0.1
    2  IPsec   AES256                   26        0        0 192.168.0.1
 1003  IKE     SHA+AES256                0        0        0 192.168.0.1

It looks like it is working, and I see packets on the both sides (below). but no response to pings.

B12_3925#
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1, len 164, input feature, MCI Check(94), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: tableid=0, s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1 (GigabitEthernet0/0), routed via RIB
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1 (GigabitEthernet0/0), len 164, rcvd 3
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1, len 164, stop process pak for forus packet
B12_3925#
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1, len 164, input feature, MCI Check(94), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: tableid=0, s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1 (GigabitEthernet0/0), routed via RIB
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1 (GigabitEthernet0/0), len 164, rcvd 3
IP: s=192.168.102.130 (GigabitEthernet0/0), d=172.30.1.1, len 164, stop process pak for forus packet
B12_3925#

Lynx-2E#
*Sep  3 17:03:07: IP: s=192.16.1.1 (Tunnel800), d=192.16.2.2, len 100, input feature, MCI Check(108), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Sep  3 17:03:07: IP: tableid=0, s=192.16.1.1 (Tunnel800), d=192.16.2.2 (Ethernet0/1), routed via RIB
*Sep  3 17:03:07: IP: s=192.16.1.1 (Tunnel800), d=192.16.2.2, len 100, rcvd 4
*Sep  3 17:03:07: IP: s=192.16.1.1 (Tunnel800), d=192.16.2.2, len 100, stop process pak for forus packet
*Sep  3 17:03:07: IP: s=192.16.1.1 (Tunnel800), d=192.16.2.2, len 100, enqueue feature, TCP Adjust MSS(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

And if I remove the IPSec it all works:

interface Tunnel800
 description VTI in a GRE tunnel (ugly)
 ip address 8.0.0.2 255.255.255.0
 tunnel source 192.168.0.1
 tunnel destination 172.30.1.1
 tunnel path-mtu-discovery
 tunnel protection ipsec profile esp-only
end

Lynx-2E#c
Enter configuration commands, one per line.  End with CNTL/Z.
Lynx-2E(config)#in tu800
Lynx-2E(config-if)#no tunnel protection ipsec profile esp-only
Lynx-2E(config-if)#^Z
Lynx-2E#p
*Sep  3 17:04:01: %SYS-5-CONFIG_I: Configured from console by console
Lynx-2E#p 8.0.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/36/48 ms
Lynx-2E#p 192.16.1.1 sou 192.16.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.16.1.1, timeout is 2 seconds:
Packet sent with a source address of 192.16.2.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 31/40/48 ms
Lynx-2E#

[westj]

Some more info. I am clearly encrypting packets on both sides, and the client (right side of drawing) is decrypting packets from the home side. But the home side is not seeing or decrypting packets from the remote.

home sending to remote:

p 192.16.2.2 sou 192.16.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.16.2.2, timeout is 2 seconds:
Packet sent with a source address of 192.16.1.1

Before encryption:
3CA22C50:                   4500007C 02A50000          E..|.%..
3CA22C60: FF2FE463 AC1E0101 C0A86682 00000800  ./dc,...@(f.....
3CA22C70: 45000064 00780000 FF0137FD C0100101  E..d.x....7}@...
3CA22C80: C0100202 0800A7DD                    @.....']         ...
After encryption:
3CA23610:                   4500009C 02A50000          E....%..
3CA23620: FF11E461 AC1E0101 C0A86682 11941194  ..da,...@(f.....
3CA23630: 00880000 39D5AF86 0000002D 00000800  ....9U/....-....
3CA23640: 45000064 00780000                    E..d.x..         ...
post_crypto_ip_encrypt: Data just encrypted, 156 bytes
Process switched encrypted packet
Punt packet to process switch.

Remote getting the packet:

*Sep  3 17:50:53: post_crypto_ip_decrypt: Data just decrypted, 124 bytes
*Sep  3 17:50:53: crypto_tun_post_decrypt_switch: oce_les_inline returned 1 with tun_decap_oce 3FA5EE08. punting PS.
*Sep  3 17:50:53: PostDecrypt::crypto_tun: calling process switching
*Sep  3 17:50:53: Punt packet to process switch
*Sep  3 17:50:53: Before encryption:

other way (remote to home):

Remote is encrypting the packets:

p 192.16.1.1 sou 192.16.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.16.1.1, timeout is 2 seconds:
Packet sent with a source address of 192.16.2.2

*Sep  3 17:51:50: Before encryption:
41F8F060:                   4500007C 02C60000          E..|.F..
41F8F070: FF2F4AC4 C0A80001 AC1E0101 00000800  ./JD@(..,.......
41F8F080: 45000064 003C0000 FF013839 C0100202  E..d.<....89@...
41F8F090: C0100101 080099EC                    @......l         ...
*Sep  3 17:51:50: After encryption:
41FAAD90: 4500009C 02C60000 FF114AC2 C0A80001  E....F....JB@(..
41FAADA0: AC1E0101 11941194 00880000 B6034179  ,...........6.Ay
41FAADB0: 0000003C 00000800 45000064 003C0000  ...<....E..d.<..
41FAADC0:                                                       ...
*Sep  3 17:51:50: post_crypto_ip_encrypt: Data just encrypted, 156 bytes
*Sep  3 17:51:50: Process switched encrypted packet
*Sep  3 17:51:50: Punt packet to process switch.
*Sep  3 17:51:52: Before encryption:

home side sees nothing.

[Marcin Latosiewicz]

You might want to try EPC to check if you're seeing those packets ingressing/egressing correctly ... but if you do ... "Dear ISP"?

http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-embedded-packet-capture/116045-productconfig-epc-00.html

https://supportforums.cisco.com/document/139686/configuration-example-embedded-packet-capture-cisco-ios-and-ios-xe

[pjain2]

99.0.0.1 and 99.0.0.2 are your public ip addresses. Am i correct? 

after you remove your ipsec profile, the crypto is disabled; hence pings from one end to 99.0.0.2 are going to go out through internet.

can you check your DMVPN config once again.

the tunnel interface ip would be some random private ip address on both the ends in the same subnet and the tunnel destination and source would be your public ip addresses.

[westj]

99.x.x.x are public IPs, but this is a stand alone network with no access to the Internet.

I have a static route to the far side of each router that uses the tunnel far IP as the destination, it does not change when I remove the crypto.

I am using VTIs now. I was testing with VTI and DMVPN, but the VTI is just easier to set up quickly.