Industrial ISR 1101 IPSec Latency Spikes over LTE 4G

[Christoph1603]

Hi,

I am experiencing a rather strange problem with the "new" ISR 1101 IoT router.

Our Setup:

VPN Hub / Gateway with dynamic Crypto Map: 2x ASA 5545-X with 8.9.4(10) HA

2 ISPs with static IP ranges, nothing out of the ordinary

Interfaces are IKEv2 enabled

VPN Spokes: 150x ISR 1101 with p-lte-gb Module (WP7607) and 2 different WISP SIM Cards. (FW 16.11.1)

Public/Private dynamic IPs, NAT-T, DPD, IKEv2, basic Tunnel Mode IPSEC, no VTI, etc.

10.66.1.1 -> ASA -> ISP -> WISP -> Cell0/1/0 -> ISR1101 -> vlan 1 -> 10.48.199.254/24

Since overall everthing is working fine except for one little detail, 

I didn't want to spam the community with the full config c&p right away.

If I am pinging the router 10.48.199.254 from 10.66.1.1 I get very unstable response times:

Antwort von 10.48.4.254: Bytes=32 Zeit=319ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=27ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=27ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=182ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=138ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=32ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=48ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=37ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=27ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=32ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=191ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=29ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=98ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=27ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=32ms TTL=254
Antwort von 10.48.4.254: Bytes=32 Zeit=32ms TTL=254

Up to 50 ms would be normal for this type of connection, but very regulary the latency shoots up to a few hundret ms.

Now this wouldn't puzzle me (MTU, unstable connection,etc) if it were not for the fact, that there is only one environmental situation that causes this to happen:

Ping from ISR1101 vlan1 to Inside ASA (LTE, WISP1&2, idle and traffic) -> normal,stable

Ping from ISR1101 vlan1 to Inside ASA (WCDMA, WISP1&2, idle and traffic) -> normal,stable

Ping from ISR1101 vlan1 to Inside ASA (TestWiredConnection, idle and traffic) -> normal,stable

Ping from Public ISR1101  to Public ASA (WCDMA/LTE, WISP1&2, idle and traffic) -> normal, stable

Ping from Public ASA to Public ISR1101 (WCDMA/LTE, WISP1&2, idle and traffic) -> normal, stable

Ping from Inside ASA to ISR1101 vlan 1 (WCDMA, WISP1&2, idle and traffic) -> normal, stable

Ping from Inside ASA to ISR1101 vlan 1 (Wired Test, idle and traffic) -> normal, stable

Ping from ISR1101 vlan1 to Inside ASA (LTE, WISP1&2, with background traffic) -> normal,stable

Ping from ISR1101 vlan1 to Inside ASA (LTE, WISP1&2, idle, no backgroundtraffic) -> HIGH Latency Spikes

As soon as the IPSEC Line is loaded with traffic the spikes vanish and latency even goes further down to 20-30ms.

Has anybody experienced such a problem before?

[Georg Pauwen]

Hello,

strange indeed, as you would expect the opposite to happen (response times going up when the link is loaded)...

When you do a traceroute rather than a ping, can you get an indication of where in the path the latency occurs ?

[Christoph1603]

Hi, thank for the reply.

Unfortunately, tracroute doesn't provide any additional information since there is no 

additional hop between ASA and Router VLAN (logically speaking) while using the IPSEC tunnel.

Using the public network path, the problem doesn't occure.

I've also done some additional testing and it doesn't matter how the Cellular interface is put under stress.

I'm pinging through the tunnel -> high latency spikes, as soon any a 3rd device (even located on the internet)

sends big icmp requests (eg 1400) to the public Cellular IP in order to generate traffic, everything normalizes.

Currently my only suspicion would be some kind of strange buffer/power scheduling problem, that doesn't properly detect ipsec packets in inbound direction.

[Georg Pauwen]

Hello,

can you post the configs of the ASA and the ISR ?

[Christoph1603]

Router Config:

Spoiler

!
crypto ikev2 proposal L2L-Prop
encryption aes-cbc-256
integrity sha512
group 19
no crypto ikev2 proposal default
!
crypto ikev2 policy L2L-Pol
proposal L2L-Prop
no crypto ikev2 policy default
!
crypto ikev2 keyring L2L-Keyring
peer vpn
address 0.0.0.0 0.0.0.0
pre-shared-key local ************
pre-shared-key remote **************
!
!
!
crypto ikev2 profile L2L-Prof
match identity remote any
identity local key-id isr-1101
authentication remote pre-share
authentication local pre-share
keyring local L2L-Keyring
!
crypto ikev2 nat keepalive 20
crypto ikev2 dpd 10 2 periodic
no crypto ikev2 http-url cert
crypto ikev2 fragmentation
!
controller Cellular 0/1/0
lte sim data-profile 10 attach-profile 10 slot 0
lte sim data-profile 12 attach-profile 12 slot 1
lte sim max-retry 65535
no lte firmware auto-sim
no lte gps enable
pdn 1 dns-ignore ipv4v6
pdn 2 dns-ignore ipv4v6
profile id 10 apn ... pdn-type ipv4
profile id 12 apn... pdn-type ipv4
!
!
vlan internal allocation policy ascending
!
crypto isakmp invalid-spi-recovery
!
!
crypto ipsec transform-set ESP-AES-SHA esp-aes 256 esp-sha-hmac
mode tunnel
!
!
!
crypto map vpn 10 ipsec-isakmp
set peer ##peerip1 default
set peer ##peerip2
set security-association lifetime kilobytes disable
set security-association lifetime seconds 86400
set security-association replay window-size 1024
set transform-set ESP-AES-SHA
set ikev2-profile L2L-Prof
match address NNOE_CALC
reverse-route
!
!
!
interface GigabitEthernet0/0/0
ip address dhcp
ip access-group NNOE_IN in
crypto map vpn
ip virtual-reassembly
!
...
interface Cellular0/1/0
ip address negotiated
ip access-group NNOE_IN in
load-interval 30
dialer in-band
dialer idle-timeout 0
dialer watch-group 1
dialer-group 1
pulse-time 1
crypto map vpn
ip virtual-reassembly
!
interface Vlan1
ip address 10.48.9.254 255.255.255.0
no autostate
!
ip http client source-interface Vlan1
ip tftp source-interface Vlan1
ip route 0.0.0.0 0.0.0.0 Cellular0/1/0 10
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0/0 dhcp 5
ip ssh source-interface Vlan1
!
ip access-list standard NNOE_IN
......
deny any
!
ip access-list extended NNOE_CALC
permit ip 10.48.9.0 0.0.0.255 10.66.0.0 0.0.255.255
permit ip 10.48.9.0 0.0.0.255 10.75.0.0 0.0.255.255
!
logging alarm minor
logging trap warnings
logging source-interface Vlan1
logging host ***************
dialer-list 1 protocol ip permit

ASA Config:

Spoiler

crypto map outside-map 9199 ipsec-isakmp dynamic isr-1101

crypto dynamic-map isr-1101 999 match address outside-9199.999
crypto dynamic-map isr-1101 999 set ikev2 ipsec-proposal AES256
crypto dynamic-map isr-1101 999 set security-association lifetime seconds 86400
crypto dynamic-map isr-1101 999 set security-association lifetime kilobytes unlimited
crypto dynamic-map isr-1101 999 set reverse-route

access-list outside-9199.999 extended permit ip object-group NET_ISR-local-permit object NET_ISR-any

group-policy GPO-ISR internal
group-policy GPO-ISR attributes
vpn-idle-timeout none
vpn-session-timeout none
vpn-filter none
vpn-tunnel-protocol ikev2
ipsec-udp enable
periodic-authentication certificate none

tunnel-group isr-1101 type ipsec-l2l
tunnel-group isr-1101 general-attributes
default-group-policy GPO-ISR

tunnel-group isr-1101 ipsec-attributes
ikev1 pre-shared-key *****
peer-id-validate nocheck
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****

nat (inside,outside) source static NET_ISR-local-permit NET_ISR-local-permit destination static NET_ISR-any NET_ISR-any no-proxy-arp route-lookup description !! isr IPSEC NAT Exception

crypto ikev2 policy 1
encryption aes-256
integrity sha512
group 19 24
prf sha512
lifetime seconds 86400

crypto ipsec ikev2 ipsec-proposal AES256
protocol esp encryption aes-256
protocol esp integrity sha-1 md5