I've been testing this now for a few days and it seems to work, so I think this is the issue -- a need to exclude the tunnel peer (static) address from NAT explicitly, since the cellular and/or isp dynamic addresses are likely to be private and otherwise included. ... View more

Yes, the issue is definitely that NAT is getting a bad cached value.  During the transition, for reasons I cannot explain, I get a local inside address if the turned-admin-down interface, and inside global address of the newly up WAN interface.  E.g.    #show ip nat trans Pro Inside global Inside local Outside local Outside global udp 172.16.1.100:4500 10.96.68.104:4500 X.X.X.X:4500 X.X.X.X:4500 That first address is the Gig 0 interface that just came up with an ISP's DHCP address, and the 10.96.68.104 address is the no-longer-active address of the Cellular 0 interface that was just shut down.  And traffic just keeps on keeping it alive.  If I clear ip nat trans * then the tunnel immediately comes up and stays up.   I've been partially successful in a kludge of doing the clear in event manager on line translation  but not completely, the translation sticks around a long time.  And I don't want to clear nat translations too long after as that will interfere with unrelated internet access through NAT for real.   What I'd really like to know is how I'm getting a nat translation so crossed up as this -- from one WAN (outside) port to the other WAN (outside) port.  And only while the interfaces are in transition (I'm not quite sure if it's when the new one goes up, or when the old comes down).    I can probably bang away on the event manager kludge of clearing translations and get this to work, but...    Any ideas why it is happening?   ... View more

cco@leferguson.com wrote: I'm worried I need a sniffer somewhere to see where the packets are (not) going.   OK, once I remembered to turn off the route cache, I could see the packets, there is some kind of addressing problem.  Here's an example:      Sep 21 21:55:38.941: IP: s=X.X.X.X (Cellular0), d=10.192.67.219, len 128, input feature Sep 21 21:55:38.941: UDP src=4500, dst=4500, Common Flow Table(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE Sep 21 21:55:38.941: IP: s=X.X.X.X (Cellular0), d=10.192.67.219, len 128, input feature Sep 21 21:55:38.941: UDP src=4500, dst=4500, Stateful Inspection(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE Sep 21 21:55:38.941: IP: s=X.X.X.X (Cellular0), d=10.192.67.219, len 128, input feature Sep 21 21:55:38.941: UDP src=4500, dst=4500, Dialer i/f override(25), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE Sep 21 21:55:38.941: IP: s=X.X.X.X (Cellular0), d=10.192.67.219, len 128, input feature Sep 21 21:55:38.941: UDP src=4500, dst=4500, Virtual Fragment Reassembly(39), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE Sep 21 21:55:38.941: IP: s=X.X.X.X (Cellular0), d=10.192.67.219, len 128, input feature Sep 21 21:55:38.941: UDP src=4500, dst=4500, Virtual Fragment Reassembly After IPSec Decryption(57), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE Sep 21 21:55:38.941: IP: s=X.X.X.X (Cellular0), d=172.16.1.100, len 128, input feature Sep 21 21:55:38.941: UDP src=4500, dst=4500, NAT Outside(92), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE Sep 21 21:55:38.941: IP: s=X.X.X.X (Cellular0), d=172.16.1.100, len 128, input feature Sep 21 21:55:38.941: UDP src=4500, dst=4500, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE Sep 21 21:55:38.941: FIBipv4-packet-proc: route packet from Cellular0 src X.X.X.X dst 172.16.1.100 Sep 21 21:55:38.941: FIBfwd-proc: Default:0.0.0.0/0 process level forwarding Sep 21 21:55:38.941: FIBfwd-proc: depth 0 first_idx 0 paths 1 long 0(0) Sep 21 21:55:38.941: FIBfwd-proc: try path 0 (of 1) v4-sp first short ext 0(-1) Sep 21 21:55:38.941: FIBfwd-proc: v4-sp valid Sep 21 21:55:38.941: FIBfwd-proc: no nh type 8 - deag Sep 21 21:55:38.941: FIBfwd-proc: ip_pak_table 0 ip_nh_table 65535 if none nh none deag 1 chg_if 0 via fib 0 path type special prefix Sep 21 21:55:38.941: FIBfwd-proc: Default:0.0.0.0/0 not enough info to forward via fib (none none) Sep 21 21:55:38.941: FIBipv4-packet-proc: packet routing failed Sep Now here's the thing.... the X.X.X.X is the headquarters outside IP, so is right.  The 10.192.67.219 is the cellular address of the moment.   Part way thru notice the destination changes from 10.192.67.219 to 172.16.1.100.  That latter WAS the IP address of the ethernet (ISP) interface that at this moment is not active.  The real destination should be, I think, the tunnel IP?   That would be 172.25.39.1 which is on the Tunnel Interface as the Lookback (for no-ip usage).    At this point I can see:    Portable1#show ip nat trans global Pro Inside global Inside local Outside local Outside global udp 10.192.67.219:4500 172.16.1.100:4500 X.X.X.X:4500 X.X.X.X:4500 I've got two NAT statements:  ip nat inside source route-map ISP6 interface Cellular0 overload ip nat inside source route-map ISP7 interface GigabitEthernet0 overload the route maps above are:   route-map ISP6 permit 10 match ip address INSIDE-TO-INSIDE-DENY-ACL match interface Cellular0 ! route-map ISP7 permit 10 match ip address INSIDE-TO-INSIDE-DENY-ACL match interface GigabitEthernet0 Ignoring for the moment the ip address, as I understand it ISP6 should have been chosen for translation of data from cellular 0, but it is using the translation in the second NAT -- more to the point it's using a translation that is to an address that no longer even exists.  Worse squared: At no time did 10.192.67.219 (the cellular interface) ever communicate with 172.16.1.100 (the ethernet interface to the ISP).   Both have IP NAT OUTSIDE on them.   So I'm at a real loss what i am doing, but the issue appears to be these two NAT statements and the route map source for them.  Which does work once -- but not after a change.   But I feel like I'm getting closer now.    ... View more

Thanks. I haven't, that may be better, and in fact I could probably then use a whole separate tunnel interface and IP address which might offer yet more segregation. FWIW I tried this in GNS 3 about 6 months ago, and had it working fine, except that the cellular was quite different in how it handles addresses (default gateway isn't inherited the same way). But the switchover was working just fine. But simulation is quite different than reality. This like 800 router is a bit odd. ... View more

OK, that was pointless, I set it up ages ago.  it's a static 1:1 nat so no PAT involved.   I'm worried I need a sniffer somewhere to see where the packets are (not) going.   ... View more

Not sure that changed anything; I am not seeing the messages from them, but then again on-demand is only if no data, and I'm sending a lot of pings already to make sure things are up.   With debug on for ikev2 I see a lot of retransmitting from both sides for payload or ENCR.  Eventually it hist max retransmissions, negotiates again (during which it appears to receive responses), then repeats.   I'm starting to wonder if I have a NAT issue of some sort.  Everything here is NAT'd twice.  The termination router is inside an ASA that's NAT'd, the Portable router is doing NAT itself, the cellular network is providing a NAT'd address also.  But I'm starting to wonder if somewhere in that I'm getting either a port reused that is mis-directing a packet to the prior translation.  I'm starting to dig there a bit, try to find why packets from one end aren't making it to the other (if that's really what's happening).  That might explain why a reboot -- and the ensuring lack of traffic for a period -- might clear a PAT entry.   ... View more

I added this to both routers, is there more that would be needed?   crypto isakmp keepalive 30 10 periodic It had no effect that I can see.   I also added some event manager code so each time there is a change, it shuts the tunnel interface down for one minute to give things time to clear (not sure if that's long enough).  That had no effect either.   ... View more

@RJI wrote: Hi, My first thought would be that you required DPD in order to clear the old SAs. Hmmm... The issue with that is the SA's look like new ones (they have the new address) but it's easy enough to try.  It will take me a bit - I'm in the middle of a different (but similar) experiment, I'm turning off the tunnel on each transition for 60 seconds, to see if that forces things to clear out.   I have also manually done a clear crypto ikev2 sa fast (or whatever that syntax was) on each router, once it gets 'stuck' in this mode, and it has no beneficial impact - the ikev2 sa's go away as expected, but come back with this same weird mode.  I realize ikev2 sa and dpd operate on a slightly different level (though not sure how related it is to clearing them).   I'll give it a try in an hour or so and update. ... View more

Here are the configs.  I've removed some irrelevant stuff for brevity, and redacted some stuff, but I realize these are still long and complex, which is why I tried to frame the original question more conceptually.  But for those who like to see all the code... here it is.  I've omitted things like the VLAN's which attach to local and other devices, etc., which play no role in the tunnels.   And yes I've love a simpler approach to this, but it has to accommodate both EIGRP and two dhcp/negotiated tunnel source addresses, and failover between. Tunnel (headquarters) router:   vtp mode transparent ! crypto ikev2 proposal IKEV2_PROP encryption aes-cbc-256 prf sha512 integrity sha1 group 2 ! crypto ikev2 policy IKEV2_POLICY proposal IKEV2_PROP ! crypto ikev2 keyring IKEV2_KEYRING peer TUNNEL_PEERS address 0.0.0.0 0.0.0.0 pre-shared-key REDACTED ! ! crypto ikev2 profile IKEV2_PROFILE match identity remote any authentication remote pre-share authentication local pre-share keyring local IKEV2_KEYRING virtual-template 1 ! vlan 2101 ! vlan 2137 name toASA lldp run ! crypto isakmp policy 100 encr aes 256 authentication pre-share group 2 ! crypto ipsec transform-set VTISET esp-aes 256 esp-sha-hmac mode tunnel ! crypto ipsec profile IKEV2_IPSEC_PROFILE set security-association lifetime kilobytes 200000 set security-association lifetime seconds 1800 set transform-set VTISET set pfs group2 set ikev2-profile IKEV2_PROFILE ! interface Loopback39 description Interface used to form tunnel ip address 172.25.39.5 255.255.255.0 ! interface GigabitEthernet7 description Internet DMZ connection to ASA firewall specifically for tunnels switchport access vlan 2137 no ip address interface Virtual-Template1 type tunnel ip unnumbered Loopback39 tunnel source Vlan2137 tunnel mode ipsec ipv4 tunnel destination dynamic tunnel protection ipsec profile IKEV2_IPSEC_PROFILE ! interface Vlan2137 description Bogus VLAN because this router won't let me have a non-switch port ip address 172.26.38.5 255.255.255.0 ! router eigrp 10 distribute-list 98 in distribute-list 98 out network 10.0.0.0 network 172.16.0.0 0.15.255.255 network 192.168.0.0 0.0.63.255 ! ip route 0.0.0.0 0.0.0.0 172.26.38.10 access-list 98 deny 0.0.0.0 access-list 98 deny 128.0.0.0 access-list 98 permit any --------------------------------- Portable1 Router: --------------------------------- ethernet lmi ce ip dhcp excluded-address 172.25.36.31 ! ip dhcp pool PORTABLE_POOL network 172.25.36.0 255.255.255.0 lease 0 0 5 ip cef ! multilink bundle-name authenticated ! chat-script lte "" "AT!CALL" TIMEOUT 20 "OK" ! license udi pid C819G-LTE-MNA-K9 sn REDACTED ! no spanning-tree vlan 1922 no spanning-tree vlan 2139 vtp mode transparent ! crypto ikev2 proposal IKEV2_PROP encryption aes-cbc-256 prf sha512 integrity sha1 group 2 ! crypto ikev2 policy IKEV2_POLICY proposal IKEV2_PROP ! crypto ikev2 keyring IKEV2_KEYRING peer TUNNEL_PEERS address x.x.x.x <<<< public address of ASA's interface that NAT's to tunnel router above pre-shared-key REDACTED ! ! crypto ikev2 profile IKEV2_PROFILE match identity remote any authentication local pre-share authentication remote pre-share keyring local IKEV2_KEYRING ! crypto ikev2 client flexvpn FLEXVPN_IKEV2_CLIENT peer 1 x.x.x.x <<< Same as x.x.x.x above source 1 GigabitEthernet0 track 7 source 2 Cellular0 track 6 client connect Tunnel31 ! controller Cellular 0 lte sim data-profile 14 attach-profile 14 slot 0 lte sim data-profile 14 attach-profile 14 slot 1 lte modem link-recovery rssi onset-threshold -110 lte modem link-recovery monitor-timer 20 lte modem link-recovery wait-timer 10 lte modem link-recovery debounce-count 6 ! track 6 ip sla 6 reachability delay down 30 up 30 ! track 7 ip sla 7 reachability delay down 30 up 30 ! crypto isakmp invalid-spi-recovery ! crypto ipsec transform-set VTISET esp-aes 256 esp-sha-hmac mode tunnel ! crypto ipsec profile IKEV2_IPSEC_PROFILE set security-association lifetime kilobytes 200000 set security-association lifetime seconds 1800 set transform-set VTISET set pfs group2 set ikev2-profile IKEV2_PROFILE ! interface Loopback39 description Interface used to form tunnel ip address 172.25.39.1 255.255.255.0 ! interface Tunnel31 ip unnumbered Loopback39 ip nat inside ip virtual-reassembly in tunnel source dynamic tunnel mode ipsec ipv4 tunnel destination dynamic tunnel protection ipsec profile IKEV2_IPSEC_PROFILE ! interface Cellular0 description Connection to GSM modem (secondary internet if primary down) ip dhcp client lease 0 0 5 ip address negotiated ip nat outside ip virtual-reassembly in encapsulation slip dialer in-band dialer string lte dialer-group 1 async mode interactive ! interface GigabitEthernet0 description Connection to local intenet (if any) ip dhcp client default-router distance 100 ip dhcp client lease 0 0 5 ip address dhcp ip nat outside ip virtual-reassembly in shutdown duplex full speed auto ! interface Serial0 no ip address shutdown clock rate 2000000 ! router eigrp 10 distribute-list 98 in distribute-list 98 out network 10.0.0.0 network 172.16.0.0 0.15.255.255 network 192.168.0.0 0.0.63.255 passive-interface GigabitEthernet0 passive-interface Cellular0 passive-interface Cellular1 ! ip local policy route-map CHOOSE-ISP ip forward-protocol nd ip nat inside source route-map ISP6 interface Cellular0 overload ip nat inside source route-map ISP7 interface GigabitEthernet0 overload ip route 0.0.0.0 0.0.0.0 Cellular0 100 track 6 ip access-list extended INSIDE-TO-INSIDE-DENY-ACL deny ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255 deny ip 10.0.0.0 0.255.255.255 192.168.0.0 0.0.63.255 deny ip 10.0.0.0 0.255.255.255 172.16.0.0 0.15.255.255 deny ip 192.168.0.0 0.0.63.255 10.0.0.0 0.255.255.255 deny ip 192.168.0.0 0.0.63.255 192.168.0.0 0.0.63.255 deny ip 192.168.0.0 0.0.63.255 172.16.0.0 0.15.255.255 deny ip 172.16.0.0 0.15.255.255 10.0.0.0 0.255.255.255 deny ip 172.16.0.0 0.15.255.255 192.168.0.0 0.0.63.255 deny ip 172.16.0.0 0.15.255.255 172.16.0.0 0.15.255.255 permit ip 10.0.0.0 0.255.255.255 any permit ip 192.168.0.0 0.0.63.255 any permit ip 172.16.0.0 0.15.255.255 any permit ip any 172.0.0.0 0.0.255.255 ! ip sla 6 icmp-echo 8.8.8.8 source-interface Cellular0 <<<< No particular reason to use different track addresses other than easier to noitice in debug tag Cellular Connection up test frequency 10 ip sla schedule 6 life forever start-time now ip sla 7 icmp-echo 75.75.75.75 source-interface GigabitEthernet0 tag Local ISP connection up test frequency 10 ip sla schedule 7 life forever start-time now dialer-list 1 protocol ip permit ! route-map CHOOSE-ISP permit 10 match ip address 106 set interface Cellular0 ! route-map CHOOSE-ISP permit 20 match ip address 107 set interface GigabitEthernet0 ! route-map ISP6 permit 10 match ip address INSIDE-TO-INSIDE-DENY-ACL match interface Cellular0 ! route-map ISP7 permit 10 match ip address INSIDE-TO-INSIDE-DENY-ACL match interface GigabitEthernet0 ! access-list 6 permit 8.8.8.8 access-list 7 permit 75.75.75.75 access-list 98 deny 0.0.0.0 access-list 98 deny 128.0.0.0 <<<<< Long irrelevant story but this is distributed as a static route in EIGRP but shouldn't be on this router access-list 98 permit any access-list 106 permit ip any host 8.8.8.8 access-list 107 permit ip any host 75.75.75.75 ! control-plane ! line 3 <<< Not sure why line3 goes with cell 0 but it does. script dialer lte modem InOut no exec event manager applet TURN-GSM-OFF-IF-INTERNET-UP description Force cellular internet off anytime the local ISP is up and tracking event track 7 state up action 1.0 cli command "enable" action 2.0 cli command "config t" action 3.0 cli command "interface Cell 0" action 4.0 cli command "shutdown" action 5.0 syslog msg "Local ISP (track 7) up, turned Interface Cellular 0 to reduce cost (also clears routes)" action 6.0 cli command "end" event manager applet TURN-GSM-ON-IF-INTERNET-DOWN description Force cellular internet on anytime the local ISP is down event track 7 state down action 1.0 cli command "enable" action 2.0 cli command "config t" action 3.0 cli command "interface Cell 0" action 4.0 cli command "no shutdown" action 5.0 syslog msg "Local ISP (track 7) down, turned Interface Cellular 0 on to enable backup" action 6.0 cli command "end" event manager applet CLEAR-LOCAL-ISP-BRIEFLY-ON-FAILURE description Force local internet off for a while (to clear routes) if it fails, falling back on cellular event track 7 state down maxrun 2000 action 1.0 cli command "enable" action 2.0 cli command "config t" action 3.0 cli command "interface GigabitEthernet 0" action 4.0 cli command "shutdown" action 4.5 syslog msg "Cycled Interface GigabitEthernet 0 down to clear routes after track failed" action 5.0 wait 900 action 6 cli command "no shutdown" action 6.5 cli command "end" action 7.0 syslog msg "Cycled interface GigabitEthernet 0 back on to try again" event manager applet TICKLE-GSM-PERIODICALLY description The SLA won't cause interesting traffic so we need to hit it manually occasionally (this does nothing when down) event timer watchdog time 30 action 0.7 cli command "enable" action 1.0 cli command "ping 8.8.8.8" ! end ... View more

Thank you. I should have updated this; I spoke to Cisco a bit ago who gave a somewhat different interpretation of process but with the same result -- on one it tracks in and out, on both there is no duplication. ... View more