ISR IPsec site-to-site VPN - cannot manage routers through tunnel

[thisd00d]

Problem: the routers at either end cannot be managed/reached (SSH, SNMP, ICMP) via the tunnel.

I have two routers located at different locations. An IPsec tunnel via the Internet is configured and working. The tunnel itself is established with IPv6 and the traffic flowing through the tunnel is IPv4. The two Cisco ISRs are both configured with ZBF - I assume this is where the problem is.

Monitoring R1 with SNMP at site A works because the SNMP server resides on this network and site. I can also monitor SNMP devices behind R2 at site B via the tunnel. However, any traffic to/from the router itself at site B is dropped.

The same is true if I attempt to access or ping the router at site A from site B - traffic is dropped and I can only reach devices behind the routers.

What confuses me is that traffic to and from the self zone is permitted by default. In that case where and why is the traffic dropped when it is transmitted via the tunnel?

The full config is 500 lines, hence just the relevant parts including ZBF and crypto of show run for R2 is inserted below (I can of course share more of the config and also for R1 too if necessary, both sites are similar).

The network 172.16.11.0/24 is the INSIDE zone on R1 and is missing in the ACL "SSH-ACCESS" for now. However, should it not be possible to ping and SNMP to the self zone?

R2#show run
Building configuration...
!
crypto ikev2 proposal IKEV2_PROP2
encryption aes-gcm-256
prf sha384
group 20
no crypto ikev2 proposal default
!
crypto ikev2 policy IKEV2_POLICY2
proposal IKEV2_PROP2
no crypto ikev2 policy default
!
!
crypto ikev2 profile IKEV2_PROF2
match identity remote address 2100:zz:zz:D00D::/128
authentication remote pre-share key key1
authentication local pre-share key key2
!
!
class-map type inspect match-any CM_VPN_TRAFFIC
match protocol dns
match protocol snmp
match protocol icmp
match protocol http
match protocol https
match protocol syslog
match protocol ssh
match protocol snmptrap
class-map type inspect match-all CM_NON_ISAKMP
match protocol ntp
class-map type inspect match-any CM_INSIDE_PROTOCOLS
match protocol tcp
match protocol udp
match protocol icmp
class-map type inspect match-any CM_ISAKMP
match protocol isakmp
match access-group name ALLOW_ESP
match access-group name ALLOW_DHCP
!
policy-map type inspect PM_ISAKMP
class type inspect CM_ISAKMP
pass
class type inspect CM_NON_ISAKMP
pass
class class-default
drop
policy-map type inspect PM_VPN_TRAFFIC
class type inspect CM_VPN_TRAFFIC
inspect
class class-default
drop
policy-map type inspect PM_INSIDE_TO_INTERNET
class type inspect CM_INSIDE_PROTOCOLS
inspect
class class-default
drop
!
zone security INSIDE
zone security INTERNET
zone security VPN
zone security IOT
zone-pair security ZP_INSIDE_TO_INTERNET source INSIDE destination INTERNET
service-policy type inspect PM_INSIDE_TO_INTERNET
zone-pair security ZP_INSIDE_TO_VPN source INSIDE destination VPN
service-policy type inspect PM_VPN_TRAFFIC
zone-pair security ZP_VPN_TO_INSIDE source VPN destination INSIDE
service-policy type inspect PM_VPN_TRAFFIC
zone-pair security ZP_INTERNET_TO_self source INTERNET destination self
service-policy type inspect PM_ISAKMP
zone-pair security ZP_IOT_TO_INTERNET source IOT destination INTERNET
service-policy type inspect PM_INSIDE_TO_INTERNET
!
no crypto isakmp default policy
!
!
no crypto ipsec transform-set default
crypto ipsec transform-set TRANS2 esp-gcm 256
mode tunnel
!
crypto ipsec profile IPSEC_PROF2
set transform-set TRANS2
set pfs group20
set ikev2-profile IKEV2_PROF2
!
!
!
!
!
interface Tunnel0
ip address 192.168.9.2 255.255.255.0
ip mtu 1362
zone-member security VPN
ip tcp adjust-mss 1280
ipv6 address FC09::2/64
ipv6 enable
tunnel source 2200:zz:zz:FEED::
tunnel mode gre ipv6
tunnel destination 2100:zz:zz:D00D::
tunnel protection ipsec profile IPSEC_PROF2
!
!
ip route 172.16.11.0 255.255.255.0 Tunnel0 192.168.9.1
ip route 172.31.255.254 255.255.255.255 Tunnel0 192.168.9.1
!
!
ip access-list extended ALLOW_DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
deny ip any any
ip access-list extended SSH-ACCESS
permit tcp 172.16.12.0 0.0.0.255 any eq 22
deny ip any any

ip access-list standard SNMPv3-ACL
permit 172.16.11.0 0.0.0.255
deny any
!
!
!
ipv6 access-list ALLOW_ESP
permit esp host 2100:zz:zz:D00D:: host 2200:zz:zz:FEED::
deny ipv6 any any

!
control-plane host
management-interface Tunnel0 allow ssh snmp
management-interface Vlan12 allow ssh snmp
!
!
control-plane
!
line con 0
logging synchronous
login local
line vty 0 4
access-class SSH-ACCESS in
login local
transport input ssh
!
end

[thisd00d]

@Rob Ingram 

Please find full configs of R1 and R2 attached.

Not sure what happens if only NAT is applied. ZBF was removed for just a brief time to test. If the ZBF config is removed the network and devices are open to the Internet.

Until today I thought NAT was the issue. However, even if the NAT config is removed from the interfaces it still does not work. So now again my guess is that ZBF is the problem. Yet we cannot be completely sure that the cause is actually ZBF combined with NAT.

With the config you find in the attached files NAT and ZBF is on, and traffic from hosts behind both routers can communicate with each other.

[Rob Ingram]

@thisd00d so you are pinging through the VPN to 172.16.11.30 which is in the INSIDE zone? which is not the same as being able to manage the router over the tunnel.

So traffic should be received from VPN to INSIDE, but your logs indicate that traffic is dropped on zone-pair ZP_INTERNET_TO_self - is this a co-incidence or is the fundamental issue now that the IKEv2/IPSec SAs have not even been established and the drop is the IKE/ESP traffic being dropped?

Run "show crypto ikev2 sa" to determine if you have IKEv2 SA, run "show crypto ipsec sa" and determine if both inbound and outbound IPSec SAs exist and if the counters are increasing.

Run "show policy-map type inspect zone-pair ZP_INTERNET_TO_self" and provide the output. Amend the class-map/ACL if required.

[thisd00d]

@MHM Cisco World  and @Rob Ingram 

Below are the outputs you asked for.
At the same time port 22 has been specifically allowed on the ACLs which are permitted by the ZBF, and disabled control-plane host. With the config changes below the router is still blocking traffic attempting to communicate via SSH and SNMP (both Putty/OpenSSH and LibreNMS are not able to connect). The notification appears exactly when a connection attempt is sent from site 1.

R2#
Aug 15 11:55:00.424: %FW-6-DROP_PKT: Dropping Unknown-l4 session [2A00:zz:zz:D00D::]:0 [2B00:zz:zz:FEED::]:0 on zone-pair ZP_INTERNET_TO_self class class-default due to DROP action found in policy-map with ip ident 12856

## Changes to ACLs and control-plane
Extended IP access list ALLOW_DHCP
5 permit tcp any any eq 22 (5 matches)
10 permit udp any any eq bootps
20 permit udp any any eq bootpc (4484 matches)
30 deny ip any any (379961 matches)

IPv6 access list ALLOW_ESP
permit esp host 2A00:zz:zz:D00D:: host 2B00:zz:zz:FEED:: sequence 10
permit tcp any any eq 22 sequence 15
deny ipv6 any any (871 matches) sequence 20

line vty 0 4
no access-class SSH-ACCESS in

no control-plane host

## Crypto and ZBF outputs
R2#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA

IPv6 Crypto IKEv2 SA

Tunnel-id fvrf/ivrf Status
2 none/none READY
Local 2B00:zz:zz:FEED::/500
Remote 2A00:zz:zz:D00D::/500
Encr: AES-GCM, keysize: 256, PRF: SHA384, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/83128 sec

R2#show crypto ipsec sa

interface: Tunnel0
Crypto map tag: Tunnel0-head-0, local addr 2B00:zz:zz:FEED::

protected vrf: (none)
local ident (addr/mask/prot/port): (2B00:zz:zz:FEED::/128/47/0)
remote ident (addr/mask/prot/port): (2A00:zz:zz:D00D::/128/47/0)
current_peer 2A00:zz:zz:D00D:: port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 424532, #pkts encrypt: 424532, #pkts digest: 424532
#pkts decaps: 424283, #pkts decrypt: 424283, #pkts verify: 424283
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 2B00:zz:zz:FEED::,
remote crypto endpt.: 2A00:zz:zz:D00D::
plaintext mtu 1426, path mtu 1500, ipv6 mtu 1500, ipv6 mtu idb GigabitEthernet8.1
current outbound spi: 0xE1319D85(3778125189)
PFS (Y/N): Y, DH group: group20

inbound esp sas:
spi: 0xC95957E3(3378075619)
transform: esp-gcm 256 ,
in use settings ={Tunnel, }
conn id: 416, flow_id: Onboard VPN:416, sibling_flags 80000041, crypto map: Tunnel0-head-0
sa timing: remaining key lifetime (k/sec): (4275918/370)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0xE1319D85(3778125189)
transform: esp-gcm 256 ,
in use settings ={Tunnel, }
conn id: 415, flow_id: Onboard VPN:415, sibling_flags 80000041, crypto map: Tunnel0-head-0
sa timing: remaining key lifetime (k/sec): (4275904/370)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas:

R2#show policy-map type inspect zone-pair ZP_INTERNET_TO_self

policy exists on zp ZP_INTERNET_TO_self
Zone-pair: ZP_INTERNET_TO_self

Service-policy inspect : PM_ISAKMP

Class-map: CM_ISAKMP (match-any)
Match: protocol isakmp
484 packets, 104056 bytes
30 second rate 0 bps
Match: access-group name ALLOW_ESP
424475 packets, 104274260 bytes
30 second rate 12000 bps
Match: access-group name ALLOW_DHCP
4494 packets, 1619979 bytes
30 second rate 0 bps
Pass
429453 packets, 105998295 bytes

Class-map: CM_NON_ISAKMP (match-all)
Match: protocol ntp
Pass
1397 packets, 36758 bytes

Class-map: class-default (match-any)
Match: any
Drop
380103 packets, 44190071 bytes

Apologies the reply is late. It's not possible for me to be at site 2 often.

If it seems to not be possible, then perhaps another solution could be to forward SNMP packets from R2 to another device at site 2 which will then forward to the SNMP server at site 1.

[MHM Cisco World]

First we will divide the issue into two part 1- Zone Issue 2- NAT Issue 

first we will divide the issue into twp part 1- Zone Issue 2- NAT Issue 
for Zone  (except the DNS Zone)
INSIDE <<- config under the VLAN interface 
OUTSIDE <<- Config under Tunnel interface and Tunnel source 
SELF <<- any traffic generate from Router will by default include in this Zone, and this Zone not config under any Interface
refer the below config for inspect.

class-map type inspect match-any Internet-cmap
 match protocol icmp
 match protocol tcp
 match protocol udp
 match protocol http
 match protocol https
 match protocol pop3
 match protocol pop3s
 match protocol smtp
class-map type inspect match-all ICMP-cmap
 match access-group name ICMP
class-map type inspect match-all IPSEC-cmap
 match access-group name ISAKMP_IPSEC
class-map type inspect match-all SSHaccess-cmap
 match access-group name SSHaccess
!
!!--- Define the Zone based firewall Policy maps

policy-map type inspect inside-outside-pmap
 class type inspect Internet-cmap
  inspect
 class type inspect ICMP-cmap
  inspect
 class class-default
  drop
policy-map type inspect outside-inside-pmap
 class type inspect ICMP-cmap
  inspect
 class class-default
  drop
policy-map type inspect Outside-Router-pmap
 class type inspect SSHaccess-cmap
  inspect
 class type inspect ICMP-cmap
  inspect
 class type inspect IPSEC-cmap
  pass
 class class-default
  drop
!
!!--- Define zones

zone security inside
zone security outside
!
!!--- Define zone-pairs

zone-pair security inside-to-outside source inside destination outside
 service-policy type inspect inside-outside-pmap
zone-pair security outside-to-router source outside destination self
 service-policy type inspect Outside-Router-pmap
zone-pair security outside-to-inside source outside destination inside
 service-policy type inspect outside-inside-pmap

ip access-list extended ICMP
 permit icmp any any echo
 permit icmp any any echo-reply
 permit icmp any any traceroute
!
ip access-list extended ISAKMP_IPSEC
 permit udp any any eq isakmp
 permit ahp any any
 permit esp any any
 permit udp any any eq non500-isakmp
!
ip access-list extended SSHaccess
 permit tcp any any eq 22

[MHM Cisco World]

as I mention before let divide the issue into part, 
first the Zone Security, 
the Zone security divide in your case into
DMZ (config under interface of ZFW DMZ )
OUTSIDE (config under interface of ZFW WAN & SVTI interface)
INSIDE (config under interface of ZFW LAN)
SELF (not config under any interface)

the traffic in your case is 
1-generate from Router and point to router 

IPSec (PASS because it have one direction only)
SSH (based on TCP so it need Inspect)
ICMP (echo and echo-reply so it need inspect for return back traffic)


2-generate from Inside to Outside (inside of the other SVTI side )
ICMP
TCP


when we add protocol to match we add one by one, 
first is IPSec 
then ICMP 
and finally SSH