ISR IPsec site-to-site VPN - cannot manage routers through tunnel

thisd00d · ‎08-01-2022

Problem: the routers at either end cannot be managed/reached (SSH, SNMP, ICMP) via the tunnel.

I have two routers located at different locations. An IPsec tunnel via the Internet is configured and working. The tunnel itself is established with IPv6 and the traffic flowing through the tunnel is IPv4. The two Cisco ISRs are both configured with ZBF - I assume this is where the problem is.

Monitoring R1 with SNMP at site A works because the SNMP server resides on this network and site. I can also monitor SNMP devices behind R2 at site B via the tunnel. However, any traffic to/from the router itself at site B is dropped.

The same is true if I attempt to access or ping the router at site A from site B - traffic is dropped and I can only reach devices behind the routers.

What confuses me is that traffic to and from the self zone is permitted by default. In that case where and why is the traffic dropped when it is transmitted via the tunnel?

The full config is 500 lines, hence just the relevant parts including ZBF and crypto of show run for R2 is inserted below (I can of course share more of the config and also for R1 too if necessary, both sites are similar).

The network 172.16.11.0/24 is the INSIDE zone on R1 and is missing in the ACL "SSH-ACCESS" for now. However, should it not be possible to ping and SNMP to the self zone?

R2#show run
Building configuration...
!
crypto ikev2 proposal IKEV2_PROP2
encryption aes-gcm-256
prf sha384
group 20
no crypto ikev2 proposal default
!
crypto ikev2 policy IKEV2_POLICY2
proposal IKEV2_PROP2
no crypto ikev2 policy default
!
!
crypto ikev2 profile IKEV2_PROF2
match identity remote address 2100:zz:zz:D00D::/128
authentication remote pre-share key key1
authentication local pre-share key key2
!
!
class-map type inspect match-any CM_VPN_TRAFFIC
match protocol dns
match protocol snmp
match protocol icmp
match protocol http
match protocol https
match protocol syslog
match protocol ssh
match protocol snmptrap
class-map type inspect match-all CM_NON_ISAKMP
match protocol ntp
class-map type inspect match-any CM_INSIDE_PROTOCOLS
match protocol tcp
match protocol udp
match protocol icmp
class-map type inspect match-any CM_ISAKMP
match protocol isakmp
match access-group name ALLOW_ESP
match access-group name ALLOW_DHCP
!
policy-map type inspect PM_ISAKMP
class type inspect CM_ISAKMP
pass
class type inspect CM_NON_ISAKMP
pass
class class-default
drop
policy-map type inspect PM_VPN_TRAFFIC
class type inspect CM_VPN_TRAFFIC
inspect
class class-default
drop
policy-map type inspect PM_INSIDE_TO_INTERNET
class type inspect CM_INSIDE_PROTOCOLS
inspect
class class-default
drop
!
zone security INSIDE
zone security INTERNET
zone security VPN
zone security IOT
zone-pair security ZP_INSIDE_TO_INTERNET source INSIDE destination INTERNET
service-policy type inspect PM_INSIDE_TO_INTERNET
zone-pair security ZP_INSIDE_TO_VPN source INSIDE destination VPN
service-policy type inspect PM_VPN_TRAFFIC
zone-pair security ZP_VPN_TO_INSIDE source VPN destination INSIDE
service-policy type inspect PM_VPN_TRAFFIC
zone-pair security ZP_INTERNET_TO_self source INTERNET destination self
service-policy type inspect PM_ISAKMP
zone-pair security ZP_IOT_TO_INTERNET source IOT destination INTERNET
service-policy type inspect PM_INSIDE_TO_INTERNET
!
no crypto isakmp default policy
!
!
no crypto ipsec transform-set default
crypto ipsec transform-set TRANS2 esp-gcm 256
mode tunnel
!
crypto ipsec profile IPSEC_PROF2
set transform-set TRANS2
set pfs group20
set ikev2-profile IKEV2_PROF2
!
!
!
!
!
interface Tunnel0
ip address 192.168.9.2 255.255.255.0
ip mtu 1362
zone-member security VPN
ip tcp adjust-mss 1280
ipv6 address FC09::2/64
ipv6 enable
tunnel source 2200:zz:zz:FEED::
tunnel mode gre ipv6
tunnel destination 2100:zz:zz:D00D::
tunnel protection ipsec profile IPSEC_PROF2
!
!
ip route 172.16.11.0 255.255.255.0 Tunnel0 192.168.9.1
ip route 172.31.255.254 255.255.255.255 Tunnel0 192.168.9.1
!
!
ip access-list extended ALLOW_DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
deny ip any any
ip access-list extended SSH-ACCESS
permit tcp 172.16.12.0 0.0.0.255 any eq 22
deny ip any any

ip access-list standard SNMPv3-ACL
permit 172.16.11.0 0.0.0.255
deny any
!
!
!
ipv6 access-list ALLOW_ESP
permit esp host 2100:zz:zz:D00D:: host 2200:zz:zz:FEED::
deny ipv6 any any

!
control-plane host
management-interface Tunnel0 allow ssh snmp
management-interface Vlan12 allow ssh snmp
!
!
control-plane
!
line con 0
logging synchronous
login local
line vty 0 4
access-class SSH-ACCESS in
login local
transport input ssh
!
end

Rob Ingram · ‎08-01-2022

@thisd00d

What confuses me is that traffic to and from the self zone is permitted by default. In that case where and why is the traffic dropped when it is transmitted via the tunnel?

That is true, until you configure self in a zone pair (which you have - ZP_INTERNET_TO_self), then you explictly need to permit traffic. So you will need to configure additional zone-pairs to be the required traffic.

thisd00d · ‎08-01-2022

Thank you for your answer.

I had already attempted to create a zone pair from VPN to self allowing all traffic - and this still did not work. Is it another zone pair that's needed?

On purpose I did not attempt to add SSH and SNMP to the zone pair ZP_INTERNET_TO_self because the traffic is expected to arrive at the router as ESP traffic then go via the VPN zone, correct?

Rob Ingram · ‎08-01-2022

@thisd00d yes the ZP_INTERNET_TO_self is for the encrypted traffic, you'd need another zone pair. When you run the ping test over the VPN tunnel, ensure you source the ping from the correct interface - as you only have a static route for a specific route and an ACL on the VTY lines. Mixing VTY ACL and ZBFW is not recommended, stick with ZBFW and remove the VTY ACL.

thisd00d · ‎08-01-2022

Sitting with R1 right now and added the following lines:

policy-map type inspect PM_VPN_TO_self
class type inspect CM_INSIDE_PROTOCOLS
pass
class class-default
drop

zone-pair security ZP_VPN_TO_self source VPN destination self
service-policy type inspect PM_VPN_TO_self
zone-pair security ZP_self_TO_VPN source self destination VPN
service-policy type inspect PM_VPN_TO_self

After this the traffic is still not reaching the self zone. From R1 attempting to reach a device behind R2 with source interfaces in both the INSIDE and VPN zone:

R1#ping 172.16.12.41 source 172.16.11.1 (interface in INSIDE zone)
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.41, timeout is 2 seconds:
Packet sent with a source address of 172.16.11.1
.....
Success rate is 0 percent (0/5)

R1#ping 172.16.12.41 source 192.168.9.1 (interface in VPN zone)
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.41, timeout is 2 seconds:
Packet sent with a source address of 192.168.9.1
.....
Success rate is 0 percent (0/5)

Edit: just adding that I know this device is reachable via the tunnel as the ICMP replies are received from another source device behind R1

Rob Ingram · ‎08-01-2022

@thisd00d Is "through" the VPN working ok? Is traffic encrypted and decrypted on both routers? run "show crypto ipsec sa" and provide the output.

Can you enable ZBFW logging, test by running the ping test again and provide the logging information, that will provide a clue.

thisd00d · ‎08-01-2022

Through the tunnel is working, yes. Right now the tunnel is actively being used for monitoring and ICMP is also going all the way to 172.16.12.41 (at R2 site) from another host behind R1.

R1#sh crypto ipsec sa

interface: Tunnel0
Crypto map tag: Tunnel0-head-0, local addr 2100:zz:zz:D00D::

protected vrf: (none)
local ident (addr/mask/prot/port): (2100:zz:zz:D00D::/128/47/0)
remote ident (addr/mask/prot/port): (2200:zz:zz:FEED::/128/47/0)
current_peer 2200:zz:zz:FEED:: port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 5743468, #pkts encrypt: 5743468, #pkts digest: 5743468
#pkts decaps: 5763458, #pkts decrypt: 5763458, #pkts verify: 5763458
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

................... rest omitted

Added the "log" keyword to the new policy map

policy-map type inspect PM_VPN_TO_self
class type inspect CM_INSIDE_PROTOCOLS
pass log
class class-default
drop log

ICMP from R1

R1#ping 172.16.12.41 source vlan 11
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.41, timeout is 2 seconds:
Packet sent with a source address of 172.16.11.1
...
Aug 1 19:02:31.409: %FW-6-PASS_PKT: (target:class)-(ZP_self_TO_VPN:CM_INSIDE_PROTOCOLS) Passing icmp pkt 172.16.11.1:0 => 172.16.12.41:0 with ip ident 992 ..
Success rate is 0 percent (0/5)

R1#ping 172.16.12.41 source 192.168.9.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.41, timeout is 2 seconds:
Packet sent with a source address of 192.168.9.1

Aug 1 19:13:58.678: %FW-6-PASS_PKT: (target:class)-(ZP_self_TO_VPN:CM_INSIDE_PROTOCOLS) Passing icmp pkt 192.168.9.1:0 => 172.16.12.41:0 with ip ident 994 .....
Success rate is 0 percent (0/5)

So the firewall is matching and passing the traffic. The log did not match any return traffic. I am considering if it is being blocked by the zone pair ZP_INTERNET_TO_self. However, that should not be the case if it arrives as ESP and from 2200:zz:zz:FEED::

MHM Cisco World · ‎08-01-2022

you use ICMP for test BUT the return traffic to VPN zone is not inspect/ allow in that Zone.
config ACL for ICMP and ICMP reply in VPN Zone and try again.

Rob Ingram · ‎08-01-2022

@thisd00d if the traffic is going over the VPN tunnel it would be encapsulated inside ESP. So on the other router it would not hit the zone pair ZP_INTERNET_TO_self. You'd need another zone-pair from VPN to self.

thisd00d · ‎08-02-2022

@MHM Cisco Worldand @Rob Ingram

The current config is now this - notice the class-map is changed to "CM_VPN_TRAFFIC" and the line "match protocol icmp" has been removed from the class-map because the traffic was otherwise still matching this:

ip access-list extended ACL_VPN_self
10 permit icmp 172.16.11.0 0.0.0.255 any
20 permit icmp host 192.168.9.1 any
21 permit icmp host 192.168.9.2 any
30 permit icmp 172.16.12.0 0.0.0.255 any
90 permit ip any any

class-map type inspect match-any CM_VPN_TRAFFIC
match protocol dns
match protocol snmp
match protocol http
match protocol https
match protocol syslog
match protocol ssh
match protocol snmptrap
match access-group name ACL_VPN_self

policy-map type inspect PM_VPN_TO_self
class type inspect CM_VPN_TRAFFIC
pass log
class class-default
drop log

zone-pair security ZP_VPN_TO_self source VPN destination self
service-policy type inspect PM_VPN_TO_self
zone-pair security ZP_self_TO_VPN source self destination VPN
service-policy type inspect PM_VPN_TO_self

_________________

The results of the ping test is again the same as the last post: 0/5 replies received for both INSIDE and VPN as source

Log from source 172.16.11.1 (INSIDE): Aug 2 18:03:40.710: %FW-6-PASS_PKT: (target:class)-(ZP_self_TO_VPN:CM_VPN_TRAFFIC) Passing icmp pkt 172.16.11.1:0 => 172.16.12.41:0 with ip ident 1037 .....
Success rate is 0 percent (0/5)

This time there is no log message showing when pinging from 192.168.9.1 (VPN zone)

ZBF stats for the two zone pairs above:

R1#show policy-firewall stats zone-pair ZP_self_TO_VPN

Class-map: CM_VPN_TRAFFIC (match-any)
..........match statements omitted
Match: access-group name ACL_VPN_self
10 packets, 800 bytes
30 second rate 0 bps
Pass
10 packets, 800 bytes

Class-map: class-default (match-any)
Match: any
Drop
0 packets, 0 bytes

R1#show policy-firewall stats zone-pair ZP_VPN_TO_self

Class-map: CM_VPN_TRAFFIC (match-any)
..........match statements omitted
Match: access-group name ACL_VPN_self
10 packets, 800 bytes
30 second rate 0 bps
Pass
10 packets, 800 bytes

Class-map: class-default (match-any)
Match: any
Drop
0 packets, 0 bytes

R1#show ip access-list

Extended IP access list ACL_VPN_self
10 permit icmp 172.16.11.0 0.0.0.255 any (8 matches)
20 permit icmp host 192.168.9.1 any (5 matches)
21 permit icmp host 192.168.9.2 any
30 permit icmp 172.16.12.0 0.0.0.255 any (10 matches)
90 permit ip any any

Edit: the fact that entry 30 has 10 matches means that all the echo-replies reached R1. Then why are the ping attempts showing success rate of 0/5?

MHM Cisco World · ‎08-03-2022

Friend I found the exact example for you

https://www.cisco.com/c/en/us/support/docs/routers/3800-series-integrated-services-routers/112051-zbf-vpn-access-config.html

IDEA HERE
there are three Zone
INSIDE OUTSIDE and SELF
INSIDE apply to LAN
OUTSIDE apply to VTI and Interface source of VTI
SELF no need to assign

FROM traffic from ONE Side to other Side is two way what that mean ?
the initiate from Side1 one to Side2 need inspect
if the Side2 initiate to Side1 then you need ACL to allow this traffic.

so in example you see
inspection for traffic initiate from Side1 to Side2
and ACL allow if the traffic initiate from Side2 to Side1

that ALL
try this config and see result.

thisd00d · ‎08-03-2022

In the last attempt there was already the ACL_VPN_self set to permit the return traffic. This ACL was applied to the class-map CM_VPN_TRAFFIC which then was linked with policy map PM_VPN_TRAFFIC going to both zone VPN and self. I would say it should have been working with that config.

However, today I just found this whitepaper Troubleshoot IOS Zone Based Policy Firewall Inspection Issues When NAT NVI Is Configured.

From the whitepaper there is a link to a bug that mention the solution: "Remove NAT-Virtual Interface configuration and apply Inside/Outside NAT." - this is already the current configuration though.

There is also a related link to this whitepaper NAT Virtual Interface which states: "To configure an interface connecting VPNs and the Internet for Network Address Translation (NAT), use the ip nat enable command" (page 8).

The way I read it does not sound fully consistent. Since it is a known issue/bug I will try both NAT methods.

It is not possible for me to test right now, it will likely be possible this weekend and I will let you know about the result. Thank you for your research and suggestion.

thisd00d · ‎08-07-2022

@MHM Cisco World and @Rob Ingram

Today I was at site 2 attempting different configurations on R2, still trying to reach the monitoring tool at site 1.

It is possible to reach the site 1 monitoring tool from R2 via the tunnel by completely removing NAT and ZBF configurations from the interfaces.

Removed the configuration below and was able to communicate with device at site 1:
no ip nat inside source list NAT-TRAFFIC interface GigabitEthernet8.1 overload

interface Loopback0
no ip nat inside
no zone-member security INSIDE

interface Vlan12
no ip nat inside
no zone-member security INSIDE

interface GigabitEthernet8.1
no ip nat outside
no zone-member security INTERNET
________________________________________________________________________

Next the zones were assigned the interfaces again and "ip nat enable" (rather than inside/outside. Also below command without the "inside" keyword:
ip nat source list NAT-TRAFFIC interface GigabitEthernet8.1 overload

Afterwards it was again not possible to reach site 1 from R2.

The interface NAT configuration is now back to inside/outside as per Cisco recommendation. NAT did not work properly with the NVI method anyway. The router was attempting to do NAT64 for a reason I don't know.

________________________________________________________________________

Next I set the log keyword on the policy map which applies to zone pair "ZP_INTERNET_TO_self".
Both with and without NAT configuration active it was the same output.

R2(config)#policy-map type inspect PM_ISAKMP
R2(config-pmap)#class class-default
R2(config-pmap-c)#drop log

R2#ping 172.16.11.30 source tun0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.11.30, timeout is 2 seconds:
Packet sent with a source address of 192.168.9.2

Aug 7 15:02:24.419: %FW-6-DROP_PKT: Dropping Unknown-l4 session [2A00:zz:zz:D00D::]:0 [2B00:zz:zz:FEED::]:0 on zone-pair ZP_INTERNET_TO_self class class-default due to DROP action found in policy-map with ip ident 3428 .....
Success rate is 0 percent (0/5)

R2#ping 172.16.11.30 source vlan 12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.11.30, timeout is 2 seconds:
Packet sent with a source address of 172.16.12.1

Aug 7 14:23:25.891: %FW-6-DROP_PKT: Dropping Unknown-l4 session [2A00:zz:zz:D00D::]:0 [2B00:zz:zz:FEED::]:0 on zone-pair ZP_INTERNET_TO_self class class-default due to DROP action found in policy-map with ip ident 5072 .....
Success rate is 0 percent (0/5)

The echo replies are blocked by "ZP_INTERNET_TO_self" so it seems the ESP packets are being blocked before they are decrypted.
I also tried to allow ICMP for the zone pair above. Same result.

Any ideas?

Rob Ingram · ‎08-07-2022

@thisd00d what zone is applied to the tunnel interface now?

What happens when you just re-apply the NAT configuration and not the ZBFW configuration, does it then work? Is NAT the issue here, with traffic unintentially translated....which causes a problem with your ZBFW rules?

Does transit traffic (host behind both routers) with NAT and ZBFW configuration applied work?

Please provide the full configurations so we can get a better understanding on what is currently configured, please provide these as separate attachments.

MHM Cisco World · ‎08-07-2022

just share the last config I will make deep lock and make change if need