Re: ISR 931 IPSEC Tunnel issue - Page 2

SargenYung06306 · ‎01-10-2020

I bought cisco ISR 931 router recently and struggled to setup 2 IPSEC tunnels. The router is put as edge router that directly faces internet with one fixed IP and connects to 2 remote sites with 2 fixed IPs for the 2 IPSEC tunnels.

To begin with, I create 2 vlans with vlan interfaces as below.

interface Vlan1
ip address 10.249.102.193 255.255.255.192
ip pim sparse-mode
ip nat inside
ip virtual-reassembly in
!
interface Vlan2
ip address 172.28.207.177 255.255.255.240
ip pim sparse-mode
ip nat inside
ip virtual-reassembly in

Then, the Gigabit WAN interface facing internet with fixed IP (x.x.x.x).

interface GigabitEthernet4
ip address x.x.x.x 255.255.255.252
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto

Then, NAT 2 vlans to access internet,

access-list 10 permit 10.249.102.192 0.0.0.63
access-list 20 permit 172.28.207.176 0.0.0.15
!
ip nat pool INTERNET x.x.x.x x.x.x.x netmask 255.255.255.252
ip nat inside source list 10 pool INTERNET overload
ip nat inside source list 20 pool INTERNET overload

Then, define 2 crypto isakmp policies.

crypto isakmp policy 1
encr 3des
authentication pre-share
!
crypto isakmp policy 2
encr 3des
hash md5
authentication pre-share

Then, define the pre-shared keys.

crypto isakmp key SECRET_KEY address y.y.y.y
crypto isakmp key SECRET_KEY address z.z.z.z

Then, define ipsec transform-set.

crypto ipsec transform-set setvpn esp-3des esp-sha-hmac
mode tunnel
crypto ipsec transform-set cmevpn esp-3des esp-md5-hmac
mode tunnel

Then, define the access list for IPSEC tunnels.

access-list 100 permit ip 10.249.102.192 0.0.0.63 69.50.112.0 0.0.0.255
access-list 110 permit ip 172.28.207.176 0.0.0.15 172.19.214.0 0.0.0.255
access-list 110 permit ip 172.28.207.176 0.0.0.15 172.19.215.0 0.0.0.255
access-list 110 permit ip 172.28.207.176 0.0.0.15 10.3.14.0 0.0.0.255
access-list 110 permit ip 172.28.207.176 0.0.0.15 10.3.27.0 0.0.0.255
access-list 110 permit ip 172.28.207.176 0.0.0.15 172.19.7.0 0.0.0.255
access-list 110 permit ip 172.28.207.176 0.0.0.15 10.22.6.0 0.0.0.255
access-list 110 permit ip 172.28.207.176 0.0.0.15 10.22.61.0 0.0.0.255
access-list 110 permit ip 172.28.207.176 0.0.0.15 10.232.6.0 0.0.0.255

Then, define the cryto map with 2 indices.

crypto map CRYTOMAP 1 ipsec-isakmp
set peer y.y.y.y
set transform-set setvpn
match address 110
crypto map CRYTOMAP 2 ipsec-isakmp
set peer z.z.z.z
set transform-set cmevpn
match address 100

Then, apply it to the WAN interface.

interface GigabitEthernet4
ip address x.x.x.x 255.255.255.252
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
crypto map CRYTOMAP

After it is applied, the 1st IPSEC phase 1 is not initiated and the 2nd IPSEC tunnel is UP-IDLE. With debug crypto ipsec on, I can see the following error,

*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: IPSEC: Expand action denied, notify RP
*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: IPSEC: Expand action denied, notify RP
*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: IPSEC: Expand action denied, discard or forward packet.
*Jan 10 14:40:57.347: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON

Would someone help give me some hints on what might be going on and how to fix it? what "Expand action denied, discard or forward packet" means? Thank you!

SargenYung06306 · ‎01-14-2020

@Georg Pauwen Thank for your suggestions! I reset the router to the factory setting and start it over again. This time I leave only 1 VLAN (vlan 1) of network (172.28.207.177 255.255.255.240) and only 1 IPSEC tunnel (the one not working). The issue still there with the same error message. Please find the new show run as below. Thank you vm!

version 15.8
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Router
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
!
!
!
!
!
!
!
ip dhcp excluded-address 172.28.207.177 172.28.207.179
!
ip dhcp pool dhcp_tfex_pool
network 172.28.207.176 255.255.255.240
default-router 172.28.207.177
dns-server 203.80.96.33
!
ip dhcp pool iDRAC
host 172.28.207.178 255.255.255.240
client-identifier 0144.a842.3405.12
default-router 172.28.207.177
dns-server 203.80.96.33
!
ip dhcp pool YASSVR
host 172.28.207.179 255.255.255.240
hardware-address 44a8.4234.0510
default-router 172.28.207.177
dns-server 203.80.96.33
!
!
!
ip name-server 203.80.96.33
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
license udi pid C931-4P sn PSZ23091D50
license accept end user agreement
license boot module c900 technology-package securityk9
!
!
!
redundancy
!
!
!
!
!
!
!
crypto isakmp policy 1
encr 3des
hash sha
authentication pre-share
group 2
crypto isakmp key C@fbrk1F address y.y.y.y
!
!
crypto ipsec transform-set setvpn esp-3des esp-sha-hmac
mode tunnel
!
!
!
crypto map CRYTOMAP 1 ipsec-isakmp
set peer y.y.y.y
set transform-set setvpn
match address 100
!
!
!
!
!
interface GigabitEthernet0
no ip address
!
interface GigabitEthernet1
no ip address
!
interface GigabitEthernet2
no ip address
!
interface GigabitEthernet3
no ip address
!
interface GigabitEthernet4
ip address x.x.x.x 255.255.255.252
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
crypto map CRYTOMAP
!
interface GigabitEthernet5
no ip address
shutdown
duplex auto
speed auto
!
interface Vlan1
ip address 172.28.207.177 255.255.255.240
ip nat inside
ip virtual-reassembly in
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip nat inside source list 101 interface GigabitEthernet4 overload
ip nat inside source static tcp 172.28.207.178 777 interface GigabitEthernet4 777
ip nat inside source static tcp 172.28.207.178 888 interface GigabitEthernet4 888
ip nat inside source static tcp 172.28.207.179 22 interface GigabitEthernet4 22
ip nat inside source static tcp 172.28.207.179 443 interface GigabitEthernet4 443
ip route 0.0.0.0 0.0.0.0 210.6.226.253
!
!
!
access-list 100 permit ip 172.28.207.176 0.0.0.15 10.232.6.0 0.0.0.255
access-list 100 permit ip 172.28.207.176 0.0.0.15 172.19.214.0 0.0.0.255
access-list 100 permit ip 172.28.207.176 0.0.0.15 172.19.215.0 0.0.0.255
access-list 100 permit ip 172.28.207.176 0.0.0.15 10.3.14.0 0.0.0.255
access-list 100 permit ip 172.28.207.176 0.0.0.15 10.3.27.0 0.0.0.255
access-list 100 permit ip 172.28.207.176 0.0.0.15 172.19.7.0 0.0.0.255
access-list 100 permit ip 172.28.207.176 0.0.0.15 10.22.6.0 0.0.0.255
access-list 100 permit ip 172.28.207.176 0.0.0.15 10.22.61.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 172.19.214.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 172.19.215.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.3.14.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.3.27.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 172.19.7.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.22.6.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.22.61.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.232.6.0 0.0.0.255
access-list 101 permit ip 172.28.207.176 0.0.0.15 any
!
control-plane
!
!
line con 0
line vty 0 4
login
transport input none
!
scheduler allocate 20000 1000
!
end

Georg Pauwen · ‎01-14-2020

Hello,

try and set the pfs group in the crypto map:

crypto map CRYTOMAP 1 ipsec-isakmp
set peer y.y.y.y
set transform-set setvpn

set pfs group2
match address 100

Also, post the output of:

debug crypto isakmp

debug crypto ipsec

I still think there is an encryption mismatch. Without seeing the configuration of the other side, it will be guessing what the right settings are...

SargenYung06306 · ‎01-14-2020

@Georg Pauwenthanks again!

try with your new suggestion, still the same. the debug log also the same

*Jan 14 10:54:56.825: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 10:54:56.825: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 10:54:56.825: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 10:54:56.825: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 10:54:56.825: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 10:54:56.825: IPSEC: Expand action denied, notify RP
*Jan 14 10:54:56.825: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 10:54:56.827: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 10:54:56.827: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON

The remote side told us the IKE policy is

Encryption: 3DES

Authentication: SHA

DH Group:2

Any ideas?

SargenYung06306 · ‎01-14-2020

Also by looking at

Router#sh cry ipsec sa

interface: GigabitEthernet4
Crypto map tag: MYCRYPTOMAP, local addr x.x.x.x

protected vrf: (none)
local ident (addr/mask/prot/port): (172.28.207.176/255.255.255.240/0/0)
remote ident (addr/mask/prot/port): (10.232.6.0/255.255.255.0/0/0)
current_peer y.y.y.y port 500
PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 25, #recv errors 0

local crypto endpt.: x.x.x.x, remote crypto endpt.: y.y.y.y

plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet4
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none

inbound esp sas:

inbound ah sas:

inbound pcp sas:

outbound esp sas:

outbound ah sas:

outbound pcp sas:

Georg Pauwen · ‎01-14-2020

The other side needs to mirror the access list you have.

Post the output of:

debug crypto ipsec

debug crypto isakmp

SargenYung06306 · ‎01-14-2020

@Georg Pauwen

debug crypto ipsec output:

*Jan 14 15:36:13.302: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 15:36:13.302: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 15:36:13.302: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 15:36:13.302: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 15:36:13.302: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 15:36:13.302: IPSEC: Expand action denied, notify RP
*Jan 14 15:36:13.302: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 15:36:13.302: IPSEC: Expand action denied, discard or forward packet.
*Jan 14 15:36:13.304: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON

debug crypto isakmp output:

no debug log displayed.

SargenYung06306 · ‎01-29-2020

@Georg PauwenJust want to let you know the IPSEC issue is no longer an issue. It is strange that although the tunnel status showing down but when the 1st ping go through the tunnel, it shows ACTIVE_UP. Thank u very much for the guidance, very appreciated!

Now I have another issue. As you know, I have 2 vlans, i.e, vlan 1 (default, network 172.28.207.176 255.255.255.240, vlan interface 172.28.207.177) and vlan 2 (network 10.249.102.192 255.255.255.192, vlan interface 10.249.102.193), a host with 2 interface cards connecting to 2 switch ports of the router for which one is assigned to vlan 1 and another is assigned to vlan 2 respectively. when I try to ping internet from the host via interface card attached to vlan 1 works fine but not working via interface card attached to vlan 2.

interface em1: 172.28.207.179, default gateway 172..28.207.177

#ping -I em1 google.com

interface em2: 10.249.102.194, default gateway 10.249.102.193

#ping -I em1 google.com

To better track the issue, I have remove the IPSEC related configurations, please find the show run as below.

version 15.8
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Router
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
!
!
!
!
!
!
!
ip dhcp excluded-address 172.28.207.177 172.28.207.179
!
ip dhcp pool dhcp_tfex_pool
network 172.28.207.176 255.255.255.240
default-router 172.28.207.177
dns-server 203.80.96.33
!
ip dhcp pool iDRAC
host 172.28.207.178 255.255.255.240
client-identifier 0144.a842.3405.12
default-router 172.28.207.177
dns-server 203.80.96.33
!
ip dhcp pool YASSVR
host 172.28.207.179 255.255.255.240
hardware-address 44a8.4234.0510
default-router 172.28.207.177
dns-server 203.80.96.33
!
ip dhcp pool dhcp_cme_pool
network 10.249.102.192 255.255.255.192
dns-server 203.80.96.33
default-router 10.249.102.193
!
!
!
ip name-server 203.80.96.33
ip multicast-routing
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
license udi pid C931-4P sn PSZ23091D50
license accept end user agreement
license boot module c900 technology-package securityk9
!
!
!
redundancy
!
!
!
interface GigabitEthernet0
no ip address
!
interface GigabitEthernet1
no ip address
!
interface GigabitEthernet2
switchport access vlan 2
switchport mode access
no ip address
spanning-tree portfast
!
interface GigabitEthernet3
no ip address
!
interface GigabitEthernet4
ip address x.x.x.x 255.255.255.252
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
!
interface GigabitEthernet5
no ip address
shutdown
duplex auto
speed auto
!
interface Vlan1
ip address 172.28.207.177 255.255.255.240
ip nat inside
ip virtual-reassembly in
!
interface Vlan2
ip address 10.249.102.193 255.255.255.192
ip nat inside
ip virtual-reassembly in
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip nat inside source list 101 interface GigabitEthernet4 overload
ip nat inside source list 111 interface GigabitEthernet4 overload
ip route 0.0.0.0 0.0.0.0 y.y.y.y
!
!
!
access-list 101 deny ip 172.28.207.176 0.0.0.15 172.19.214.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 172.19.215.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.3.14.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.3.27.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 172.19.7.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.22.6.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.22.61.0 0.0.0.255
access-list 101 deny ip 172.28.207.176 0.0.0.15 10.232.6.0 0.0.0.255
access-list 101 permit ip 172.28.207.176 0.0.0.15 any
access-list 111 deny ip 10.249.102.192 0.0.0.63 69.50.112.0 0.0.0.255
access-list 111 permit ip 10.249.102.192 0.0.0.63 any
!
control-plane
!
!
line con 0
line vty 0 4
login
transport input none
!
scheduler allocate 20000 1000
!
end

Thanks again!

SargenYung06306 · ‎01-29-2020

@Georg Pauwenthe issue has been fixed by adding a routing in the host. please ignore this post. thanks!

Georg Pauwen · ‎01-29-2020

Glad to hear that it is resolved...

SargenYung06306 · ‎01-29-2020

@Georg Pauwenyou are such a great mentor!