VPN Tunnel missing child_SAs

[zietgiestt]

Hello,

I have a vpn tunnel built between me (ASA 5506X) and another site (Palo Alto).

After some work, the tunnel came up and connected just fine.

Palo alto has 3 private subnets I have NATed to 2 of my private IPs (more needed but I'm only using these 2 for testing)

Problem is, PA can only connect to 1 of my IPs at a time. 

PA subnets=10.21.0.0/16, 10.22.0.0/16 & 172.22.0.0/16

my IPs=172.16.5.246/32 is a corp vlan,

10.201.119.6/32 is a hosted DC vlan connected via SD_WAN.

sh cry is sa= 

IKEv2 SAs:

Session-id:7196, Status:UP-ACTIVE, IKE count:1, CHILD count:1

Tunnel-id Local Remote Status Role
2691979667 12.X.X.X/500 170.X.X.X/500 READY RESPONDER
Encr: AES-CBC, keysize: 192, Hash: SHA256, DH Grp:21, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 28800/7270 sec
Child sa: local selector 172.16.5.246/0 - 172.16.5.246/65535
remote selector 10.22.0.0/0 - 10.22.255.255/65535
ESP spi in/out: 0x998bb96f/0xf41dacc6

sh cry ip sa peer=

peer address: 170.X.X.X
Crypto map tag: outside_map0, seq num: 13, local addr: 12.X.X.X

access-list WS_TUNNEL_ACL extended permit ip host 172.16.5.246 10.22.0.0 255.255.0.0
local ident (addr/mask/prot/port): (h.hqc.pc.172.16.5.246/255.255.255.255/0/0)
remote ident (addr/mask/prot/port): (10.22.0.0/255.255.0.0/0/0)
current_peer: 170.X.X.X

#pkts encaps: 8391, #pkts encrypt: 8391, #pkts digest: 8391
#pkts decaps: 8409, #pkts decrypt: 8409, #pkts verify: 8409
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 8391, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: 12.X.X.X/500, remote crypto endpt.: 170.X.X.X/500
path mtu 1500, ipsec overhead 78(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: F41DACC6
current inbound spi : 998BB96F

inbound esp sas:
spi: 0x998BB96F (2576071023)
SA State: active
transform: esp-aes-256 esp-sha-256-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 320429, crypto-map: outside_map0
sa timing: remaining key lifetime (kB/sec): (4284947/21130)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xF41DACC6 (4095585478)
SA State: active
transform: esp-aes-256 esp-sha-256-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 320429, crypto-map: outside_map0
sa timing: remaining key lifetime (kB/sec): (4331028/21130)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

NAT rule= 

nat (inside,outside) source static DM_INLINE_NETWORK_66 DM_INLINE_NETWORK_66 destination static DM_INLINE_NETWORK_67 DM_INLINE_NETWORK_67

sh run object-group id DM_INLINE_NETWORK_66
object-group network DM_INLINE_NETWORK_66
network-object object DaveB
network-object object tplebsdb201

sh run object-group id DM_INLINE_NETWORK_67
object-group network DM_INLINE_NETWORK_67
network-object object 10.21.0.0
network-object object 10.22.0.0
network-object object 172.22.0.0

ACL=

access-list WS_TUNNEL_ACL extended permit object-group DM_INLINE_PROTOCOL_9 object 10.22.0.0 object DaveB
access-list WS_TUNNEL_ACL extended permit object-group DM_INLINE_PROTOCOL_10 object DaveB object 10.22.0.0
access-list WS_TUNNEL_ACL extended permit object-group DM_INLINE_SERVICE_20 object tplebsdb201 object 10.22.0.0
access-list WS_TUNNEL_ACL extended permit object-group DM_INLINE_SERVICE_21 object 10.22.0.0 object tplebsdb201

When PA pings 172.16.5.246, I am able to ping his IP (10.22.48.21)

When PA pings 10.201.119.6, I am not able to ping his IP (10.22.48.21)

At the time of writing this, PA has a ping -t running to both of my IPs, but only 172.16.5.246 is responding, 10.201.119.6 is timing out

I may be missing something very basic, but I'm stuck here and would really appreciate any help or guidance.

Thanks,
D

[ccieexpert]

It is possible the crypto ACL is not matching, which is the identities between ASA and PALO.

can you get the "show access-list <crypto map acl>" out

Please run the debugs so we can see why it is failing to establish the other SAs:

https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/115935-asa-ikev2-debugs.html

[MHM Cisco World]

Can you share config of ASA 

MHM

[zietgiestt]

@ccieexpert  attached debugs for protocol and platform & ACL

[zietgiestt]

@MHM Cisco World The object groups are defined as the same as in the acl:

sh run object-group id DM_INLINE_NETWORK_66
object-group network DM_INLINE_NETWORK_66
network-object object DaveB
network-object object tplebsdb201

sh run object-group id DM_INLINE_NETWORK_67
object-group network DM_INLINE_NETWORK_67
network-object object 10.21.0.0
network-object object 10.22.0.0
network-object object 172.22.0.0

ACL=

access-list WS_TUNNEL_ACL extended permit object-group DM_INLINE_PROTOCOL_9 object 10.22.0.0 object DaveB
access-list WS_TUNNEL_ACL extended permit object-group DM_INLINE_PROTOCOL_10 object DaveB object 10.22.0.0
access-list WS_TUNNEL_ACL extended permit object-group DM_INLINE_SERVICE_20 object tplebsdb201 object 10.22.0.0
access-list WS_TUNNEL_ACL extended permit object-group DM_INLINE_SERVICE_21 object 10.22.0.0 object tplebsdb201

The PA's alc is matching mine for the moment, We will add the 10.21.0.0 & 172.22.0.0 networks later after we can confirm all networks will pass traffic together.

Does the NAT statement and ACL need to match before more than 1 network will will negotiate a child_sa?

Should I re-write my ANT statement using identities rather than objects? 

[MHM Cisco World]

I know what issue now 
you use two ACL for both direction where you need only one ACL line for one direction between two peer 
i.e.
LAN1 connect to ASA 
LAN2 connect to palo 
the ACL will be 
permit LAN1 to LAN2 <<- ONLY there is no need and it make issue if you add permit LAN2 to LAN1 

Note:- In palo use mirror not same 

MHM

[zietgiestt]

I see what you're saying, but I need to allow the PA access to specific IPs. some IPs are in my corp vlan (72.16.5.0), some in a hosted DC (10.202.116.0) and some in s hosted dev DC (10.201.119.0). 

to match what you describe, I will need LAN1, 2, 3 & 4.

Each would require it's own  ACL line.

thinking about this more...if I need the PA to access my networks, should I just have ACLs for:

permit palo to asa

[MHM Cisco World]

Only list the IP in Palo and Asa here And I will share acl and no-NAT config

MHM

[ccieexpert]

Its very simple   your network---ASA--internet---Palo

everything behind your ASA is part of the ACL and part of source subnet

anything behind the PALO is part of the destination subnet in the ACL

no need to reverse them again in the ACL.. always source is your networks behind ASA

[ccieexpert · Accepted solution]

the reverse direction will be just ignored as it doesnt match anything.. it is extraneous and should not have any impact other than making the acl bigger and taking up resources.

The real problem seems to be the DH group:

IKEv2-PROTO-4: Selected IKEv2 encryption algorithm (AES-CBC-192) is not strong enough to secure proposed IPsec encryption algorithm (AES-CBC-256).
IKEv2-PROTO-2: (611): The peer's KE payload contained the wrong DH group
IKEv2-PROTO-7: (611): SM Trace-> SA: I_SPI=E9943FE1D67A4218 R_SPI=240FEB0816D01C68 (R) MsgID = 0000C3C9 CurState: CHILD_R_IPSEC Event: EV_INV_KE
IKEv2-PROTO-4: (611): Sending invalid ke notification, peer sent group 21, local policy prefers group 0

Please make sure your DH group is 21 so that it matches the PALO.

crypto map mymap 10 set pfs group21

Please rerun debugs again if there are issue.

**please rate as helpful if this is useful**

[zietgiestt]

@ccieexpert that was it! I didn't think the tunnel would negotiate with a mismatched group. and with a mismatched group, how would 1 network work but not the other? Guess I have some reading to do.

Thanks CCIEEXPERT for the help.

[MHM Cisco World]

Check below

MHM

[ccieexpert]

The jist of this is that there is a different between who is the initiator vs responder of the SA.

That is why  you could get different results depending on which side initiates, that is why one SA comes up as it may be initiated from the other side (different from non working). So as a best practice, always keep the PFS group the same...

Read this and i will try to shed more light later from the RFC:

https://community.cisco.com/t5/network-security/ipsec-tunnel-pfs-groups-need-to-match/td-p/3914964

Cheers to MHM as well

[MHM Cisco World]

Hi 
No need to waste your time anymore 
I do some check there is bug no. CSCtz64224 
this bug mention that only first SA will build and other SA not build if there is pfs group mismatch or there is missing in one side

So you are right I was wrong but I learn something new here 
Cheers 

MHM

[zietgiestt]

I appreciate all the help.

After I matched the group to 21 in the crypto map, I instantly saw both child SAs associate and was told by the other side pings stated to both IPs.

I have not made a change to my ACL yet.

The only thing I'm having a problem with is allowing the PA network to ssh my servers on the 10.20.119.X network. 

He gets a timeout. 

attached are debugs. tunnel in question is 170.x.x.x