Site to Site VPN - PA to FPR1120 - Can't get tunnel to come up

[matthewkylestephens]

Tunnel went down all of a sudden and now we can't get the thing to come back up. We've rebuilt several times, swapped out PSKs, just about everything we can think of. Hoping maybe someone has some insight.

Here are debugs on the FPR side (responder)

Message #488 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_SVC_TMO
Message #489 : IKEv2-PROTO-7: (258): Action: Action_Null
Message #490 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_AUTH_FAIL
Message #491 : IKEv2-PROTO-4: (258): Verification of peer's authentication data FAILED
Message #492 : IKEv2-PROTO-4: (258): Sending authentication failure notify
Message #493 : IKEv2-PROTO-4: (258): Building packet for encryption.
Message #494 : (258):
Payload contents:
Message #495 : (258): NOTIFY(AUTHENTICATION_FAILED)Message #496 : (258): Next payload: NONE, reserved: 0x0, length: 8
Message #497 : (258): Security protocol id: IKE, spi size: 0, type: AUTHENTICATION_FAILED
Message #498 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_ENCRYPT_MSG

to note, logs show PSK and cryptomap proxy match and pass successfully.

[Flavio Miranda]

@matthewkylestephens 

 Are responsible for both VPN peers?

[matthewkylestephens]

I have to get someone else on the other side to help out and those times are random. 

Here is more output from a "show console-output"

Message #406 : IKEv2-PROTO-4: (258): Received Packet [From PA:500/To FPR:500/VRF i0:f0]
Message #407 : (258): Initiator SPI : C13A16583F7D4ED8 - Responder SPI : B609EEBD3422C056 Message id: 1
Message #408 : (258): IKEv2 IKE_AUTH Exchange REQUESTMessage #409 : IKEv2-PROTO-5: (258): Next payload: ENCR, version: 2.0 Message #410 : (258): Exchange type: IKE_AUTH, flags: INITIATOR Message #411 : (258): Message id: 1, length: 196Message #412 : (258):
Payload contents:
Message #413 : (258):
Message #414 : (258): Decrypted packet:Message #415 : (258): Data: 196 bytes
Message #416 : (258): REAL Decrypted packet:Message #417 : (258): Data: 136 bytes
Message #418 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_WAIT_AUTH Event: EV_RECV_AUTH
Message #419 : IKEv2-PROTO-4: (258): Stopping timer to wait for auth message
Message #420 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_WAIT_AUTH Event: EV_CHK_NAT_T
Message #421 : IKEv2-PROTO-4: (258): Checking NAT discovery
Message #422 : IKEv2-PROTO-4: (258): NAT not found
Message #423 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_WAIT_AUTH Event: EV_PROC_ID
Message #424 : IKEv2-PROTO-7: (258): Received valid parameteres in process id
Message #425 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_WAIT_AUTH Event: EV_CHK_IF_PEER_CERT_NEEDS_TO_BE_FETCHED_FOR_PROF_SEL
Message #426 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_WAIT_AUTH Event: EV_GET_POLICY_BY_PEERID
Message #427 : IKEv2-PROTO-4: (258): Searching policy based on peer's identity 'PA IP address' of type 'IPv4 address'
Message #428 : IKEv2-PLAT-4: (258): Site to Site connection detected
Message #429 : IKEv2-PLAT-4: (258): my auth method set to: 2
Message #430 : IKEv2-PLAT-4: (258): P1 ID = 0
Message #431 : IKEv2-PLAT-4: (258): Translating IKE_ID_AUTO to = 255
Message #432 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_WAIT_AUTH Event: EV_SET_POLICY
Message #433 : IKEv2-PROTO-7: (258): Setting configured policies
Message #434 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_WAIT_AUTH Event: EV_VERIFY_POLICY_BY_PEERID
Message #435 : IKEv2-PROTO-4: (258): Verify peer's policy
Message #436 : IKEv2-PROTO-4: (258): Peer's policy verified
Message #437 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_WAIT_AUTH Event: EV_CHK_AUTH4EAP
Message #438 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_WAIT_AUTH Event: EV_CHK_POLREQEAP
Message #439 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_CHK_AUTH_TYPE
Message #440 : IKEv2-PROTO-4: (258): Get peer's authentication method
Message #441 : IKEv2-PROTO-4: (258): Peer's authentication method is 'PSK'
Message #442 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_GET_PRESHR_KEY
Message #443 : IKEv2-PROTO-4: (258): Get peer's preshared key for PA IP address
Message #444 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_VERIFY_AUTH
Message #445 : IKEv2-PROTO-4: (258): Verify peer's authentication data
Message #446 : IKEv2-PROTO-4: (258): Use preshared key for id PA IP address, key len 12
Message #447 : IKEv2-PROTO-4: (258): Verification of peer's authenctication data PASSED
Message #448 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_CHK4_IC
Message #449 : IKEv2-PROTO-4: (258): Processing INITIAL_CONTACT
Message #450 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_CHK_REDIRECT
Message #451 : IKEv2-PROTO-7: (258): Redirect check is not needed, skipping it
Message #452 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_NOTIFY_AUTH_DONE
Message #453 : IKEv2-PLAT-4: (258): Completed authentication for connection
Message #454 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_CHK_CONFIG_MODE
Message #455 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_GET_CONFIG_MODE
Message #456 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_PROC_SA_TS
Message #457 : IKEv2-PROTO-4: (258): Processing IKE_AUTH message
Message #458 : IKEv2-PLAT-4: (258): PROXY MATCH on crypto map s2sCryptoMap seq 1
Message #459 : IKEv2-PROTO-7: (258): SM Trace-> SA: I_SPI=C13A16583F7D4ED8 R_SPI=B609EEBD3422C056 (R) MsgID = 00000001 CurState: R_VERIFY_AUTH Event: EV_NO_EVENT

[Flavio Miranda]

Seems ok to me. If you issue show crypt isamak sa you dont see any tunnel?

[matthewkylestephens]

There are no IKEv1 SAs

There are no IKEv2 SAs

[Aref Alsouqi]

These logs are not showing any errors or failed comms between the two peers. Is it still not working?

[matthewkylestephens]

correct, but we do get an error, it's in the original post. However, I think I know why.

[guillermo-serrano]

do you have encap and decap paquets? can you see it with "show crypto ipsec sa"

[matthewkylestephens]

There are no ipsec sas

[MHM Cisco World]

Hi friend 

this issue appear suddenly??

did you  do any upgrade for any device ?

can you do below 

1- undebug all
2-  

debug crypto ikev2 protocol 10
debug crypto ikev2 platform 10

Thanks 

MHM 

[matthewkylestephens]

Hello,

Yes, on the surface it was relayed to me that it was all of a sudden although, I wasn't first POC. It's possible there was a failover event or a power outage etc...I just don't know. I did run those commands however the device is stuck in a deployment and needs to rebooted so we are waiting on someone to do that. In the mean time, I did find that the initial problem may be in the fact that SHA1 is being used with DH Group 20, which is an unsupported config. I was able to get the PA side to drop their side to DH 5, but the Cisco side is now stuck on a deployment and requires a reboot. Apologies for the late response. I will update when I change the ikev2 config. also, we changed the psk to be sure they both matched and that didn't resolve the issue. (fyi) Thank you all for your suggestions thus far.

[matthewkylestephens]

For reference, this is what both sides were setup as. For testing purposes, I had the Palo Alto admin drop the DH to 5 so as to meet requirements for the auth and encryption types. If the tunnel comes up, I'm going to raise the security levels on everything as obviously these are very deprecated at this point.

[Aref Alsouqi]

If the Palo is not sitting behind a NAT device I would ask its admins to ensure no identification values are configured in IKE gateway.

[MHM Cisco World]

match DH group as I know there is Palo bug is peer use dh5. 

so match dh group and try use different dh group

MHM