ISAKMP malformed payload received

pamirian76 · ‎11-12-2004

Hi,

I have a pix 520 with version 6.3(4) installed and everything works fine excpet...

I have a VPN tunnel with a client using checkpoint NG. it works fine but sometimes people get disconnected from the servers and in my pix logs I get this

the client got disconnected around 11:58 and this is my log

Nov 12 11:54:57 1.3.2.3 %PIX-7-702202: ISAKMP Phase 1 delete sent (local A.A.A.A (responder), remote B.B.B.B)

Nov 12 11:55:08 1.3.2.3 %PIX-7-702208: ISAKMP Phase 1 exchange started (local A.A.A.A (responder), remote B.B.B.B)

Nov 12 11:55:08 1.3.2.3 %PIX-7-702210: ISAKMP Phase 1 exchange completed (local A.A.A.A (responder), remote B.B.B.B)

Nov 12 11:55:08 1.3.2.3 %PIX-6-602201: ISAKMP Phase 1 SA created (local A.A.A.A/500 (responder), remote B.B.B.B/500, authentication=pre-share, encryption=3DES-CBC, hash=MD5, group=2, lifetime=86400s)

Nov 12 11:55:08 1.3.2.3 %PIX-6-602302: deleting SA, (sa) sa_dest= A.A.A.A, sa_prot= 50, sa_spi= 0xe7b44330(3887350576), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 20

Nov 12 11:55:08 1.3.2.3 %PIX-6-602302: deleting SA, (sa) sa_dest= B.B.B.B, sa_prot= 50, sa_spi= 0xfdbcedb1(4257017265), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 19

Nov 12 11:55:09 1.3.2.3 %PIX-6-602302: deleting SA, (sa) sa_dest= A.A.A.A, sa_prot= 50, sa_spi= 0xa733eeef(2805198575), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 8

Nov 12 11:55:09 1.3.2.3 %PIX-6-602302: deleting SA, (sa) sa_dest= B.B.B.B, sa_prot= 50, sa_spi= 0xff225c47(4280441927), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 7

Nov 12 11:56:31 1.3.2.3 %PIX-7-702209: ISAKMP Phase 2 exchange started (local A.A.A.A (responder), remote B.B.B.B)

Nov 12 11:56:31 1.3.2.3 %PIX-6-602301: sa created, (sa) sa_dest= A.A.A.A, sa_prot= 50, sa_spi= 0x3f6a5007(1063931911), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 7

Nov 12 11:56:31 1.3.2.3 %PIX-6-602301: sa created, (sa) sa_dest= B.B.B.B, sa_prot= 50, sa_spi= 0x94355919(2486524185), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 8

Nov 12 11:56:31 1.3.2.3 %PIX-7-702211: ISAKMP Phase 2 exchange completed (local A.A.A.A (responder), remote B.B.B.B)

Nov 12 11:56:31 1.3.2.3 %PIX-7-702206: ISAKMP malformed payload received (local A.A.A.A (responder), remote B.B.B.B)

Nov 12 12:02:59 1.3.2.3 %PIX-7-702209: ISAKMP Phase 2 exchange started (local A.A.A.A (responder), remote B.B.B.B)

Nov 12 12:02:59 1.3.2.3 %PIX-6-602301: sa created, (sa) sa_dest= A.A.A.A, sa_prot= 50, sa_spi= 0xbfe98b9d(3219753885), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 19

Nov 12 12:02:59 1.3.2.3 %PIX-6-602301: sa created, (sa) sa_dest= B.B.B.B, sa_prot= 50, sa_spi= 0x5b58253a(1532503354), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 20

Nov 12 12:03:00 1.3.2.3 %PIX-7-702211: ISAKMP Phase 2 exchange completed (local A.A.A.A (responder), remote B.B.B.B)

Nov 12 12:03:00 1.3.2.3 %PIX-7-702206: ISAKMP malformed payload received (local A.A.A.A (responder), remote B.B.B.B)

can "ISAKMP malformed payload received" cause the disconnections?

Cisco says this

Error Message %PIX-7-702206: ISAKMP malformed payload received (local <ip>

(initiator|responder), remote <ip>)

Explanation ISAKMP received an illegal or malformed message. May indicate an out of sync problem with the remote peer, a problem decrypting a message, or a message received out of order.

Recommended Action If using preshared key, verify local preshared key is configured correctly on local and remote device. Check local and remote configuration, additional troubleshooting may be required if SA fails to come up.

I've checked everythin with the checkpoint admin and everything seems to be ok.

any ideas?

thank you.

ehirsel · ‎11-12-2004

Are the clients able to reconnect without error after a period of time? Or do you have to take action, such as running the clear cry sa (clears IPSec/phase 2 sa's) or clear cry isa sa (clear phase 1 sa/s)?

I noted that the log message about the phase 1 delete, and I what I think is the phase 2 sa deletion (messages that appear right after the phase 1 complete) but the other may not have deleted it phase 2 sa's until a certain amount of time passed.

One other question I have is what is the life time of the phase 2 sa? Are they the same as phase 1? If so, I would make them different. Phase 1 has an 86400 sec lifetime from what I can tell by your log messages - which is one 24 hr period. Make the phase 2 sa's much shorter, say 8 hours. This way they will expire before phase 1 and phase 1 won't renegotiate until it sees that the phase 2 sa's have expired and note that it too has expired.

Let me know if this helps.

pamirian76 · ‎11-12-2004

yes they can reconnect right away but the citrix sessions gets disconnected and they lose what they had open.

no I do not have to do anything manually, no clear commands, it comes back right away!

the life time for phase1 is 86400 and phase 2 is 3600 seconds.

thank you for your help.

ehirsel · ‎11-12-2004

Are both the ISA and IPSec sa lifetimes configured the same on both sides? With regards to isa keepalive - is that configured on both sides? If so, are the keepalive values the same on both ends?

pamirian76 · ‎11-12-2004

hi,

well I don't think it has anything to do with keepalives because as I said, it worked without any problems for a week and a half and suddenly it's doing this... I mean if the keepalives are not the same on both sides, I'll start having problems within a day 86400 seconds... right?

thanks.

ehirsel · ‎11-13-2004

You noted in your last post about not having a problem for a week and a half, yet I didn't see that stated in your initial post.

In the past week and a half - did you upgrade the pix os to 6.3.4? Did you or the remote peer add any new vpn connections? Modify the global phase 1 or phase 2 policy priorities?

One item that I noted, that I overlooked was the approx. 6 min. interval when the phase 2 process was renegotiated (once at 11:56 and again at 12:02).

With regards to the sa lifetimes, do the values that you stated match your peers? Is your pix 3des and aes capable? Or can it only do single des encryption? It could be that the other end is sending a frame to re-start the phase 2 renegotiation that you do not understand - or it is trying to send messages using expired sa values.

I agree with you that if keepalive was an issue, you would have come across it by now - but it does not hurt to validate it while you validate the sa lifetimes.

Let me know what you find.

pamirian76 · ‎11-15-2004

well I had this problem with my pix at version 5.3 then I decided to upgrade to 6.3(4) and again I have this same problem so I don't think that the pix version has anything to do with my problem.

I'm sending my configs just in case something's not right...

----------

access-list someclient permit ip 1.0.50.0 255.255.255.0 172.16.16.0 255.255.255.0

access-list someclient permit ip 1.0.50.0 255.255.255.0 172.16.20.0 255.255.255.0

access-list someclient permit ip 1.0.50.0 255.255.255.0 172.16.18.0 255.255.255.0

access-list someclient permit ip 1.0.50.0 255.255.255.0 172.16.40.0 255.255.255.0

access-list someclient permit ip 1.0.50.0 255.255.255.0 172.16.36.0 255.255.255.0

access-list someclient permit ip 1.0.50.0 255.255.255.0 172.16.38.0 255.255.255.0

access-list someclient permit ip 1.0.50.0 255.255.255.0 172.16.32.0 255.255.252.0

access-list someclient permit ip 1.0.50.0 255.255.255.0 172.16.193.0 255.255.255.0

access-list someclient permit ip 1.0.50.0 255.255.255.0 172.16.194.0 255.255.255.0

crypto ipsec transform-set 3DES-MD5 esp-3des esp-md5-hmac

crypto map testmap 21 ipsec-isakmp

crypto map testmap 21 match address someclient

crypto map testmap 21 set peer x.x.x.x

crypto map testmap 21 set transform-set 3DES-MD5

crypto map testmap 21 set security-association lifetime seconds 3600 kilobytes 4608000

isakmp key mysecretkey address x.x.x.x netmask 255.255.255.255

isakmp policy 4 authentication pre-share

isakmp policy 4 encryption 3des

isakmp policy 4 hash md5

isakmp policy 4 group 2

isakmp policy 4 lifetime 86400

----------

I'm not sure but I think that the checkpoint does not have the cryptomap lifetime or it's maybe the isakmp lifetime. can this be the problem? I have both lifetimes but the checkpoint NG has only 1. can it be the cause?

thanks for your time.

oh and yes my pix does support aes and 3des.

jmia · ‎11-15-2004

Patrick,

Cross-check your config on the pix and checkpoint NG with the following document:

http://www.cisco.com/en/US/tech/tk583/tk372/technologies_configuration_example09186a00800ef796.shtml

Let me know how you get on, your initial config posted above looks ok and I presume you've got sysopt connection permit-ipsec enabled on the pix.

My thoughts are that this problem you are seeing is related to the Checkpoint NG side, would be interested to see the result.

Jay

pamirian76 · ‎11-15-2004

Hi Jay,

the link you sent me is what I've used to setup the vpn on MY (Cisco PIX) side.

I've also sent this same document to the client (checkpoint NG) but the client says that the document is old (screens are from older version of checkpoint) and no luck.

YES, I also have

sysopt connection permit-ipsec

in my configs, I forgot to copy/paste it :)

is there any way for me to prove that the problem is on the client side and not my side?

jmia · ‎11-15-2004

Patrick,

Try this document:

http://secureknowledge.checkpoint.com/pub/sk/docs/public/firewall1/4_1/pdf/pixvpn.pdf

Hope this helps and let me know how you get on.

Jay

ehirsel · ‎11-16-2004

It could very well be that you having 2 lifetimes coded and the other end only one can be the source of the issue.

What I would do is to contact the remote peer admin and review the isa and ipsec config and insure that all lifetimes for both sets of sa's agree.

Most likely you having coded a kb lifetime is what the difference is; most cases just use a seconds lifetime. The KB default is approx. 4GB (you ahve about 4 MB coded).

Let me know how it proceeds.

pamirian76 · ‎11-17-2004

hi,

you know what, I think your 4 mb limit makes alot of sense.

now, can I use this command without giving it a kilobyte limit? I don't want to use a limit for kilobyte

cyrpto map testmap 30 set security-association lifetime seconds 3600

it automatically puts a 4mb limit

I know that I could add "kilobytes 4608000 00" for 460 mb but I want NO LIMIT on KB.

can it be done?

thanks.

pamirian76 · ‎11-18-2004

so on the pix 6.3(4) is it possible to

cyrpto map testmap 30 set security-association lifetime seconds 3600

without giving it the kilobytes option because I don't want to use a limit on the kilobytes.

thanks.

pamirian76 · ‎11-18-2004

ok forget about all this,

4608000 kb is 4 gigs not 4 megs.

thanks anyway.

ehirsel · ‎11-18-2004

My mistake on the volume limit; I misread one of your earlier posts.

It could be that the issue is the time limit.

Were you able to verify all aspects of your crypto map config with the partner's administator? You want to insure that if you code a lifetime on your crypto map seq 30 statements, that your partner configs the SAME values on theirs. Note the cisco does default to a 4 GB volume limit, I don't know about other vendor's products. However the lifetime limit can be different, and if one side specs a value that is not the default, then it will be negotiated and if the other side does not spec a value, then negotiation can fail.

Contact the other side's admin and let me know how things proceed from here.