Getting "IPSEC(epa_des_crypt): decrypted packet failed SA identi

frankdurda4 · ‎10-11-2012

One computer at COMPANY-A is attempting to communicate with two

computers located at COMPANY-B, via an IPsec tunnel between the

two companies.

All communications are via TCP protocol.

All devices present public IP addresses to one another, although they

may have RFC 1918 addresses on other interfaces, and NAT may be in use

on the COMPANY-B side. (NAT is not being used on the COMPANY-A side.)

The players:(Note: first three octets have been changed for security reasons)

COMPANY-A computer 1.2.3.161

COMPANY-A router 1.2.3.8 (also IPsec peer)

COMPANY-A has 1.2.3.0/24 with no subnetting.

COMPANY-B router 4.5.6.228 (also IPsec peer)

COMPANY-B computer #1 4.5.7.94 (this one has no issues)

COMPANY-B computer #2 4.5.7.29 (this one fails)

COMPANY-B has 4.5.6.0/23 subnetted in various ways.

COMPANY-B also has 9.10.11.0/24, but it is not involved in the issue.

What works:

The COMPANY-A computer 1.2.3.161 can communicate via the single IPsec

tunnel to COMPANY-B computer #1 4.5.7.94 without problems.

The "show crypto session detail" command shows Inbound/Outbound packets

flowing in the dec'ed and enc'ed positions.

What doesn't:

When the COMPANY-A computer 1.2.3.161 attempts to communicate

via the single IPsec tunnel with the COMPANY-B computer #2 4.5.7.29,

the COMPANY-A router eventually reports five of these messages:

Oct 9 15:24:54.327: IPSEC(epa_des_crypt): decrypted packet failed SA identity check

Oct 9 15:24:57.327: IPSEC(epa_des_crypt): decrypted packet failed SA identity check

Oct 9 15:25:03.327: IPSEC(epa_des_crypt): decrypted packet failed SA identity check

Oct 9 15:25:15.328: IPSEC(epa_des_crypt): decrypted packet failed SA identity check

Oct 9 15:25:39.329: IPSEC(epa_des_crypt): decrypted packet failed SA identity check

Oct 9 15:26:27.328: IPSEC(epa_des_crypt): decrypted packet failed SA identity check

and the "show crypto session detail" shows inbound packets being dropped.

The COMPANY-A computer that opens the TCP connection never gets past the

SYN_SENT phase of the TCP connection whan trying to communicate with the

COMPANY-B computer #2, and the repeated error messages are the retries of

the SYN packet.

On the COMPANY-A side, this IPsec configuration has been set up on a 3745,

a 3725, and some 76xx routers were tried, all with similar behavior,

with packets from one far-end computer passing fine, and packets from

another far-end computer in the same netblock passing through the same

IPsec tunnel failing with the "failed SA identity" error.

The COMPANY-A computer directs all packets headed to COMPANY-B via the

COMPANY-A router at 1.2.3.8 with this set of route settings:

netstat -r -n

Kernel IP routing table

Destination Gateway Genmask Flags MSS Window irtt Iface

4.5.7.0 1.2.3.8 255.255.255.0 UG 0 0 0 eth3

1.2.3.8.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3

10.1.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0

169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth3

10.0.0.0 10.1.1.1 255.0.0.0 UG 0 0 0 eth0

0.0.0.0 1.2.3.1 0.0.0.0 UG 0 0 0 eth3

The first route line shown is selected for access to both COMPANY-B computers.

The COMPANY-A router (IPsec tunnel endpoint, 1.2.3.8) has this

configuration:

...

crypto isakmp policy 10

encr 3des

authentication pre-share

group 2

lifetime 28800

crypto isakmp key XXXXXXXXXXXXXXXXXXXXXXX address 4.5.6.228

!

crypto ipsec security-association lifetime seconds 86400

!

crypto ipsec transform-set COMPANY-B01 esp-3des esp-sha-hmac

!

crypto map COMPANY-BMAP1 10 ipsec-isakmp

description COMPANY-B VPN

set peer 4.5.6.228

set transform-set COMPANY-B01

set pfs group2

match address 190

...

interface FastEthernet0/0

ip address 1.2.3.8 255.255.255.0

no ip redirects

ip virtual-reassembly

duplex auto

speed auto

no cdp enable

crypto map COMPANY-BMAP1

...

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 1.2.3.1

ip route 10.0.0.0 255.0.0.0 10.1.1.1

ip route 1.2.3.8.0 255.255.255.0 FastEthernet0/0

...

access-list 190 permit ip host 1.2.3.161 4.5.7.0 0.0.0.255

access-list 190 permit ip host 1.2.3.161 9.10.11.0 0.0.0.255

bridge 1 protocol ieee

...

One of the routers tried had this IOS/hardware configuration:

Cisco IOS Software, 3700 Software (C3725-ADVIPSERVICESK9-M), Version 12.4(25c),

RELEASE SOFTWARE (fc2)

isco 3725 (R7000) processor (revision 0.1) with 115712K/15360K bytes of memory.

Processor board ID XXXXXXXXXXXXXXX

R7000 CPU at 240MHz, Implementation 39, Rev 3.3, 256KB L2 Cache

2 FastEthernet interfaces

4 ATM interfaces

DRAM configuration is 64 bits wide with parity disabled.

55K bytes of NVRAM.

31296K bytes of ATA System CompactFlash (Read/Write)

250368K bytes of ATA Slot0 CompactFlash (Read/Write)

Configuration register is 0x2102

#show crypto sess

Crypto session current status

Interface: FastEthernet0/0

Session status: UP-ACTIVE

Peer: 4.5.6.228 port 500

IKE SA: local 1.2.3.8/500 remote 4.5.6.228/500 Active

IPSEC FLOW: permit ip host 1.2.3.161 4.5.7.0/255.255.255.0

Active SAs: 2, origin: crypto map

IPSEC FLOW: permit ip host 1.2.3.161 9.10.11.0/255.255.255.0

Active SAs: 0, origin: crypto map

#show crypto sess det

Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer Detection

K - Keepalives, N - NAT-traversal, X - IKE Extended Authentication

Interface: FastEthernet0/0

Session status: UP-ACTIVE

Peer: 4.5.6.228 port 500 fvrf: (none) ivrf: (none)

Phase1_id: 4.5.6.228

Desc: (none)

IKE SA: local 1.2.3.8/500 remote 4.5.6.228/500 Active

Capabilities:(none) connid:1 lifetime:06:26:27

IPSEC FLOW: permit ip host 1.2.3.161 4.5.7.0/255.255.255.0

Active SAs: 2, origin: crypto map

Inbound: #pkts dec'ed 651 drop 16 life (KB/Sec) 4496182/23178

Outbound: #pkts enc'ed 574 drop 2 life (KB/Sec) 4496279/23178

IPSEC FLOW: permit ip host 1.2.3.161 9.10.11.0/255.255.255.0

Active SAs: 0, origin: crypto map

Inbound: #pkts dec'ed 0 drop 0 life (KB/Sec) 0/0

Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 0/0

The COMPANY-B device on their end of the IPsec VPN is a Juniper SSG1000

Version 6.1 (ScreenOS)

We only have a limited view into the Juniper device configuration.

What we were allowed to see was:

COMPANY-B-ROUTER(M)-> sh config | incl COMPANY-A

set address "Untrust" "oss-COMPANY-A-1.2.3.161" 1.2.3.161 255.255.255.255

set ike gateway "COMPANY-A-1-GW" address 1.2.3.8 Main outgoing-interface "ethernet2/1" preshare xxxxxxxxxxxxxxxxxxxxxx proposal "pre-g2-3des-sha"

set vpn "COMPANY-A-1-IKE" gateway "COMPANY-A-1-GW" no-replay tunnel idletime 0 proposal "g2-esp-3des-sha-28800"

set policy id 2539 from "Untrust" to "Trust" "oss-COMPANY-A-1.2.3.161" "9.10.11.0-24" "ANY" tunnel vpn "COMPANY-A-1-IKE" id 0x309a pair-policy 2500

set policy id 2500 from "Trust" to "Untrust" "9.10.11.0-24" "oss-COMPANY-A-1.2.3.161" "ANY" tunnel vpn "COMPANY-A-1-IKE" id 0x309a pair-policy 2539

set policy id 2541 from "Trust" to "Untrust" "4.5.7.0-24" "oss-COMPANY-A-1.2.3.161" "ANY" tunnel vpn "COMPANY-A-1-IKE" id 0x309b pair-policy 2540

set policy id 2540 from "Untrust" to "Trust" "oss-COMPANY-A-1.2.3.161" "4.5.7.0-24" "ANY" tunnel vpn "COMPANY-A-1-IKE" id 0x309b pair-policy 2541

COMPANY-B-ROUTER(M)->

I suspect that this curious issue is due to a configuration setting on the

Juniper device, but neither party has seen this error before. COMPANY-B

operates thousands of IPsec VPNs and they report that this is a new error

for them too. The behavior that allows traffic from one IP address to

work and traffic from another to end up getting this error is also unique.

As only the Cisco side emits any error message at all, this is the only

clue we have as to what is going on, even if this isn't actually an IOS

problem.

What we are looking for is a description of exactly what the Cisco

IOS error message:

IPSEC(epa_des_crypt): decrypted packet failed SA identity check

is complaining about, and if there are any known causes of the behavior

described that occur when running IPsec between Cisco IOS and a Juniper

SSG device. Google reports many other incidents of the same error

message (but not the "I like that IP address but hate this one" behavior),

and not just with a Juniper device on the COMPANY-B end, but for those cases,

not one was found where the solution was described.

It is hoped that with a better explanation of the error message

and any known issues with Juniper configuration settings causing

this error, we can have COMPANY-B make adjustments to their device.

Or, if there is a setting change needed on the COMPANY-A router,

that can also be implemented.

Thanks in advance for your time in reading this, and any ideas.

Harish Balakrishnan · ‎10-12-2012

Hello Frank,

Just a small query to start with.. Does the comupter 4.5.7.29 has 2 NIC cards by any chance ?

also while the communication happening to this PC, can you check whether both encapsulation and decapsulation happening in 'show crypto ipsec sa'

regards

Harish.

frankdurda4 · ‎10-12-2012

Hello Harish,

It is believed that:

COMPANY-B computer #1 4.5.7.94 (this one has no issues)

COMPANY-B computer #2 4.5.7.29 (this one fails)

both have at least two network interfaces, one with a public IP address

(which we are supposedly conversing with) and one with a RFC 1918 type

address. COMPANY-B is reluctant to disclose details of their network or

servers setup, so this is not 100% certain.

Because of that uncertainty, it occurred to me that perhaps COMPANY-B

computer #2 might be incorrectly routing via the RFC 1918 interface.

In theory, such packets should have been blocked by the access-list on both

COMPANY-A router, and should not have even made it into the IPsec VPN

if the Juniper access settings work as it appears they should. So I turned up

debugging on COMPANY-A router so that I could see the encrypted and

decrypted packet hex dumps.

I then hand-disassembled the decoded ACK packet IP header received just

prior to the "decrypted packet failed SA check" error being emitted and

found the expected source and destination IP addresses (4.5.7.29 and 1.2.3.161),

in the unecapsulated packet. I also found the expected port numbers of the TCP

conversation that was trying to be established in the TCP header. So, it

looks like COMPANY-B computer #2 is emitting the packets out the right

interface.

The IP packet header of the encrypted packet showed the IP addresses of the

two routers at each terminus of the IPsec VPN, but since I don't know what triggers

the "SA check" error message or what it is complaining about, I don't know what

other clues to look for in the packet dumps.

As to your second question, "can you check whether both encapsulation and

decapsulation happening in 'show crypto ipsec sa'", the enc'ed/dec'ed

counters were both going up by the correct quantities. When communicating

with the uncooperative COMPANY-B computer #2, you would also see the

received Drop increment for each packet decrypted. When communicating

with the working COMPANY-B computer #1, the Drop counters would not

increment, and the enc'ed/dec'ed would both increment.

#show crypto sess det

Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer Detection

K - Keepalives, N - NAT-traversal, X - IKE Extended Authentication

Interface: FastEthernet0/0

Session status: UP-ACTIVE

Peer: 4.5.6.228 port 500 fvrf: (none) ivrf: (none)

Phase1_id: 4.5.6.228

Desc: (none)

IKE SA: local 1.2.3.8/500 remote 4.5.6.228/500 Active

Capabilities:(none) connid:1 lifetime:07:59:54

IPSEC FLOW: permit ip host 1.2.3.161 4.5.7.0/255.255.255.0

Active SAs: 2, origin: crypto map

Inbound: #pkts dec'ed 376 drop 5 life (KB/Sec) 4458308/28784

Outbound: #pkts enc'ed 401 drop 3 life (KB/Sec) 4458308/28784

...

Attempt a TCP communication to COMPANY-B computer #2...

show crypto sess det

Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer Detection

K - Keepalives, N - NAT-traversal, X - IKE Extended Authentication

Interface: FastEthernet0/0

Session status: UP-ACTIVE

Peer: 4.5.6.228 port 500 fvrf: (none) ivrf: (none)

Phase1_id: 4.5.6.228

Desc: (none)

IKE SA: local 1.2.3.8/500 remote 4.5.6.228/500 Active

Capabilities:(none) connid:1 lifetime:07:59:23

IPSEC FLOW: permit ip host 1.2.3.161 4.5.7.0/255.255.255.0

Active SAs: 2, origin: crypto map

Inbound: #pkts dec'ed 376 drop 6 life (KB/Sec) 4458307/28753

Outbound: #pkts enc'ed 402 drop 3 life (KB/Sec) 4458307/28753

...

Note Inbound "drop" changed from 5 to 6. (I didn't let it sit for all

the retries.)

#show crypto ipsec sa

interface: FastEthernet0/0

Crypto map tag: COMPANY-BMAP1, local addr 1.2.3.8

protected vrf: (none)

local ident (addr/mask/prot/port): (1.2.3.161/255.255.255.255/0/0)

remote ident (addr/mask/prot/port): (4.5.7.0/255.255.255.0/0/0)

current_peer 4.5.6.228 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 402, #pkts encrypt: 402, #pkts digest: 402

#pkts decaps: 376, #pkts decrypt: 376, #pkts verify: 376

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 3, #recv errors 6

local crypto endpt.: 1.2.3.8, remote crypto endpt.: 4.5.6.228

path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0

current outbound spi: 0xDF2CC59C(3744253340)

inbound esp sas:

spi: 0xD9D2EBBB(3654478779)

transform: esp-3des esp-sha-hmac ,

in use settings ={Tunnel, }

conn id: 2004, flow_id: SW:4, crypto map: COMPANY-BMAP1

sa timing: remaining key lifetime (k/sec): (4458307/28600)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0xDF2CC59C(3744253340)

transform: esp-3des esp-sha-hmac ,

in use settings ={Tunnel, }

conn id: 2003, flow_id: SW:3, crypto map: COMPANY-BMAP1

sa timing: remaining key lifetime (k/sec): (4458307/28600)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

...

The "send" errors appear to be related to the tunnel reverting to a

DOWN state after periods of inactivity, and you appear to get one

each time the tunnel has to be re-negotiated and returned to

an ACTIVE state. There is no relationship between Send errors

incrementing and working/non-working TCP conversations to the

two COMPANY-B servers.

Thanks for pondering this very odd behavior.

Getting "IPSEC(epa_des_crypt): decrypted packet failed SA identity check" messages on packets from only one of two far-end sources sharing the same tunnel, the other source works fine. What exactly does this error mean?