topic Re: Odd crypto-map behavior in Network Security

Odd crypto-map behavior

timdeadman1 — Mon, 10 Jun 2019 15:20:54 GMT

Our IKEv2 VPN is showing some very odd behavior. We have two devices our end that need to be seen as interesting traffic, they are n.n.8.4 and n.n.9.4. When these were configured I put in 8.4 first followed by the 9.4, the tunnel came up, traffic passed through it, but only traffic from 8.4, 9.4 traffic was dropped as "(vpn-context-expired) Expired VPN context". On doing a packet trace I got this message.
*********************************************************************
WARNING: An existing decryption SA was not found. Please confirm the
IPsec Phase 2 SA or Anyconnect Tunnel is established.
*********************************************************************

Changing the pack trace to 8.4 gave me traffic all the way through.

By deleting 8.4 from the crypto-map I was able to get 9.4 to use the tunnel, but then even after I added 8.4 back in, it portrayed the same characteristics as 9.4 had at the start.

This is driving me nuts.

Re: Odd crypto-map behavior

Sheraz.Salim — Mon, 10 Jun 2019 19:43:45 GMT

which ASA code you on? and does this ASA is in context mode?

check this if you hitting a bug

Cisco Bug: CSCup37416

Re: Odd crypto-map behavior

timdeadman1 — Tue, 11 Jun 2019 14:07:05 GMT

It is running 9.10(1) and is in single context mode.

Hope this helps

Re: Odd crypto-map behavior

Sheraz.Salim — Wed, 12 Jun 2019 07:44:33 GMT

what does asp drop shows you. also confirm you are using a object network in access-cryto map or you using the ip address in cryto map access-list.

Re: Odd crypto-map behavior

timdeadman1 — Wed, 12 Jun 2019 09:55:38 GMT

Hi Sheraz,

We are using network objects that resolve to an IP address, these are configured as a network group that is called by the crypto map.

Cheers

Re: Odd crypto-map behavior

Sheraz.Salim — Wed, 12 Jun 2019 10:29:26 GMT

share the output command of show asp drop

Re: Odd crypto-map behavior

timdeadman1 — Wed, 12 Jun 2019 13:08:30 GMT

sh asp drop

Frame drop:
NAT-T keepalive message (natt-keepalive) 1138
SVC Module does not have a channel for reinjection (mp-svc-no-channel) 955
SVC Module does not have a session (mp-svc-no-session) 692
SVC Module is in flow control (mp-svc-flow-control) 1257304
SVC Module unable to fragment packet (mp-svc-no-fragment) 7
Expired VPN context (vpn-context-expired) 25
Flow is being freed (flow-being-freed) 26
ttl exceeded (ttl-exceeded) 195494
Invalid TCP Length (invalid-tcp-hdr-length) 90
No valid adjacency (no-adjacency) 9493864
No route to host (no-route) 1647335
Flow is denied by configured rule (acl-drop) 347562058
No same-security-traffic configured (no-same-security-traffic) 4586429
Flow denied due to resource limitation (unable-to-create-flow) 78
First TCP packet not SYN (tcp-not-syn) 1404901
Bad TCP checksum (bad-tcp-cksum) 3
TCP data send after FIN (tcp-data-past-fin) 5
TCP failed 3 way handshake (tcp-3whs-failed) 75239
TCP RST/FIN out of order (tcp-rstfin-ooo) 6446971
TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 211
TCP SYNACK on established conn (tcp-synack-ooo) 334
TCP packet SEQ past window (tcp-seq-past-win) 54968
TCP invalid ACK (tcp-invalid-ack) 2215
TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 186
TCP RST/SYN in window (tcp-rst-syn-in-win) 1731
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 459
TCP packet failed PAWS test (tcp-paws-fail) 224
SSL first record invalid (ssl-first-record-invalid) 30
CTM returned error (ctm-error) 152
Slowpath security checks failed (sp-security-failed) 2851032
IP option drop (invalid-ip-option) 4503
Expired flow (flow-expired) 75876
ICMP Inspect bad icmp code (inspect-icmp-bad-code) 1236
ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 1420
ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 25
DNS Inspect invalid packet (inspect-dns-invalid-pak) 7
DNS Inspect invalid domain label (inspect-dns-invalid-domain-label) 5
DNS Inspect id not matched (inspect-dns-id-not-matched) 8806
FP L2 rule drop (l2_acl) 95
Interface is down (interface-down) 320
Dropped pending packets in a closed socket (np-socket-closed) 24212
NAT failed (nat-xlate-failed) 702
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 844124

Last clearing: Never

Flow drop:
Tunnel being brought up or torn down (tunnel-pending) 62
Need to start IKE negotiation (need-ike) 12
SVC spoof packet detected (svc-spoof-detect) 4
SVC replacement connection established (svc-replacement-conn) 1128
VPN overlap conflict (vpn-overlap-conflict) 27770
VPN decryption missing (vpn-missing-decrypt) 12834
Flow is denied by access rule (acl-drop) 428156
NAT reverse path failed (nat-rpf-failed) 114
Inspection failure (inspect-fail) 968834
SSL bad record detected (ssl-bad-record-detect) 1249
SSL handshake failed (ssl-handshake-failed) 9137
DTLS hello processed and closed (dtls-hello-close) 3279
SSL record decryption failed (ssl-record-decrypt-error) 4

Last clearing: Never

Re: Odd crypto-map behavior

Sheraz.Salim — Wed, 12 Jun 2019 13:41:52 GMT

This look interesting.

VPN overlap conflict (vpn-overlap-conflict) 27770
VPN decryption missing (vpn-missing-decrypt) 12834

you mind if you can share the configuration. you can hide the ip addresses or make up to different ip address and also remove the password or any other sensitive information.

also when you interesting acl for vpn in place. could you issue the command show crypto ipsec sa peer x.x.x.x and show the output.

Re: Odd crypto-map behavior

timdeadman1 — Wed, 12 Jun 2019 14:36:53 GMT

Thanks for your help and interest in this Sheraz, afraid I can't post the sh run output as we have too much sensitive information here but the output of "sh crypto ipsec sa peer n.n.8.4 and 9.4" is the same. "There are no ipsec sa for peer n.n.8.4" and "There are no ipsec sa for peer n.n.9.4"

The fact that I can get 8.4 working by deleting the re-adding 9.4 or get 9.4 working by doing the same with 8.4 just seems to point to a bug rather than a config error. I have tried deleting the config and re installing by CLI and have also tried making a new object group with IP addresses rather than object names. Neither of these have worked. I still only see one working at a time.

Re: Odd crypto-map behavior

timdeadman1 — Wed, 12 Jun 2019 14:42:04 GMT

This is the output of "packet trace input WAN-New tcp A.B.C.D sql n.n.8.4 sql decrypt detail."

*********************************************************************
WARNING: An existing decryption SA was not found. Please confirm the
IPsec Phase 2 SA or Anyconnect Tunnel is established.
*********************************************************************

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc771e20ac0, priority=13, domain=capture, deny=false
hits=202804823, user_data=0x7fc7792b7ed0, cs_id=0x0, l3_type=0x0
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0000.0000.0000
input_ifc=WAN-New, output_ifc=any

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
<--- More ---> Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc76efc2110, priority=1, domain=permit, deny=false
hits=104853406, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=WAN-New, output_ifc=any

Phase: 3
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop x.x.x.254 using egress ifc INSIDE

Phase: 4
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (BRS-INSIDE-75,WAN-New) source static DM_INLINE_NETWORK_33 DM_INLINE_NETWORK_33 destination static ACL-remote ACL-remote no-proxy-arp route-lookup description ACL IPsec Link
Additional Information:
NAT divert to egress interface BRS-INSIDE-75
<--- More ---> Untranslate n.n.8.4/1521 to n.n.8.4/1521

Result:
input-interface: WAN-New
input-status: up
input-line-status: up
output-interface: INSIDE
output-status: up
output-line-status: up
Action: drop
Drop-reason: (vpn-context-expired) Expired VPN context

Re: Odd crypto-map behavior

timdeadman1 — Wed, 12 Jun 2019 14:54:34 GMT

And this is from the working 9.4.

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc771e20ac0, priority=13, domain=capture, deny=false
hits=202780541, user_data=0x7fc7792b7ed0, cs_id=0x0, l3_type=0x0
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0000.0000.0000
input_ifc=WAN-New, output_ifc=any

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc76efc2110, priority=1, domain=permit, deny=false
hits=104841307, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=WAN-New, output_ifc=any

Phase: 3
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop E.F.H.U using egress ifc INSIDE

Phase: 4
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (INSIDE,WAN-New) source static DM_INLINE_NETWORK_33 DM_INLINE_NETWORK_33 destination static ACL-remote ACL-remote no-proxy-arp route-lookup description ACL IPsec Link
Additional Information:
NAT divert to egress interface INSIDE
Untranslate n.n.9.4/1521 to n.n.9.4/1521

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (INSIDE,WAN-New) source static DM_INLINE_NETWORK_33 DM_INLINE_NETWORK_33 destination static ACL-remote ACL-remote no-proxy-arp route-lookup description ACL IPsec Link
Additional Information:
Static translate A.B.C.D/1521 to A.B.C.D/1521
Forward Flow based lookup yields rule:
in id=0x7fc775478e00, priority=6, domain=nat, deny=false
hits=516826, user_data=0x7fc767b650b0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=A.B.C.D, mask=255.255.255.255, port=0, tag=any
dst ip/id=n.n.9.4, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=WAN-New, output_ifc=INSIDE

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc762057770, priority=0, domain=nat-per-session, deny=false
hits=877943179, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 7
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc76af8e080, priority=0, domain=permit, deny=true
hits=297776, user_data=0xa, cs_id=0x0, use_real_addr, flags=0x1000, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=WAN-New, output_ifc=any

Phase: 8
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc7762b7510, priority=0, domain=inspect-ip-options, deny=true
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=WAN-New, output_ifc=any

Phase: 9
Type: SFR
Subtype:
Result: ALLOW
Config:
class-map SFR
match access-list SFR
policy-map global_policy
class SFR
sfr fail-open
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc773d09d40, priority=71, domain=sfr, deny=false
hits=634112, user_data=0x7fc76412e400, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=WAN-New, output_ifc=any

Phase: 10
Type: INSPECT
Subtype: inspect-sqlnet
Result: ALLOW
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect sqlnet
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc7762c60f0, priority=70, domain=inspect-sqlnet, deny=false
hits=6, user_data=0x7fc7640ffb30, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=1521, tag=any, dscp=0x0
input_ifc=WAN-New, output_ifc=any

Phase: 11
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fc7715902f0, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=1676, user_data=0xe6a34, cs_id=0x7fc77928cf80, reverse, flags=0x0, protocol=0
src ip/id=A.B.C.D, mask=255.255.255.255, port=0, tag=any
dst ip/id=n.n.9.4, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=WAN-New, output_ifc=any

Phase: 12
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (INSIDE,WAN-New) source static DM_INLINE_NETWORK_33 DM_INLINE_NETWORK_33 destination static ACL-remote ACL-remote no-proxy-arp route-lookup description ACL IPsec Link
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7fc765ae1f30, priority=6, domain=nat-reverse, deny=false
hits=516826, user_data=0x7fc772801f60, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=A.B.C.D, mask=255.255.255.255, port=0, tag=any
dst ip/id=n.n.9.4, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=WAN-New, output_ifc=INSIDE

Phase: 13
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7fc764139d60, priority=0, domain=user-statistics, deny=false
hits=562426620, user_data=0x7fc7640a4f80, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=INSIDE

Phase: 14
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7fc762057770, priority=0, domain=nat-per-session, deny=false
hits=877943181, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 15 Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7fc762e01d70, priority=0, domain=inspect-ip-options, deny=true
hits=564379902, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=INSIDE, output_ifc=any

Phase: 16
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0x7fc772776030, priority=70, domain=encrypt, deny=false
hits=1674, user_data=0xe4efc, cs_id=0x7fc77928cf80, reverse, flags=0x0, protocol=0
src ip/id=n.n.9.4, mask=255.255.255.255, port=0, tag=any
dst ip/id=A.B.C.D, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=WAN-New

Phase: 17
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0x7fc765812000, priority=0, domain=user-statistics, deny=false
hits=2788553, user_data=0x7fc7640a4f80, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=WAN-New

Phase: 18
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 602131890, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_ipsec_tunnel_flow
snp_fp_tcp_normalizer
snp_sfr
snp_fp_punt <inspect_sqlnet>
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_sfr
snp_fp_punt <inspect_sqlnet>
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_encrypt
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Result:
input-interface: WAN-New
input-status: up
input-line-status: up
output-interface: INSIDE
output-status: up
output-line-status: up
Action: allow

Re: Odd crypto-map behavior

Sheraz.Salim — Wed, 12 Jun 2019 16:03:47 GMT

You need to open a TAC CASE.

While looking at the cisco documentation.

it says,

Name: vpn-context-expired
Expired VPN context:
This counter will increment when the security appliance receives a packet that requires encryption or decryption, and the ASP VPN context required to perform the operation is no longer valid.

Recommendation:
This indicates that a software error should be reported to the Cisco TAC.

Syslogs:
None"

Re: Odd crypto-map behavior

timdeadman1 — Thu, 13 Jun 2019 07:40:27 GMT

Thanks Sheraz

Re: Odd crypto-map behavior

harveyheer — Fri, 14 Jun 2019 12:39:15 GMT

Hi Tim,

Try and run it through ASDM Packet Tracer

Harvey

Re: Odd crypto-map behavior

timdeadman1 — Sat, 15 Jun 2019 09:36:04 GMT

Harvey,

No idea why it worked, but it did. I'd run it through the CLI packet tracer many times, but twice through the ASDM PT and it burst into life.

You sir are a genius.....

Tim