08-29-2021 09:18 PM - edited 08-29-2021 09:22 PM
Hi guys, there comes to me a weird problem:
I have a pair of ASA 5525, between which I need to set an IPSec Site-to-Site VPN over internet:
ASA-5525-A: Cisco Adaptive Security Appliance Software Version 8.6(1)2 Device Manager Version 6.6(1) outside IP: Y.Y.Y.Y inside network Y: 192.168.200.0/24
ASA-5525-B: Cisco Adaptive Security Appliance Software Version 9.1(2) Device Manager Version 7.1(3) outside IP: X.X.X.X inside network X: 172.16.1.0/24
I have settings for Site-to-Site and Client VPN on both devices:
ASA-5525-A# show running-config crypto crypto ipsec ikev1 transform-set remote esp-3des esp-md5-hmac crypto ipsec ikev1 transform-set l2l esp-3des esp-md5-hmac crypto dynamic-map remote 10 set ikev1 transform-set remote crypto dynamic-map remote 10 set reverse-route crypto map remote 10 ipsec-isakmp dynamic remote crypto map remote 100 match address l2l_list crypto map remote 100 set peer X.X.X.X crypto map remote 100 set ikev1 transform-set l2l crypto map remote 100 set reverse-route crypto map remote interface outside crypto ikev1 enable outside crypto ikev1 policy 10 authentication pre-share encryption 3des hash sha group 2 lifetime 86400 crypto ikev1 policy 20 authentication pre-share encryption des hash sha group 2 lifetime 86400 crypto ikev1 policy 100 authentication pre-share encryption 3des hash sha group 2 lifetime 14400 crypto ikev1 policy 65535 authentication pre-share encryption 3des hash sha group 2 lifetime 86400
ASA-5525-B# show running-config crypto crypto ipsec ikev1 transform-set remote esp-3des esp-md5-hmac crypto ipsec ikev1 transform-set l2l esp-3des esp-md5-hmac crypto ipsec security-association pmtu-aging infinite crypto dynamic-map remote 10 set ikev1 transform-set remote crypto dynamic-map remote 10 set reverse-route crypto map remote 10 ipsec-isakmp dynamic remote crypto map remote 100 match address l2l_list crypto map remote 100 set peer Y.Y.Y.Y crypto map remote 100 set ikev1 transform-set l2l crypto map remote 100 set reverse-route crypto map remote interface outside crypto ca trustpool policy crypto ikev1 enable outside crypto ikev1 policy 10 authentication pre-share encryption 3des hash sha group 2 lifetime 86400 crypto ikev1 policy 20 authentication pre-share encryption des hash sha group 2 lifetime 86400 crypto ikev1 policy 100 authentication pre-share encryption 3des hash sha group 2 lifetime 14400
The ACL l2l_list:
access-list l2l_list extended permit ip object-group Y object-group X
and a mirror one on another ASA.
when I try ping from 192.168.200.100 to 172.16.1.8, the IPSec SA established but no icmp responding:
Crypto map tag: remote, seq num: 100, local addr: Y.Y.Y.Y access-list l2l_list extended permit ip 192.168.200.0 255.255.255.0 172.16.1.0 255.255.255.0 local ident (addr/mask/prot/port): (192.168.200.0/255.255.255.0/0/0) remote ident (addr/mask/prot/port): (172.16.1.0/255.255.255.0/0/0) current_peer: X.X.X.X #pkts encaps: 10, #pkts encrypt: 14, #pkts digest: 14 #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 14, #pkts comp failed: 0, #pkts decomp failed: 0 #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0 #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0 #send errors: 0, #recv errors: 0 local crypto endpt.: Y.Y.Y.Y/0, remote crypto endpt.: X.X.X.X/0 path mtu 1500, ipsec overhead 58, media mtu 1500 current outbound spi: 3A2E1DB8 current inbound spi : 23A0CA7D inbound esp sas: spi: 0x23A0CA7D (597740157) transform: esp-3des esp-md5-hmac no compression in use settings ={L2L, Tunnel, } slot: 0, conn_id: 5865472, crypto-map: remote sa timing: remaining key lifetime (kB/sec): (4374000/28786) IV size: 8 bytes replay detection support: Y Anti replay bitmap: 0x00000000 0x00000001 outbound esp sas: spi: 0x3A2E1DB8 (976100792) transform: esp-3des esp-md5-hmac no compression in use settings ={L2L, Tunnel, } slot: 0, conn_id: 5865472, crypto-map: remote sa timing: remaining key lifetime (kB/sec): (4373998/28783) IV size: 8 bytes replay detection support: Y Anti replay bitmap: 0x00000000 0x00000001
Crypto map tag: remote, seq num: 10, local addr: X.X.X.X local ident (addr/mask/prot/port): (172.16.1.0/255.255.255.0/0/0) remote ident (addr/mask/prot/port): (192.168.200.0/255.255.255.0/0/0) current_peer: Y.Y.Y.Y #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0 #pkts decaps: 74, #pkts decrypt: 74, #pkts verify: 74 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0 #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0 #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0 #TFC rcvd: 0, #TFC sent: 0 #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0 #send errors: 0, #recv errors: 0 local crypto endpt.: X.X.X.X/0, remote crypto endpt.: Y.Y.Y.Y/0 path mtu 1500, ipsec overhead 58(36), media mtu 1500 PMTU time remaining (sec): 0, DF policy: copy-df ICMP error validation: disabled, TFC packets: disabled current outbound spi: 23A0CA7D current inbound spi : 3A2E1DB8 inbound esp sas: spi: 0x3A2E1DB8 (976100792) transform: esp-3des esp-md5-hmac no compression in use settings ={L2L, Tunnel, IKEv1, } slot: 0, conn_id: 24215552, crypto-map: remote sa timing: remaining key lifetime (kB/sec): (3914993/28724) IV size: 8 bytes replay detection support: Y Anti replay bitmap: 0xFFFFFFFF 0xFFFFFFFF outbound esp sas: spi: 0x23A0CA7D (597740157) transform: esp-3des esp-md5-hmac no compression in use settings ={L2L, Tunnel, IKEv1, } slot: 0, conn_id: 24215552, crypto-map: remote sa timing: remaining key lifetime (kB/sec): (3915000/28724) IV size: 8 bytes replay detection support: Y Anti replay bitmap: 0x00000000 0x00000001
As you can see above, there are pkts encrypt on ASA-A and pkts decrypt on ASA-B.
So I run packet-tracer, and got ALLOW on ASA-A but DROP on ASA-B:
ASA-5525-A# packet-tracer input inside icmp 192.168.200.100 8 0 172.16.1.8 detailed
Phase: 1 Type: ROUTE-LOOKUP Subtype: input Result: ALLOW Config: Additional Information: in 0.0.0.0 0.0.0.0 outside Phase: 2 Type: IP-OPTIONS Subtype: Result: ALLOW Config: Additional Information: Forward Flow based lookup yields rule: in id=0x7fff37815e60, priority=0, domain=inspect-ip-options, deny=true hits=49557613, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0 src ip/id=0.0.0.0, mask=0.0.0.0, port=0 dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0 input_ifc=inside, output_ifc=any Phase: 3 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: class-map inspection_default match default-inspection-traffic policy-map global_policy class inspection_default inspect icmp service-policy global_policy global Additional Information: Forward Flow based lookup yields rule: in id=0x7fff3824cdc0, priority=70, domain=inspect-icmp, deny=false hits=55282, user_data=0x7fff3824af30, cs_id=0x0, use_real_addr, flags=0x0, protocol=1 src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0 dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, dscp=0x0 input_ifc=inside, output_ifc=any Phase: 4 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: Additional Information: Forward Flow based lookup yields rule: in id=0x7fff37815880, priority=66, domain=inspect-icmp-error, deny=false hits=55358, user_data=0x7fff37814df0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1 src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0 dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, dscp=0x0 input_ifc=inside, output_ifc=any Phase: 5 Type: DEBUG-ICMP Subtype: Result: ALLOW Config: Additional Information: Forward Flow based lookup yields rule: in id=0x7fff384894e0, priority=13, domain=debug-icmp-trace, deny=false hits=127989, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=1 src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0 dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, dscp=0x0 input_ifc=inside, output_ifc=any Phase: 6 Type: NAT Subtype: Result: ALLOW Config: nat (inside,outside) source static Y Y destination static X X no-proxy-arp route-lookup Additional Information: Static translate 192.168.200.100/0 to 192.168.200.100/0 Forward Flow based lookup yields rule: in id=0x7fff388a0770, priority=6, domain=nat, deny=false hits=421, user_data=0x7fff38488f50, cs_id=0x0, use_real_addr, flags=0x0, protocol=0 src ip/id=192.168.200.0, mask=255.255.255.0, port=0 dst ip/id=172.16.1.0, mask=255.255.255.0, port=0, dscp=0x0 input_ifc=inside, output_ifc=outside Phase: 7 Type: VPN Subtype: encrypt Result: ALLOW Config: Additional Information: Forward Flow based lookup yields rule: out id=0x7fff3892c140, priority=70, domain=encrypt, deny=false hits=2, user_data=0x4969e4, cs_id=0x7fff388cd430, reverse, flags=0x0, protocol=0 src ip/id=192.168.200.0, mask=255.255.255.0, port=0 dst ip/id=172.16.1.0, mask=255.255.255.0, port=0, dscp=0x0 input_ifc=any, output_ifc=outside Phase: 8 Type: VPN Subtype: ipsec-tunnel-flow Result: ALLOW Config: Additional Information: Reverse Flow based lookup yields rule: in id=0x7fff3892b730, priority=69, domain=ipsec-tunnel-flow, deny=false hits=2, user_data=0x499544, cs_id=0x0, reverse, flags=0x0, protocol=0 src ip/id=172.16.1.0, mask=255.255.255.0, port=0 dst ip/id=192.168.200.0, mask=255.255.255.0, port=0, dscp=0x0 input_ifc=outside, output_ifc=any Phase: 9 Type: DEBUG-ICMP Subtype: Result: ALLOW Config: Additional Information: Reverse Flow based lookup yields rule: in id=0x7fff386d6780, priority=13, domain=debug-icmp-trace, deny=false hits=227724, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=1 src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0 dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, dscp=0x0 input_ifc=outside, output_ifc=any Phase: 10 Type: IP-OPTIONS Subtype: Result: ALLOW Config: Additional Information: Reverse Flow based lookup yields rule: in id=0x7fff37867c60, priority=0, domain=inspect-ip-options, deny=true hits=48610462, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0 src ip/id=0.0.0.0, mask=0.0.0.0, port=0 dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0 input_ifc=outside, output_ifc=any Phase: 11 Type: FLOW-CREATION Subtype: Result: ALLOW Config: Additional Information: New flow created with id 50623807, packet dispatched to next module Module information for forward flow ... snp_fp_tracer_drop snp_fp_inspect_ip_options snp_fp_inspect_icmp snp_fp_translate snp_fp_dbg_icmp snp_fp_adjacency snp_fp_encrypt snp_fp_fragment snp_ifc_stat Module information for reverse flow ... snp_fp_tracer_drop snp_fp_inspect_ip_options snp_fp_ipsec_tunnel_flow snp_fp_translate snp_fp_inspect_icmp snp_fp_dbg_icmp snp_fp_adjacency snp_fp_fragment snp_ifc_stat Result: input-interface: inside input-status: up input-line-status: up output-interface: outside output-status: up output-line-status: up Action: allow
ASA-5525-B# packet-tracer input inside icmp 172.16.1.8 8 0 192.168.200.100 detailed Phase: 1 Type: ROUTE-LOOKUP Subtype: input Result: ALLOW Config: Additional Information: in 192.168.200.0 255.255.255.0 outside Phase: 2 Type: UN-NAT Subtype: static Result: ALLOW Config: nat (inside,outside) source static X X destination static Y Y no-proxy-arp route-lookup Additional Information: NAT divert to egress interface outside Untranslate 192.168.200.100/0 to 192.168.200.100/0 Phase: 3 Type: NAT Subtype: Result: ALLOW Config: nat (inside,outside) source static X X destination static Y Y no-proxy-arp route-lookup Additional Information: Static translate 172.16.1.8/0 to 172.16.1.8/0 Forward Flow based lookup yields rule: in id=0x7fff37083aa0, priority=6, domain=nat, deny=false hits=99, user_data=0x7fff33d56110, cs_id=0x0, flags=0x0, protocol=0 src ip/id=172.16.1.0, mask=255.255.255.0, port=0, tag=0 dst ip/id=192.168.200.0, mask=255.255.255.0, port=0, tag=0, dscp=0x0 input_ifc=inside, output_ifc=outside Phase: 4 Type: NAT Subtype: per-session Result: ALLOW Config: Additional Information: Forward Flow based lookup yields rule: in id=0x7fff32fc1830, priority=0, domain=nat-per-session, deny=true hits=1795177580, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0 src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0 input_ifc=any, output_ifc=any Phase: 5 Type: IP-OPTIONS Subtype: Result: ALLOW Config: Additional Information: Forward Flow based lookup yields rule: in id=0x7fff33b10af0, priority=0, domain=inspect-ip-options, deny=true hits=7026612090, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0 src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0 dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0 input_ifc=inside, output_ifc=any Phase: 6 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: class-map inspection_default match default-inspection-traffic policy-map global_policy class inspection_default inspect icmp service-policy global_policy global Additional Information: Forward Flow based lookup yields rule: in id=0x7fff349ef4b0, priority=70, domain=inspect-icmp, deny=false hits=3382798, user_data=0x7fff349ed850, cs_id=0x0, use_real_addr, flags=0x0, protocol=1 src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0 dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0 input_ifc=inside, output_ifc=any Phase: 7 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: Additional Information: Forward Flow based lookup yields rule: in id=0x7fff33b10420, priority=66, domain=inspect-icmp-error, deny=false hits=3826653, user_data=0x7fff33b0f990, cs_id=0x0, use_real_addr, flags=0x0, protocol=1 src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0 dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0 input_ifc=inside, output_ifc=any Phase: 8 Type: VPN Subtype: encrypt Result: DROP Config: Additional Information: Forward Flow based lookup yields rule: out id=0x7fff34dd44a0, priority=70, domain=encrypt, deny=false hits=195, user_data=0x0, cs_id=0x7fff37081210, reverse, flags=0x0, protocol=0 src ip/id=172.16.1.0, mask=255.255.255.0, port=0, tag=0 dst ip/id=192.168.200.0, mask=255.255.255.0, port=0, tag=0, dscp=0x0 input_ifc=any, output_ifc=outside Result: input-interface: inside input-status: up input-line-status: up output-interface: outside output-status: up output-line-status: up Action: drop Drop-reason: (acl-drop) Flow is denied by configured rule
It seemed that the acl I set on ASA-B didn't work...
So I manually do the match again on ASA-B when the IPSec SA already established and the ping is still runinng on 192.168.200.100:
ASA-5525-B(config)# crypto map remote 100 match address l2l_list
And amazing things happened...
[root@192.168.200.100 ~]# ping 172.16.1.8 PING 172.16.1.8 (172.16.1.8) 56(84) bytes of data. 64 bytes from 172.16.1.8: icmp_seq=283 ttl=62 time=14.4 ms 64 bytes from 172.16.1.8: icmp_seq=284 ttl=62 time=8.93 ms 64 bytes from 172.16.1.8: icmp_seq=285 ttl=62 time=12.8 ms 64 bytes from 172.16.1.8: icmp_seq=286 ttl=62 time=4.32 ms 64 bytes from 172.16.1.8: icmp_seq=287 ttl=62 time=15.0 ms 64 bytes from 172.16.1.8: icmp_seq=288 ttl=62 time=5.14 ms 64 bytes from 172.16.1.8: icmp_seq=289 ttl=62 time=6.28 ms 64 bytes from 172.16.1.8: icmp_seq=290 ttl=62 time=12.1 ms 64 bytes from 172.16.1.8: icmp_seq=291 ttl=62 time=13.0 ms 64 bytes from 172.16.1.8: icmp_seq=292 ttl=62 time=7.60 ms 64 bytes from 172.16.1.8: icmp_seq=293 ttl=62 time=7.63 ms ^C --- 172.16.1.8 ping statistics --- 293 packets transmitted, 11 received, 96% packet loss, time 292019ms rtt min/avg/max/mdev = 4.325/9.772/15.087/3.680 ms [root@192.168.200.100 ~]#
Everything suddenly goes fine... until the IPSec SA renegotiate.
So anyone has any idea? Is this a known bug?
08-30-2021 12:48 PM
Hi @Seazon,
You are running really old IOS versions (8.6 and 9.1), which are EoL long time ago. Discussing about bugs with versions so old is really ungrateful, as there were too many bugs and fixes in between. You should upgrade ASAP, and then test process again, to see if you see same behavior.
Btw, if you ran packet-tracer for the first time, while tunnel was not up and running, it will always show drop in VPN phase. You can just repeat packet-tracer again right after, and next time it would be allowed.
BR,
Milos
08-30-2021 07:25 PM
Thanks @Milos_Jovanovic , will try the upgrade
08-31-2021 01:17 AM
running old version noted. but you also running a legacy VPN setting which are deprecated to use in production network. move away from these weak encryption.
crypto ikev1 policy 20 authentication pre-share encryption des hash sha group 2 lifetime 86400 crypto ikev1 policy 100 authentication pre-share encryption 3des hash sha group 2 lifetime 14400
Des and 3Des must be avoided in production network. Also DH group 1/2/5 must be avoided in network.
here is the document for your guide.
Crypto map tag: remote, seq num: 100, local addr: Y.Y.Y.Y access-list l2l_list extended permit ip 192.168.200.0 255.255.255.0 172.16.1.0 255.255.255.0 local ident (addr/mask/prot/port): (192.168.200.0/255.255.255.0/0/0) remote ident (addr/mask/prot/port): (172.16.1.0/255.255.255.0/0/0) current_peer: X.X.X.X #pkts encaps: 10, #pkts encrypt: 14, #pkts digest: 14 #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0 Crypto map tag: remote, seq num: 10, local addr: X.X.X.X local ident (addr/mask/prot/port): (172.16.1.0/255.255.255.0/0/0) remote ident (addr/mask/prot/port): (192.168.200.0/255.255.255.0/0/0) current_peer: Y.Y.Y.Y #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0 #pkts decaps: 74, #pkts decrypt: 74, #pkts verify: 74
In the above Firewall A is encap the traffic and Firewall B is not decap. is there a routing in place for these end client. normally these kind of issue occurred when there is a routing issues between end client.
09-01-2021 03:12 AM - edited 09-01-2021 03:14 AM
Thanks for suggestion about weak encryption, I will change the settings once the IPSec SA could work normally.
And as you can see on Firewall B:
Crypto map tag: remote, seq num: 10, local addr: X.X.X.X local ident (addr/mask/prot/port): (172.16.1.0/255.255.255.0/0/0) remote ident (addr/mask/prot/port): (192.168.200.0/255.255.255.0/0/0) current_peer: Y.Y.Y.Y #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0 #pkts decaps: 74, #pkts decrypt: 74, #pkts verify: 74
It does have decaps, which means the pkts do arrive at Firewall B through the IPSec tunnel.
And if I execute the crypto match cmd manually, the traffic goes fine immediately.
So i'm sure there's no routing issue.
09-01-2021 04:28 AM
just looking again you running legacy phase 1 and phase 2 setting.
could you please share the relevent configuration inculding the NAT and crypto map on both firewalls.
could you show the output of theses command.
capture ASP type asp-drop match ip host x.x.x.x host y.y.y.y
and they sent tcp request from either side or both side (either client pc or server)
also could you give us the output NAT statement of the vpn-tunnel.
could you also capture few more information.
Firewall-A
capture VPN isakmp interface outside match ip host 1.1.1.1 (your outside interface IP) host 2.2.2.2
Firewall-B
capture VPN isakmp interface outside match ip host 2.2.2.2 (your outside interface IP) host 1.1.1.1
to offload to your computer or on flash disk
copy capture:VPN flash:VPN.pcap
also run some debug too. it wont be necessary as Capture VPN will tell us what interested traffic is agreed.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: