Cisco ASA - need to trace the traffic

[Pavelpro7]

The problem is some subnet from our backend is able to connect to public NTP server (actually to multiple public networks) through our Cisco ASA - which is strange, because there's no ACL or NAT that will allow it to pass. Trying to get detailed information with ASA's internal packet-tracer but had no luck. It just went trhough on all steps. The thing that confuses me is Phase: 7 - Type: VPN - Subtype: ipsec-tunnel-flow. Is this indicating that traffic is going by one of the ipsec tunnels? If so - how can I track which one it is? Tried to look up the id=0x7f433312d310 from output, but there's no match with any ipsec. Did someone experienced situation like this?

CISCO_ASA# packet-tracer input inside udp 1.1.1.1 123 89.109.251.21 123 detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 3.3.3.3 using egress ifc outside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_in in interface inside
access-list inside_in extended permit ip any any
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f434cd37eb0, priority=13, domain=permit, deny=false
hits=3526614492, user_data=0x7f43582ff9c0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f43354cf680, priority=0, domain=nat-per-session, deny=true
hits=9334231202, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f434d237f50, priority=0, domain=inspect-ip-options, deny=true
hits=11970178801, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 5
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f434d7d9bc0, priority=20, domain=lu, deny=false
hits=635936820, user_data=0x0, cs_id=0x0, flags=0x0, protocol=17
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 6
Type: FLOW-EXPORT
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f433accfa90, priority=18, domain=flow-export, deny=false
hits=4388888776, user_data=0x7f43385957e0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 7
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f433312d310, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=4303490516, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 8
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f433aeefd40, priority=0, domain=user-statistics, deny=false
hits=10525119331, user_data=0x7f4338592150, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=outside

Phase: 9
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f43354cf680, priority=0, domain=nat-per-session, deny=true
hits=9334231204, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 10
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f4346040e00, priority=0, domain=inspect-ip-options, deny=true
hits=8793199143, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 11
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW

Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0x7f433aef3b00, priority=0, domain=user-statistics, deny=false
hits=12544955565, user_data=0x7f4338592150, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=inside

Phase: 12
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 3784177591, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

[Milos_Jovanovic]

Hi @Pavelpro7,

Yes, this means that your traffic is going via VPN somewhere. Determining through which VPN tunnel is harder task. For that, I would normaly go via info presented in the packet tracer, but you have this:

src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any

Based on this, I would think that you have some route-based VPN, and that you should check your routing for the destination (route-based VPNs have quad zeros in crypto domain).

BR,

Milos

[Pavelpro7]

No luck with anything you advised. There's no cryptomap acl that matches these particular source/dest ip. I even analyzed the hitcounts from the packet-tracer output with hitcounts from all acl lines - they don't match.
hits=4303490516 - is the hitcount on output which I got yesterday. So today the matching acl should have increased counter, but there's only two lines that get close to it, but they are lower - (hitcnt=4254446208) and (hitcnt=4155233505).
That's a mystery.
Although have this weird cryptomap - can it cause such behavior?

 

[Milos_Jovanovic]

Analyzing ACL hit counts - I don't think it would get you somewhere.

Your crypto map is looking different than I'm used to, and it does state no traffic is selected. However, I'm not using ASDM often, so there could be I'm missing something. Could you please post CLI output of the 'show run crypto' command (make sure not to disclose some unwanted info)? Based on this, we should be able to get some starting point.

BR,

Milos

[Pavelpro7]

Here is the output. Hope it can be helpful

ASA-1# show run crypto
crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec ikev1 transform-set vpnset esp-aes esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec ikev2 ipsec-proposal SHA1-AES256
protocol esp encryption aes-256
protocol esp integrity sha-1
crypto ipsec ikev2 ipsec-proposal AES256-SHA256
protocol esp encryption aes-256
protocol esp integrity sha-256
crypto ipsec ikev2 ipsec-proposal AES256-sha256
protocol esp encryption aes-256
protocol esp integrity sha-256
crypto ipsec security-association pmtu-aging infinite
crypto dynamic-map vpndyn 10 set ikev1 transform-set vpnset ESP-3DES-SHA
crypto dynamic-map vpndyn 10 set security-association lifetime seconds 28800
crypto dynamic-map vpndyn 10 set security-association lifetime kilobytes 4608000
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set ikev1 transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map vpnmap 10 ipsec-isakmp dynamic vpndyn
crypto map vpnmap 20...
************
crypto map vpnmap 130 set security-association lifetime seconds 28800
crypto map vpnmap interface outside
crypto map vpnmap interface 5G
crypto map inside_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map inside_map interface inside
crypto ca trustpoint **ServerCA**
no validation-usage
crl configure
crypto ca trustpoint localtrust
enrollment self
fqdn bpn.HUBBA_BUBBA.moc
subject-name CN=bpn.HUBBA_BUBBA.moc
keypair sslvpnkey
crl configure
crypto ca trustpoint DigiDigDigi
enrollment terminal
crl configure
crypto ca trustpoint FRESH_HUBBA_BUBBA
keypair FRESH_HUBBA_BUBBA
no validation-usage
crl configure
crypto ca trustpoint OLD_FRESH_HUBBA_BUBBA
keypair FRESH_HUBBA_BUBBA_OLD
no validation-usage
crl configure
crypto ca trustpool policy
crypto ca cert...
************
crypto isakmp nat-traversal 30
crypto ikev2 policy 5
encryption aes-256
integrity sha256
group 19
prf sha256
lifetime seconds 86400
crypto ikev2 policy 10
encryption aes-256
integrity sha256
group 5
prf sha256
lifetime seconds 86400
crypto ikev2 policy 15
encryption aes-256
integrity sha256
group 19
prf sha256
lifetime seconds 28800
crypto ikev2 enable outside
crypto ikev1 enable outside
crypto ikev1 enable 5G
crypto ikev1 enable inside
crypto ikev1 policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
crypto ikev1 policy 65535
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400

[Pavelpro7]

No luck with anything you advised. There's no cryptomap acl that matches these particular source/dest ip. I even analyzed the hitcounts from the packet-tracer output with hitcounts from all acl lines - they don't match.
hits=4303490516 - is the hitcount on output which I got yesterday. So today the matching acl should have increased counter, but there's only two lines that get close to it, but they are lower - (hitcnt=4254446208) and (hitcnt=4155233505).
That's a mystery.
Although have this weird cryptomap - can it cause such behavior?

[Mohammed al Baqari]

Hi,

Try a trace route from test host to public IP and from the hops you should
be able to find the tunnel used. Make sure to allow trace route on asa
inspection policy.

**** please remember to rate useful posts