Re: SXP between ASA and ISE fails

andy!doesnt!like!uucp · ‎08-06-2021

Guys

i need advice with subject. ASA runs 9.12.4(10), ISE is 2.1 latest patch

things before SXP setup:

1) ASA was configured for CTS & it was prepared on the ISE as NAD. ASA has been successfully able to fetch SGT from ISE. config on ASA:

aaa-server ISE protocol radius

dynamic-authorization

aaa-server NewSite_ISE protocol radius
dynamic-authorization

!below node is PAN ; actually only PAN in this setup was needed because of some circumstances

aaa-server NewSite_ISE (inside) host 10.3.5.50
key *****
authentication-port 1812
accounting-port 1813

!below node is PSN1
aaa-server NewSite_ISE (inside) host 10.3.5.52
key *****
authentication-port 1812
accounting-port 1813

!below node is PSN2
aaa-server NeewSite_ISE (inside) host 10.4.5.52
key *****
authentication-port 1812
accounting-port 1813

cts server-group ISE

2) config on ASA was rolled back because of need to do all stuff from CSM (4.19 ; and ISE integration with it only possible starting from 2.3 version).

SXP setup done from CSM (only ISE server group is defined under TrustSec/SXP Settings and SXP itself obviously) :

1) after deployment on ASA

aaa-server NewSite_ISE protocol radius
dynamic-authorization
aaa-server NewSite_ISE (inside) host 10.3.5.50
key *****
authentication-port 1812
accounting-port 1813
aaa-server NewSite_ISE (inside) host 10.3.5.52
key *****
authentication-port 1812
accounting-port 1813
aaa-server NewSite_ISE (inside) host 10.4.5.52
key *****
authentication-port 1812
accounting-port 1813
cts server-group NewSite_ISE
cts sxp enable

& this is where **bleep** started to happen - connection on ISE shown as pending ; on ASA no CTS SXP connections shown as pending and password is not set. Ok. adding default password in TrustSec / SXP Connection Peers but w/o any effect.

cts sxp default password *****

Ok. adding PSNs as peers as noticed in ASA's events monitoring (2nd PSN is talking to ASA on TCP/64999 but no actual evidences ASA is trying to talk to ISE:

cts sxp connection peer 10.4.5.52 source 10.1.20.100 password default mode local listener
cts sxp connection peer 10.3.5.52 source 10.1.20.100 password default mode local listener

turning on debug cts sxp all. result (needless to say traffic was very initially permitted with ACLs on ASA):

[cts sxp conn error]: sxp_eval_src_ip: Peer address 10.3.5.52 cannot be reached from source ip 10.1.20.100
[cts sxp conn error]: sxp_listen_sock_may_init: invalid vpifnum: 10.1.20.100
[cts sxp conn error]: sxp_eval_src_ip: Peer address 10.3.5.52 cannot be reached from source ip 10.1.20.100
[cts sxp conn error]: sh_create_conn: SXP Socket Open failed, conn index = 1
[cts sxp conn error]: cts_sxp_cfg_setup_conn: Conn request failed; ip_addr = 10.3.5.52
[cts sxp conn error]: sxp_eval_src_ip: Peer address 10.4.5.52 cannot be reached from source ip 10.1.20.100
[cts sxp conn error]: sxp_listen_sock_may_init: invalid vpifnum: 10.1.20.100
[cts sxp conn error]: sxp_eval_src_ip: Peer address 10.4.5.52 cannot be reached from source ip 10.1.20.100
[cts sxp conn error]: sh_create_conn: SXP Socket Open failed, conn index = 2
[cts sxp conn error]: cts_sxp_cfg_setup_conn: Conn request failed; ip_addr = 10.4.5.52
[cts sxp conn error]: sxp_eval_src_ip: Peer address 10.3.5.52 cannot be reached from source ip 10.1.20.100
[cts sxp conn error]: sxp_eval_src_ip: Peer address 10.4.5.52 cannot be reached from source ip 10.1.20.100
[cts sxp conn error]: sxp_eval_src_ip: Peer address 10.3.5.52 cannot be reached from source ip 10.1.20.100
[cts sxp conn error]: sxp_eval_src_ip: Peer address 10.4.5.52 cannot be reached from source ip 10.1.20.100
[cts sxp conn error]: sxp_eval_src_ip: Peer address 10.3.5.52 cannot be reached from source ip 10.1.20.100

Damien Miller · ‎08-06-2021

You don't show it but I'm assuming ISE is already set up with two SXP speaker connections for the ASA and the persona/role is enabled on both ISE nodes?

The log message you shared indicated the ASA is not able to reach ISE from the source IP cannot be reached from source ip 10.1.20.100 (routing) or the traffic is not getting back to the ASA IP cannot be reached from source ip 10.1.20.100 (ACL/FW policy). If you do a packet tracer on the CLI or ASDM specifying source cannot be reached from source ip 10.1.20.100, destination 10.3.5.52 TCP 64999, does it get dropped?

You can also tcpdump 10.3.5.52 with the filter "ip host 10.1.20.100" from the PAN GUI, should show you if the SXP connection reachability is ok.

andy!doesnt!like!uucp · ‎08-06-2021

Hi Damien

in fact there is a rule allowing to communicate ASA IP 10.1.20.100 with ISE's nodes. But it's quite strange that packet-tracer fails TCP with src of active ASA (10.1.20.100) to any external IP (actually ISE is just behind L2L tunnel). In fact none of allowed by ACLs TCP/64999 to ISE's IP is successful if it's originated from Active ASA's IP within packet-tracer. It's successful from other IPs of subnets in encryption domain. it's successful from standby ASA's IP but not from active. Next action for me is to take tcpdump from relevant ISE's nodes. But i have a feeling it wont change much.

UPD:

tcpdump on PSN shows SYN & its retransmits to ASA's IP.

with capture on ASA i cannot catch traffic of interest except of permitted TCP/64999 from PSN but attempts in opposite direction

and with "show asp table socket" i cannot see ASA listening on TCP/64999.

10.1.20.100 is inside interface

UPD: ASA reconfigured with below delta w/o improvement

cts sxp default source-ip 10.1.20.100
cts sxp connection peer 10.4.5.52 password default mode local listener
cts sxp connection peer 10.3.5.52 password default mode local listener

hslai · ‎08-14-2021

> with "show asp table socket" i cannot see ASA listening on TCP/64999.

Is SXP enabled in ASA? If so, try disabling it ("no cts sxp enable") and then re-enabling it ("cts sxp enable"). If that does not do it, then please engage TAC to troubleshoot on the ASA.

andy!doesnt!like!uucp · ‎08-14-2021

Hi

sxp was obviously enabled. disabling/reenabling didnt help (

will call TAC by chance.

andrewswanson · ‎08-15-2021

Hi

have you tried adding the following to your ASA to permit SSH from ISE:

ssh <ISE IP> inside

I ran into issues with ISE/ASA/SXP on an old 5500 and found the following:

"Some platforms do not support Cisco ISE's "Push" feature for Change of Authorization (CoA), for example: some versions of the Nexus network device. For this case, ISE will connect to the network device and make it to trigger an updated configuration request towards ISE. To achieve this, ISE opens an SSHv2 tunnel to the network device, and the Cisco ISE sends a command that triggers a refresh of the TrustSec policy matrix."

https://www.cisco.com/c/en/us/td/docs/security/ise/2-1/admin_guide/b_ise_admin_guide_21/b_ise_admin_guide_20_chapter_010111.html

hth
Andy

andy!doesnt!like!uucp · ‎08-16-2021

Hi

will try by case but i'm not sure how switching from CoA to SSH on ISE in Adv TrustSec settings can make enforce ASA to listen on SXP port or rectify it from failure to initiate SXP session toward ISE.

andy!doesnt!like!uucp · ‎08-23-2021

Hi

i've configured ASA's admin creds under Device Configuration Deployment but ISE reports failure:

ERROR: ASA-IP(10.1.20.100) : Could not connect: Device not reachable

Though with tcpdump on ISE i can see the ASA resets session at once after last ACK:

1 2021-08-23 11:14:27.923167 10.3.5.50 10.1.20.100 TCP 74 61858 → 22 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=1593744651 TSecr=0 WS=128
2 2021-08-23 11:14:27.949282 10.1.20.100 10.3.5.50 TCP 60 22 → 61858 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1360
3 2021-08-23 11:14:27.949303 10.3.5.50 10.1.20.100 TCP 54 61858 → 22 [ACK] Seq=1 Ack=1 Win=29200 Len=0
4 2021-08-23 11:14:27.949394 10.3.5.50 10.1.20.100 SSHv2 74 Client: Protocol (SSH-2.0-JSCH-0.1.51)
5 2021-08-23 11:14:27.975282 10.1.20.100 10.3.5.50 TCP 60 22 → 61858 [ACK] Seq=1 Ack=21 Win=32768 Len=0
6 2021-08-23 11:14:27.975331 10.1.20.100 10.3.5.50 TCP 60 22 → 61858 [RST] Seq=1 Win=32768 Len=0

hslai · ‎08-23-2021

Is ISE IP allowed to SSH to this ASA?

andy!doesnt!like!uucp · ‎08-23-2021

Hi

TCP handshake passes :0)

jeaves@cisco.com · ‎08-24-2021

Obvious to say but this looks to be a pure routing/access issue between ISE and the ASA, not particularly related to features like SXP. If you log into the ISE CLI you should be able to SSH to the ASA: 'admin# ssh <ASA IP> <username>'. You may be prompted asking if you're sure, if there are perhaps V1/V2 discrepancies with RSA key fingerprint for example, but if routing is good then answering that you're sure should let you in. If you do have an interaction with answering yes or no then ISE itself wouldn't handle that which may be the reason that adding IP:SGT mappings via SSH may not work. If there's no interaction then you should be able to add mappings via ISE using SSH although I agree with the comments above that has no bearing on why SXP isn't coming up. I would double check the access rules and statements similar to 'access-group outside_access_in in interface outside' to check if the right suitable policies are in place.

andy!doesnt!like!uucp · ‎08-24-2021

Hi

of course it has nothing to do with SXP topic but it's just next penny to ASA-ISE integration problems :0)

i dont think it's either interaction or (especially) routing problem.

When i do SSH from ISE i receive reset immediately:

ISE/admin# ssh 10.1.20.100 admin version 2
ssh_exchange_identification: read: Connection reset by peer
ISE/admin#

In parallel tcp dump on ISE caught below about above session (FW sends RST after ISE presents its client version)

1 2021-08-24 12:48:52.144757 10.3.5.50 10.1.20.100 TCP 74 19744 → 22 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=1685808872 TSecr=0 WS=128
2 2021-08-24 12:48:52.171040 10.1.20.100 10.3.5.50 TCP 60 22 → 19744 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1360
3 2021-08-24 12:48:52.171087 10.3.5.50 10.1.20.100 TCP 54 19744 → 22 [ACK] Seq=1 Ack=1 Win=29200 Len=0
4 2021-08-24 12:48:52.171359 10.3.5.50 10.1.20.100 SSHv2 75 Client: Protocol (SSH-2.0-OpenSSH_6.6)
5 2021-08-24 12:48:52.197532 10.1.20.100 10.3.5.50 TCP 60 22 → 19744 [RST] Seq=1 Win=32768 Len=0

In another one parallel CSM event viewer registers flow on exactly target FWs (FWHQ on the ISE side & FWNEWSITE)

Receive Time Severity Event Type ID Event Name Device Source Source User Identity Source Service Destination Destination FQDN Destination Service Direction Protocol Action Connection ID Policy Map Class Map ACL Name Description Event ID
8/24/21 12:43:51 PM Error 302014 Teardown TCP FWNEWSITE 10.3.5.50 tcp/19164 FWNEWSITE_10.1.20.100 tcp/22 tcp teardown 9010803 Teardown tcp connection 9010803 for Outside.1010:10.3.5.50/19164 to identity:10.1.20.100/22 duration 0:00:00 bytes 0 TCP Reset by appliance 543563431472
8/24/21 12:43:51 PM Error 302014 Teardown TCP FW-HQ 10.3.5.50 tcp/19164 FWNEWSITE_10.1.20.100 tcp/22 tcp teardown 2765751720 Teardown tcp connection 2765751720 for inside.20:10.3.5.50/19164 to outside.900:10.1.20.100/22 duration 0:00:00 bytes 21 TCP Reset-O from outside.900 543563431468
8/24/21 12:43:51 PM Error 302013 Built TCP FWNEWSITE 10.3.5.50 tcp/19164 FWNEWSITE_10.1.20.100 tcp/22 inbound tcp built 9010803 Built inbound tcp connection 9010803 for Outside.1010:10.3.5.50/19164 (10.3.5.50/19164) to identity:10.1.20.100/22 (10.1.20.100/22) 543563431461
8/24/21 12:43:51 PM Error 302014 Teardown TCP FWNEWSITE 10.3.5.50 tcp/19164 FWNEWSITE_10.1.20.100 tcp/22 tcp teardown 9010801 Teardown tcp connection 9010801 for Outside.1010:10.3.5.50/19164 to identity:10.1.20.100/22 duration 0:00:00 bytes 0 Flow terminated by TCP Intercept 543563431127
8/24/21 12:43:51 PM Error 302013 Built TCP FWNEWSITE 10.3.5.50 tcp/19164 FWNEWSITE_10.1.20.100 tcp/22 inbound tcp built 9010801 Built inbound tcp connection 9010801 for Outside.1010:10.3.5.50/19164 (10.3.5.50/19164) to identity:10.1.20.100/22 (10.1.20.100/22) 543563431125
8/24/21 12:43:51 PM Error 302013 Built TCP FW-HQ 10.3.5.50 tcp/19164 FWNEWSITE_10.1.20.100 tcp/22 inbound tcp built 2765751720 Built inbound tcp connection 2765751720 for inside.20:10.3.5.50/19164 (10.3.5.50/19164) to outside.900:10.1.20.100/22 (10.1.20.100/22) 543563430873

What i can see though is ISE doesnt match ciphers with C9.3K switches on new site:

ISE/admin# ssh 10.1.20.1 admin
Failed to establish session with 10.1.20.1
no matching cipher found with 10.1.20.1: client aes256-cbc,aes128-cbc,aes128-gcm@openssh.com,aes256-gcm@openssh.com server aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
ISE/admin# ssh 10.1.16.18 admin
Failed to establish session with 10.1.16.18
no matching cipher found with 10.1.16.18: client aes256-cbc,aes128-cbc,aes128-gcm@openssh.com,aes256-gcm@openssh.com server aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr

capture on FWNEWSITE/ (capture ISE real-time match ip 10.3.5.50) catches nothing

hslai · ‎08-24-2021

> .. tcp connection 2765751720 for inside.20:10.3.5.50/19164 to outside.900:10.1.20.100/22 ...

Should it not be inside to inside?

The reset is due to Configuring TCP Options

Try (Cisco Community) Troubleshooting Access Problems Using Packet-Tracer ?