SSH not working with ISE

[darrenmiyamoto]

I have a problem with my SSH and cannot figure out why.  Config below.  I have some working and some not. 

There are no ACLs on the management switch. 

I can source ping from my Management VRF to the ISE servers and management IPs

I've reconfigured my crypto key (crypto key generate rsa 4096)

When I check my RADIUS live logs in ISE, Im not getting any success/failure logs

Next step is to span my management switch port and capture traffic.   

When I switch to local login, SSH works (aaa authentication login default local)

vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family

aaa new-model
!
!
aaa group server radius ISE
server name ISE01
server name ISE02
ip radius source-interface GigabitEthernet0/0
!
aaa authentication login default group ISE local

interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
ip address 10.0.0.100 255.255.255.0
negotiation auto

radius server ISE01
address ipv4 10.0.0.1 auth-port 1645 acct-port 1646
timeout 10
retransmit 5
key MYKEY
!
radius server ISE02
address ipv4 10.0.0.2 auth-port 1645 acct-port 1646
timeout 10
retransmit 5
key MYKEY2

*********************************************************************************

debug logs - AS NOTED ABOVE, I CAN PING THE ISE SERVERS BUT IT STATES THEY ARE HAVING ISSUES.  NO ACL ON ISE

*********************************************************************************

May 19 01:14:02.349: SSH1: protocol version id is - SSH-2.0-OpenSSH_for_Windows_8.1
May 19 01:14:02.351: SSH2 1: Using kex_algo = ecdh-sha2-nistp256
May 19 01:14:02.732: %SSH-5-SSH2_SESSION: SSH2 Session request from 10.0.0.251 (tty = 1) using crypto cipher 'aes128-ctr', hmac 'hmac-sha2-256-etm@openssh.com' Succeeded
ACCESS_SWITCH#
May 19 01:14:02.958: AAA/BIND(00000051): Bind i/f
May 19 01:14:02.958: AAA/AUTHEN/LOGIN (00000051): Pick method list 'default'
ACCESS_SWITCH#
May 19 01:14:08.676: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: USERNAME] [Source: 10.0.0.251] [localport: 22] at 09:14:08 SGT Thu May 19 2022
May 19 01:14:08.676: %SSH-5-SSH2_USERAUTH: User 'USERNAME' authentication for SSH2 Session from 10.0.0.251 (tty = 1) using crypto cipher 'aes128-ctr', hmac 'hmac-sha2-256-etm@openssh.com' Succeeded

May 19 01:17:25.345: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: ADUSER] [Source: 10.0.0.251] [localport: 22] [Reason: Login Authentication Failed] at 09:17:25 SGT Thu May 19 2022
ACCESS_SWITCH#
May 19 01:17:25.348: AAA/AUTHEN/LOGIN (00000053): Pick method list 'default'
May 19 01:18:40.157: AAA/SG/TEST: server test info not found
May 19 01:18:40.157: AAA/SG/TEST: server test info not found

May 19 01:19:36.818: SSH1: protocol version id is - SSH-2.0-OpenSSH_for_Windows_8.1
May 19 01:19:36.820: SSH2 1: Using kex_algo = ecdh-sha2-nistp256
ACCESS_SWITCH#
May 19 01:19:37.201: %SSH-5-SSH2_SESSION: SSH2 Session request from 10.0.0.251 (tty = 1) using crypto cipher 'aes128-ctr', hmac 'hmac-sha2-256-etm@openssh.com' Succeeded
ACCESS_SWITCH#
May 19 01:19:37.424: AAA/BIND(00000054): Bind i/f
May 19 01:19:37.424: AAA/AUTHEN/LOGIN (00000054): Pick method list 'default'
ACCESS_SWITCH#
May 19 01:20:02.302: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.0.1:1645,1646 is not responding.
May 19 01:20:02.302: AAA/SG/TEST: server test info not found
ACCESS_SWITCH#
May 19 01:20:02.303: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.0.1:1645,1646 is being marked alive.
ACCESS_SWITCH#
May 19 01:21:02.480: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.0.2:1645,1646 is not responding.
May 19 01:21:02.480: AAA/SG/TEST: server test info not found
ACCESS_SWITCH#
May 19 01:21:02.480: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.0.2:1645,1646 is being marked alive.
ACCESS_SWITCH#
May 19 01:21:44.609: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: ADUSER] [Source: 10.0.0.251] [localport: 22] [Reason: Login Authentication Failed] at 09:21:44 SGT Thu May 19 2022

[balaji.bandi]

what switch mode and what code running:  Make sure both the side Key Matches device and ISE side

https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/200208-Configure-ISE-2-0-IOS-TACACS-Authentic.html

[darrenmiyamoto]

switch is a 9300.  IOS is 17.03.05

The shared secret keys match.  I duplicated them from one source and made 8 network devices based on that one original.  7 of the 8 work.  this is the single one that doesnt work.  I beleive Id get shared sercret mismatch errors for my ISE live logs but Im not getting a single pass/fail.  This leads me to believe something is wrong with the communication.  As seen in the logs above, ISE is being marked as dead the being marked as alive. There is no loop to ISE and I can test connectivity with source ping.

[carstenlymann1]

Wich version of ISE  ?

There is a bug in 3.1 patch1 -> CSCwa59924

Where SSH wont work.. It is fixed in patch3

[darrenmiyamoto]

I was on Version: 3.1.0.518 Patch 1.  I just upgraded to Patch 3.  Still no joy

[darrenmiyamoto]

I just read this bug report.  This is issues when you SSH into ISE.  I'm using ISE to authenticate my users to AD for SSH into my switches.  

[carstenlymann1]

ahh okay.. So its TACACS+ ??  From what i understand from first post, some users are able to login and some is not. 

Saying to me your solution is generally working. Could it be that the users that is not working, is not in the same security group as the others that is working ? Or if looking on title, if there could be a spelling error or something like that? 

[darrenmiyamoto]

Im using RADIUS.

My solution is 100% working on the other switches in my network.  This single switch fails 100% of the time.  There is a communication issue somewhere, thus the logs that state my RADIUS server is marked as DEAD/ALIVE.  I should see some hits on my ISE live logs but there is no AAA messages coming from the one switch in question.  

The user (me) is the same thus my Security Group should not be in question.  Spelling error is not the case.  I took a working config and did a compare in Notepad++ and there are no differences in the AAA config.

[ahollifield]

Routing issue?  Is this switch at a different location?  Firewall rules? ip radius source-interface?

[darrenmiyamoto]

Not a routing issue.  I can source ping from the Mgmt-vrf to the ISE server.  Switch is directly connected via management port to the managment switch.  I have a similar switch with pretty much the same config that works.  Same code version and everything 17.3.5...

Yes my radius is set to ip radius source-interface Gi0/0...

No ACL on the port.   I will try to default the port and re-configure and try again...

[ahollifield]

But your ip radius source-interface is get to 0/0?  But you are pinging from management VRF?  Is 0/0 part of the management VRF?  Is 0/0 a routed port?  What if you source ping to ISE from 0/0?

[darrenmiyamoto]

From the config above, (pasted below for your convenience) Gi0/0 is the management port on the rear of the switch next to the console.

I can ping from the Mgmt-vrf (which is Gi 0.0) with the source ping to anywhere on my /24 management network.  This is a flat network and no routing is needed in the OOBM as all my assets are in the same subnet.  ISE is on the management network.  It is also dual homed from the ISE appliance (second NIC) to the production where it talks to AD and authenticates my users.

interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
ip address 10.0.0.100 255.255.255.0
negotiation auto

[thomas]

As @ahollifield said, RADIUS server DEAD/ALIVE issues are typically routing/flapping if your ISE services are working 100% everywhere else.

This thread has been going for nearly 2 weeks... why not call TAC already to help you troubleshoot?

[darrenmiyamoto]

So for flapping... (not routing since there is no routing on the /24).  Its hard to believe this is the case.  This is the only switch with this issue.  If ISE were flapping, I would see that on all other switches.  My Gi0/0 cannot be flapping since its a single port on the management network.  

I tried TAC but apparently my asset doesnt have coverage.  I know we have an enterprise coverage so I need to reengage.  I have other things blowing up like vCenter thus this isnt a priority.  I have it set to local user login at the moment and no login required on console so its just a nagging issue to fix for consistency sake.

[darrenmiyamoto · Accepted solution]

REMOVE THIS:
ip radius source-interface GigabitEthernet0/0 

REPLACE WITH THIS:
ip vrf forwarding  Mgmt-vrf

VERIFY WITH THIS:
test aaa group ise-group username password legacy

Works like a champ now.............