Can neither SSH or Console to 9500-48Y4C

[Bahman]

Hi,

I have recently set up 2 sets of c9500 switches (stackwise) for a customer. The first site runs c9500-16x and the other site does c9500-48Y4C. Everything worked very well for the first site without any issues. The second site with switches (c9500-48Y4C), has connection issue wih SSH and Console to active(master) switch. I get SSH prompt but it seems that the authentication for both SSH and Console not working.

I can only log in into the standby switch via console port however I am not able to run any commands except show commands in privileged mode. Ping is successful between ISE and c9500-48Y4C.

I remember that this issue showed up after adding these commands on active switch:

aaa group server radius ISE
server name ise01
server name ise02

ISE is running 2.6.0.156 which is compatible with the switches IOS which is 17.06.03 according:

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-6/release_notes/ol-17-6-9500.html#concept_ryg_xtd_ddb

Does anyone know what could be behind this issue?

[pieterh]

above commands alone are not enough to make ISE the cause of the problem
you need additional commands to make authentication/authorization go through ISE.

it is not unexpected expected that the standby switch has limited commands available
-> make the console connection to the master switch
or do a switchover to make the current switch master
     redundancy force-switchover

[Bahman]

Thank you for your reply Pieterh.

The problem is, no other cammands than show commands working with standby switch. I have to unplug the active switch to make the standby switch become active.

I remember that during configuration of switches for this stack, I encountered with a simillar problem with Console login as soon as I added " aaa new-model" command. After adding this command I could not login via Console port again to the master (active) switch anymore so I had to remove this command from the standby switch in order to log in to the active switch. Configuration of con & vty lines are like this:

line con 0
stopbits 1
line aux 0
line vty 0 4
exec-timeout 120 0
privilege level 15
transport input ssh
line vty 5 15
exec-timeout 120 0
privilege level 15
transport input none

Do I need to add some commands here on "con & vty" lines?

[pieterh]

"aaa new-model" means you want to use userrname/password to login, not just secret +  enable secret
-> you need to create a local account first with sufficient privilege level
if you have not completed this yet, then you cannot login......
Configure Basic AAA on an Access Server - Cisco
Warning: The aaa new-model command immediately applies local authentication to all lines and interfaces (except console line line con 0). If a telnet session is opened to the router after this command is enabled (or if a connection times out and has to reconnect), then the user has to be authenticated with the local database of the router. It is recommended to define a username and password on the access server before you start the AAA configuration, so you are not locked out of the router. See the next code example.

in addition to this you may want to complete AAA configuration to use ISE as radius-server

Use RADIUS for Device Administration with Identity Services Engine - Cisco

[Bahman]

Local account with password has already been configuered on the switches. I forgot to mention that c9500 switches replaced the customer's old core switche models (catalyst 4500) so I actually have copied all AAA configuration from the old core switches to the new core switches. Therefore I wonder if there is some missing or extra commands which cause this issue? Also why it has only affected Console login on active switch and not Standby switch?

AAA configuration:

aaa authentication login default group ISE local
aaa authentication dot1x default group ISE
aaa authorization console
aaa authorization exec default group ISE local
aaa authorization network default group ISE
aaa accounting dot1x default start-stop group ISE
aaa accounting exec default start-stop group ISE
aaa accounting network default start-stop group ISE

aaa group server radius ISE
server name ise01
server name ise02

[pieterh]

ok these AAA config will help

1) did you also copied coresponding  radius-server configuration ?
radius server ise01 
address ipv4 xxx.xxx.xxx.xxx auth-port 1812 acct-port 1813
timeout 4
retransmit 3
key 7 *****************

2) has the ip-address of the switch changed from the old switch?
   then you may also look into the ISE configuration to mark the new switch as an ISE-client

[Bahman]

1) Yes, configuration for AAA, ISE and line con & VTY on new switches are identical with old ones except "key 7" which is deprecated and has been configured with key 6. 

2) We use the same IP address and DNS name for the new switches so there shouldn't be any issue with ise connection. When I run a putty SSH connection to the new switches I get SSH prompt, however, authentication doesn't work. The only difference in configuration between site 1 with c9500-16x and site 2 with c9500-48Y4C is:

site1:                                                             site2:

line vty 5 15                                                    line vty 5 15
transport input ssh                                          transport input none

Could it be the cause of this issue?

[pieterh]

no this is not necessarily a problem; this only limits the number of simultaneous VTY sessions to 1-4

maybe these commands can help
line con 0
   authorization exec ISE
   login authentication ISE
line vty 0 4
   authorization exec ISE
   login authentication ISE

[Bahman]

I tired these commands as well but it didn't work. I noticed that during connection by SSH to switch, these message showed up on CLI:

%RADIUS-3-NOSERVERS: No Radius hosts configured or no valid server present in the server group ISE

I removed och reconfigured whole aaa & radius configuration but I still get "%RADIUS-3-NOSERVERS" message while trying to SSH to switches.

[pieterh]

please post the whole configuration, 
i have no Idea what your current configuration is, and which lines you added

>>> %RADIUS-3-NOSERVERS: No Radius hosts configured or no valid server present in the server group ISE<<
indicates your radius-server configuration is not complete

commands needs to be a complete set,

- individual aaa servers need to be configured
- these servers are configured as member of a group
- and a group is referenced in an aaa authorization method
  NB! one line for login and one line for exec (privileged) mode
- in "line con 0" for direct console access or "line vty 1-4" for SSH access the authorization method are referenced
  NB! you can use different method for CONSOLE and for SSH-access (e.g. console local account only)
- the switch must be registered in your ISE-server as radius-client
   else ISE will not respond to the radius-request from the switch  -> look in the ISE logs if this occurs
for details reed the document in the link i posted

aaa authentication login default group ISE local     ! this means ISE is prefered, local accounts is fallback
aaa authorization exec default group ISE local      

aaa group server radius ISE
     server name ise01
     server name ise02

radius server ise01 
     address ipv4 <ISE01 ip-address>  auth-port 1812 acct-port 1813
     key 0 <ISE01 shared secret>                  ! keytype may vary "0" is unecrypted key other values for encrypted keys
radius server ise02
     address ipv4 <ISE02 ip-address> auth-port 1812 acct-port 1813
     key 0 <ISE02 shared secret>

line vty 0 4
   authorization exec ISE
   login authentication ISE

[Bahman]

Thank you for your reply Pieterh.

Here is current configuration for aaa & radius parts:

aaa group server radius ISE
server name ise01
server name ise02
!
aaa authentication login default group ISE local
aaa authentication dot1x default group ISE
aaa authorization console
aaa authorization exec default group ISE local
aaa authorization network default group ISE
aaa accounting dot1x default start-stop group ISE
aaa accounting exec default start-stop group ISE
aaa accounting network default start-stop group ISE
!

aaa server radius dynamic-author
client 172.16.10.1 server-key
client 172.16.10.2 server-key

radius server ise01
address ipv4 172.16.10.1 auth-port 1812 acct-port 1813
automate-tester username blabla
key
radius server hk-ise-02
address ipv4 172.16.10.2 auth-port 1812 acct-port 1813
automate-tester username blabla
key

line con 0
stopbits 1
line vty 0 4
exec-timeout 120 0
privilege level 15
transport input ssh
line vty 5 97
exec-timeout 120 0
privilege level 15
transport input ssh

As I mentioned before, the site with c9500-16X switches with same configuration work properly but the other site with c9500-48Y4C model got this issue with SSH and Console connection.

[MHM Cisco World]

only what you need for SSH or console to auth the password with radius is assing the interface use between SW and radius 
ip source interface radius 

this IP must reachable for radius server