C9800 unable to SSH via TACACS on Service Port

[stephendrkw]

Hi all,

I have setup AAA on my new C9800 Anchor WLC replacement for an old WLC 5508. Mobility tunnels are up with other Anchor and Foreign 5508's running IRCM image. Problem I have now for some reason TACACS is not working properly to Manage WLC via out of band Service Port. I would like to use TACACS Mgmt via Service Port like my 5508's.

For some reason I can login to the console port successfully using my TACACS username/password but not SSH (haven't setup http yet as there command to enable tacacs for HTTPS access)

Enter my tacacs username via SSH..........

WLC console logg -  %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:username]

C9800>en
% Error in authentication.

Below are my commands, you can see I had to add ip tacacs route to force via Service Port, inbound and outbound are working through my Firewall Cluster once I added specific route.

I did not add "aaa authorization commands" - log message saying not supported in future XE releases hidden command

aaa new-model
!
aaa group server tacacs+ TAC_EXT
server name TACACS_SVR_AUTH_ACT_ATHR_2
server name TACACS_SVR_AUTH_ACT_ATHR_3
ip vrf forwarding Mgmt-intf
!
aaa authentication login default group TAC_EXT local
aaa authorization network default group TAC_EXT local
!
ip tacacs source-interface GigabitEthernet0 vrf Mgmt-intf

ip route vrf Mgmt-intf 10.x.x.x(TACACS.SERVERS) 255.255.255.255 GigabitEthernet0 10.x.x.x (return packets on Firewall Cluster logs for UDP49 from WLC to TACACS Servers started working via SP after I added this route before this route return packets were denied)
!
ip ssh rsa keypair-name SSH-KEY
ip ssh version 2
!
tacacs server TACACS_SVR_AUTH_ACT_ATHR_2
address ipv4 10.x.x.x
key 7 *********
timeout 5
tacacs server TACACS_SVR_AUTH_ACT_ATHR_3
address ipv4 10.x.x.x
key 7 *********
timeout 5

[stephendrkw]

line vty 0
exec-timeout 0 0
length 0
transport input ssh
line vty 1 4
length 0
transport input ssh
line vty 5 15
transport input ssh
!

[marce1000]

   - Review the C9800   configuration with the CLI command : show  tech   wireless , have the output analyzed by  https://cway.cisco.com/tools/WirelessAnalyzer/  , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer.               Checkout all advisories!

 M.

[stephendrkw]

Great little tool, thanks. I gather this is the main issue..........

VRF: VRF have been configured, this is not a supported feature in 9800 controllers, and it will lead to severe functionality impact
Action: Disable the feature, use the command 'no vrf definition NAMEOFVRF'
Isn't Mgmt-inf configured by default? Should I remove the vrf-definition?
As you can I see I want mgmt traffic NTP, etc, etc to float via SP Gi0 to avoid Firewall changes, though this can be changed if need be. Plus some Radius/TACACS Servers will need to be amended at the far end IP wise if I went via Mgmt Interface...you can say Mgmt interface is trunked, therefore redundancy..... though I have 2 Anchor WLC's.

#show ip vrf br
Name Default RD Interfaces Mgmt-intf <not set> Gi0

sh run | i vrf
vrf definition Mgmt-intf
ip vrf forwarding Mgmt-intf
ip name-server vrf Mgmt-intf 39.x.x.x 39.x.x.x
ip domain name vrf Mgmt-intf ***************
vrf forwarding Mgmt-intf
ip route vrf Mgmt-intf 0.0.0.0 0.0.0.0 ****
ip route vrf Mgmt-intf 10.***** 255.255.255.255 ****
ip route vrf Mgmt-intf 10.******* 255.255.255.255 ****
ip route vrf Mgmt-intf 10.******255.255.255.255 ****
ip route vrf Mgmt-intf 10.**** 255.255.255.255 GigabitEthernet0 ****
ip route vrf Mgmt-intf 10.****255.255.255.255 ****
ip tacacs source-interface GigabitEthernet0 vrf Mgmt-intf

[marce1000]

 - If you have defined it somewhere manually (any 'personal' vrf definitions)  then that should probably be removed , for the rest if it is only used as a 'pointer' such as for NTP or routing statements that should do no harm. For all config changes it is best to run WirelessAnalyzer again , until the mentioned warning  does  no longer appear (for instance). Check if you can ping the tacacs authentication server for instance through Mgmt-intf vrf.

 M.

[stephendrkw]

I am thinking this might be the issue also.....but I seem to be running 17.6.x

Note:     As of release 17.6, the following protocols are supported through the Service Port (SP): HTTP/HTTPs, SSH, NetFlow, NTP, SNMP, Syslog, RADIUS, and TACACS+.

sh ver
Cisco IOS XE Software, Version 17.06.03
Cisco IOS Software [Bengaluru], C9800 Software (C9800_IOSXE-K9), Version 17.6.3, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2022 by Cisco Systems, Inc.
Compiled Wed 30-Mar-22 23:12 by mcpre

[marce1000]

                   - FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt60584

 M.

[Rich R]

It definitely works - we're using it.
You should have "ip tacacs source-interface GigabitEthernet0" under "aaa group server tacacs+ TAC_EXT"

[stephendrkw]

Thanks added to "ip tacacs source-interface GigabitEthernet0" to "aaa group server tacacs+ TAC_EXT" ..........though no difference

[Rich R]

Then you're going to have to use debugs and packet captures to work out why ...
What does "show tacacs" show you?

Check obvious things like whether the WLC IP is allowed on your TACACS server, the return routing, firewall(s), the shared key for TACACS etc ...

[stephendrkw]

If you see below tty3 I'm pretty sure from memory this is when the request is sent off the TACACS Server and "default to eanble password" means the TACACS Server is down? 

Oct 3 09:13:40.766: %SYS-6-LOGOUT: User user1 has exited tty session 3(10.***)
Oct 3 09:13:42.999: AAA/BIND(000026B0): Bind i/f
Oct 3 09:13:43.000: AAA/AUTHEN/LOGIN (000026B0): Pick method list 'default'
Oct 3 09:13:43.763: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: user1] [Source: 10.****] [localport: 22] at 09:13:43 GMT Mon Oct 3 2022
Oct 3 09:13:43.784: AAA/AUTHOR (000026B0): Method list id=0 not configured. Skip author
Oct 3 09:13:46.350: AAA/AUTHOR: auth_need : user= 'user1' ruser= 'ukgrelg-192-wcon-sp'rem_addr= '10.***' priv= 0 list= '' AUTHOR-TYPE= 'commands'
Oct 3 09:13:46.351: AAA: parse name=tty3 idb type=-1 tty=-1
Oct 3 09:13:46.351: AAA: name=tty3 flags=0x11 type=5 shelf=0 slot=0 adapter=0 port=3 channel=0
Oct 3 09:13:46.351: AAA/MEMORY: create_user (0x7F1D0DE7D8B8) user='user1' ruser='NULL' ds0=0 port='tty3' rem_addr='10.****' authen_type=ASCII service=ENABLE priv=15 initial_task_id='0', vrf= (id=0) key=1E4AA6A6
Oct 3 09:13:46.351: AAA/AUTHEN/START (3727276776): port='tty3' list='' action=LOGIN service=ENABLE
Oct 3 09:13:46.351: AAA/AUTHEN/START (3727276776): non-console enable - default to enable password
Oct 3 09:13:46.351: AAA/AUTHEN/START (3727276776): Method=ENABLE
Oct 3 09:13:46.351: AAA/AUTHEN(3727276776): can't find any passwords
Oct 3 09:13:46.351: AAA/AUTHEN (3727276776): status = ERROR
Oct 3 09:13:46.351: AAA/AUTHEN/START (3727276776): no methods left to try
Oct 3 09:13:46.351: AAA/AUTHEN (3727276776): status = ERROR
Oct 3 09:13:46.351: AAA/AUTHEN/START (3727276776): failed to authenticate
Oct 3 09:13:46.351: AAA/MEMORY: free_user (0x7F1D0DE7D8B8) user='user1' ruser='NULL' port='tty3' rem_addr='10.***' authen_type=ASCII service=ENABLE priv=15 vrf= (id=0)

[Rich R]

right try this:
line vty 0 15
authorization exec TAC_EXT
login authentication TAC_EXT

By the way if read through the best practice guide you'll see you're recommended to have a lot more VTY's (than just 15) because the GUI can use lots of them.
https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#WebuserinterfaceWebUI
We have it set to 97 (the max).

[stephendrkw]

line vty 0 15
authorization exec TAC_EXT
login authentication TAC_EXT

Added, no luck. I have logged a TAC and about to setup some packet captures on trunk and SP ports. I'll let you know the results and when I hear back from a TAC Engineer.

[Arshad Safrulla]

Hi, Below is the template I follow for my deployments. Please edit as per your requirements

aaa new-model

aaa group server tacacs+ tacacs1

server-private Y.Y.Y.Y key ZXCZXCZXCZCZXC

ip vrf forwarding Mgmt-intf

ip tacacs source-interface GigabitEthernet0

!

aaa authentication login AAA-SSH group tacacs1 local

aaa authentication login CONSOLE local

aaa authorization exec AAA-SSH group tacacs1 local

aaa authorization commands 0 AAA-SSH group tacacs1 local

aaa authorization commands 1 AAA-SSH group tacacs1 local

aaa authorization commands 15 AAA-SSH group tacacs1 local

aaa accounting exec default stop-only group tacacs+

aaa accounting commands 0 default start-stop group tacacs+

aaa accounting commands 0 AAA-SSH start-stop group tacacs1

aaa accounting commands 1 AAA-SSH start-stop group tacacs1

aaa accounting commands 15 default start-stop group tacacs+

aaa accounting commands 15 AAA-SSH start-stop group tacacs1

!

ip tacacs source-interface GigabitEthernet0

!

line con 0

session-timeout 5

 exec-timeout 5 0

privilege level 15

login authentication CONSOLE

stopbits 1

!

line vty 0 50

session-timeout 5

 access-class 12 in vrf-also

exec-timeout 5 0

privilege level 15

authorization commands 0 AAA-SSH

authorization commands 1 AAA-SSH

authorization commands 15 AAA-SSH

authorization exec AAA-SSH

accounting commands 0 AAA-SSH

accounting commands 1 AAA-SSH

accounting commands 15 AAA-SSH

login authentication AAA-SSH

transport preferred none

transport input ssh

transport output ssh

[stephendrkw]

I changed my config to as below and I'm getting closer I hope!

Confirming with our team who look after this TACACS ISE as you can see from my debug output, that a AV definition "role1=ALL" was setup years ago for the WLC 5508's which of course work fine. C9800 doesn't like it! You cannot add a custom attribute on the C9800 other than for Access Point AAA Authentication policy if I'm correct, so would this be a ISE config of some sort to adjust the shell profile, but to what?

aa authentication login Networks group TAC_EXT local
aaa authorization exec Networks group TAC_EXT
aaa authorization network Networks group TAC_EXT local
aaa authorization commands 0 Networks group TAC_EXT if-authenticated
aaa authorization commands 1 Networks group TAC_EXT if-authenticated
aaa authorization commands 15 Networks group TAC_EXT if-authenticated
aaa accounting exec Networks start-stop group TAC_EXT
aaa accounting commands 0 Networks stop-only group TAC_EXT
aaa accounting commands 1 Networks stop-only group TAC_EXT
aaa accounting commands 15 Networks stop-only group TAC_EXT

Oct  5 11:35:55.369: TPLUS(00000C45)/0/NB_WAIT/7F20EE572048: Started 5 sec timeout

Oct  5 11:35:55.370: TPLUS(00000C45)/0/NB_WAIT: socket event 2

Oct  5 11:35:55.370: TPLUS(00000C45)/0/NB_WAIT: wrote entire 63 bytes request

Oct  5 11:35:55.370: TPLUS(00000C45)/0/READ: socket event 1

Oct  5 11:35:55.370: TPLUS(00000C45)/0/READ: Would block while reading

Oct  5 11:35:55.560: TPLUS(00000C45)/0/READ: socket event 1

Oct  5 11:35:55.560: TPLUS(00000C45)/0/READ: read entire 12 header bytes (expect 16 bytes data)

Oct  5 11:35:55.560: TPLUS(00000C45)/0/READ: socket event 1

Oct  5 11:35:55.560: TPLUS(00000C45)/0/READ: read entire 28 bytes response

Oct  5 11:35:55.560: TPLUS(00000C45) login timer stopped

Oct  5 11:35:55.560: TPLUS(00000C45)/0/7F20EE572048: Processing the reply packet

Oct  5 11:35:55.560: TPLUS: Failed to decode unknown AV role1=ALL - FAIL

Oct  5 11:35:55.560: TPLUS(00000C45)/0/REQ_WAIT/7F20EE572048: timed out