TACACS problem

[mediaworksnz]

Hello, I am running a '15.0(2)SE2
WS-C2960S-24PS-L '.

Tacacs is configured identically to other switches in my inventory. The problem is, when I log in, there is a big delay waiting for the password prompt. Then when I do enter the password, I am denied entry even though credentials are fine and proven working on other TACACS configured switches.

Does anyone have any ideas ?

Here is my configuration and the 'debug aaa authentication' logs:

CONFIG:

!
boot-start-marker
boot-end-marker
!
logging discriminator DISCRIM msg-body drops 10.21.250.110
logging buffered discriminator DISCRIM 409600
!
username xxxxxx
aaa new-model
!
!
aaa group server tacacs+ TACACS_Group
server 10.21.250.212
server 10.21.132.28
!
aaa group server radius RADIUS_GROUP
server 10.5.1.89
server 10.21.130.19
ip radius source-interface Vlan10
!
aaa authentication login default group tacacs+ group TACACS_Group local
aaa authentication enable default group tacacs+ enable
aaa authentication dot1x default group RADIUS_GROUP
aaa authorization console
aaa authorization exec default group tacacs+ local
aaa authorization commands 1 default group tacacs+ local if-authenticated
aaa authorization commands 15 default group tacacs+ local if-authenticated
aaa authorization network default group RADIUS_GROUP
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 1 default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
!
aaa session-id common
clock timezone NZST 12 0
clock summer-time NZDT recurring 4 Sun Sep 2:00 1 Sun Apr 3:00
switch 1 provision ws-c2960s-24ps-l
!
!
no ip domain-lookup
ip domain-name xxxx
ip igmp snooping querier max-response-time 25
ip igmp snooping querier timer expiry 205
ip igmp snooping querier
login on-failure log
login on-success log
vtp domain xxxxx
vtp mode transparent
!
!
!
dot1x system-auth-control
spanning-tree mode pvst
spanning-tree extend system-id
!

vlan internal allocation policy ascending
!
vlan 10
name DATA
!
vlan 999
!
ip telnet source-interface Vlan10
ip ssh source-interface Vlan10
ip ssh version 2
!
!
!
interface FastEthernet0
no ip address
shutdown
!
interface GigabitEthernet1/0/1
description UPLINK_TO_VF_4G_RTR
switchport access vlan 10
switchport mode access
!
interface GigabitEthernet1/0/2
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/3
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/4
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/5
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/6
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/7
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/8
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/9
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/10
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/11
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/12
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/13
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/14
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/15
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/16
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/17
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/18
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/19
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/20
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/21
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/22
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/23
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/24
description User Desktop/VoIP
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface Vlan1
no ip address
shutdown
!
interface Vlan10
description DATA
ip address 10.21.235.5 255.255.255.0
!
ip default-gateway 10.21.235.1
no ip http server
no ip http secure-server
!
ip tacacs source-interface Vlan10
ip sla enable reaction-alerts
logging source-interface Vlan10
logging host 10.21.250.23 discriminator DISCRIM
logging host 10.21.250.110
!
snmp-server group xxx v3 priv read ALL-ACCESS access ACL_Restrict_SNMP
snmp-server view ALL-ACCESS iso included
snmp-server community precision RO ACL_Restrict_SNMP
tacacs-server host 10.21.250.212
tacacs-server host 10.21.132.28
tacacs-server timeout 60
tacacs-server directed-request
tacacs-server key 7 09424323133C0439294716122E71042A2F
!
radius-server attribute 6 on-for-login-auth
radius-server dead-criteria time 10 tries 3
radius-server host 10.5.1.89 key 7 0208321A552E39324F6A511D3C144C
radius-server deadtime 5
!
!
no vstack
!
line con 0
line vty 0 4
transport input ssh
line vty 5 15
transport input ssh
!
ntp server 10.14.254.116 prefer source Vlan10
ntp server 10.14.254.117 source Vlan10
end

Logs:

Oct 18 02:47:34.535: TPLUS: Queuing AAA Accounting request 247 for processing
.Oct 18 02:47:34.535: TPLUS: Queuing AAA Accounting request 247 for processing
.Oct 18 02:47:34.535: TPLUS: processing accounting request id 247
.Oct 18 02:47:34.535: TPLUS: Sending AV task_id=375
.Oct 18 02:47:34.535: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:47:34.535: TPLUS: Sending AV service=shell
.Oct 18 02:47:34.535: TPLUS: Sending AV start_time=1539830854
.Oct 18 02:47:34.535: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:47:34.535: TPLUS: Sending AV cmd=do-exec term mon <cr>
.Oct 18 02:47:34.535: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:47:34.535: TPLUS: using previously set server 10.21.250.212 from group tacacs+
.Oct 18 02:47:34.535: TPLUS(000000F7)/1/IDLE/52D70C4: got immediate connect on new 1
.Oct 18 02:47:34.535: TPLUS(000000F7)/1/WRITE/52D70C4: Started 120 sec timeout
.Oct 18 02:47:34.535: TPLUS(000000F7)/1/WRITE: wrote entire 148 bytes request
.Oct 18 02:47:34.535: TPLUS: processing accounting request id 247
.Oct 18 02:47:34.540: TPLUS: Sending AV task_id=376
.Oct 18 02:47:34.540: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:47:34.540: TPLUS: Sending AV service=shell
.Oct 18 02:47:34.540: TPLUS: Sending AV start_time=1539830854
.Oct 18 02:47:34.540: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:47:34.540: TPLUS: Sending AV cmd=terminal monitor <cr>
.Oct 18 02:47:34.540: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:47:34.540: TPLUS: using previously set server 10.21.250.212 from group tacacs+
.Oct 18 02:47:34.540: TPLUS(000000F7)/1/IDLE/4D5C964: got immediate connect on new 1
.Oct 18 02:47:34.540: TPLUS(000000F7)/1/WRITE/4D5C964: Started 120 sec timeout
.Oct 18 02:47:34.540: TPLUS(000000F7)/1/WRITE: wrote entire 148 bytes request
.Oct 18 02:47:46.588: AAA/BIND(000000FD): Bind i/f
.Oct 18 02:47:46.588: AAA/AUTHEN/LOGIN (000000FD): Pick method list 'default'
.Oct 18 02:47:46.588: TPLUS: Queuing AAA Authentication request 253 for processing
.Oct 18 02:47:46.588: TPLUS: processing authentication start request id 253
.Oct 18 02:47:46.588: TPLUS: Authentication start packet created for 253(adminPF)
.Oct 18 02:47:46.588: TPLUS: Using server 10.21.250.212
.Oct 18 02:47:46.588: TPLUS(000000FD)/1/IDLE/32F5374: got immediate connect on new 1
.Oct 18 02:47:46.588: TPLUS(000000FD)/1/WRITE/32F5374: Started 120 sec timeout
.Oct 18 02:47:46.588: TPLUS(000000FD)/1/WRITE: wrote entire 44 bytes request
.Oct 18 02:47:55.606: TPLUS(000000F7)/1/READ/5518548: timed out
.Oct 18 02:47:55.606: TPLUS: Sending AV task_id=364
.Oct 18 02:47:55.606: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:47:55.606: TPLUS: Sending AV service=shell
.Oct 18 02:47:55.606: TPLUS: Sending AV start_time=1539830815
.Oct 18 02:47:55.606: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:47:55.606: TPLUS: Sending AV cmd=do-exec debug i <cr>
.Oct 18 02:47:55.606: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:47:55.606: TPLUS(000000F7)/1/READ/5518548: timed out, clean up
.Oct 18 02:47:55.606: TPLUS(000000F7)/1/5518548: Processing the reply packet
.Oct 18 02:48:04.142: TPLUS(000000FC)/1/READ/32F737C: timed out
.Oct 18 02:48:04.142: TPLUS: Authentication start packet created for 252(adminpf)
.Oct 18 02:48:04.142: TPLUS(000000FC)/1/READ/32F737C: timed out, clean up
.Oct 18 02:48:04.147: TPLUS(000000FC)/1/32F737C: Processing the reply packet
.Oct 18 02:49:06.611: TPLUS(000000F7)/1/READ/55712B0: timed out
.Oct 18 02:49:06.611: TPLUS: Sending AV task_id=366
.Oct 18 02:49:06.611: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:06.611: TPLUS: Sending AV service=shell
.Oct 18 02:49:06.611: TPLUS: Sending AV start_time=1539830826
.Oct 18 02:49:06.611: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:06.611: TPLUS: Sending AV cmd=tacacs-server timeout 120 <cr>
.Oct 18 02:49:06.611: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:06.611: TPLUS(000000F7)/1/READ/55712B0: timed out, clean up
.Oct 18 02:49:06.611: TPLUS(000000F7)/1/55712B0: Processing the reply packet
.Oct 18 02:49:10.695: TPLUS(000000F7)/1/READ/32FBA38: timed out
.Oct 18 02:49:10.695: TPLUS: Sending AV task_id=367
.Oct 18 02:49:10.695: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:10.695: TPLUS: Sending AV service=shell
.Oct 18 02:49:10.695: TPLUS: Sending AV start_time=1539830830
.Oct 18 02:49:10.695: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:10.695: TPLUS: Sending AV cmd=do-exec sh ru | i tacacs <cr>
.Oct 18 02:49:10.695: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:10.695: TPLUS(000000F7)/1/READ/32FBA38: timed out, clean up
.Oct 18 02:49:10.695: TPLUS(000000F7)/1/32FBA38: Processing the reply packet
.Oct 18 02:49:10.700: TPLUS(000000F7)/1/READ/5570AB0: timed out
.Oct 18 02:49:10.700: TPLUS: Sending AV task_id=368
.Oct 18 02:49:10.700: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:10.700: TPLUS: Sending AV service=shell
.Oct 18 02:49:10.700: TPLUS: Sending AV start_time=1539830830
.Oct 18 02:49:10.700: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:10.700: TPLUS: Sending AV cmd=show running-config <cr>
.Oct 18 02:49:10.700: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:10.700: TPLUS(000000F7)/1/READ/5570AB0: timed out, clean up
.Oct 18 02:49:10.700: TPLUS(000000F7)/1/5570AB0: Processing the reply packet
.Oct 18 02:49:19.508: TPLUS(000000F7)/1/READ/5539F50: timed out
.Oct 18 02:49:19.508: TPLUS: Sending AV task_id=369
.Oct 18 02:49:19.508: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:19.508: TPLUS: Sending AV service=shell
.Oct 18 02:49:19.508: TPLUS: Sending AV start_time=1539830839
.Oct 18 02:49:19.508: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:19.508: TPLUS: Sending AV cmd=do-exec term no mon <cr>
.Oct 18 02:49:19.508: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:19.508: TPLUS(000000F7)/1/READ/5539F50: timed out, clean up
.Oct 18 02:49:19.508: TPLUS(000000F7)/1/5539F50: Processing the reply packet
.Oct 18 02:49:19.508: TPLUS(000000F7)/1/READ/5538C54: timed out
.Oct 18 02:49:19.508: TPLUS: Sending AV task_id=370
.Oct 18 02:49:19.508: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:19.508: TPLUS: Sending AV service=shell
.Oct 18 02:49:19.508: TPLUS: Sending AV start_time=1539830839
.Oct 18 02:49:19.508: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:19.508: TPLUS: Sending AV cmd=terminal no monitor <cr>
.Oct 18 02:49:19.508: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:19.508: TPLUS(000000F7)/1/READ/5538C54: timed out, clean up
.Oct 18 02:49:19.508: TPLUS(000000F7)/1/5538C54: Processing the reply packet
.Oct 18 02:49:22.423: TPLUS(000000F7)/1/READ/32F8D84: timed out
.Oct 18 02:49:22.423: TPLUS: Sending AV task_id=371
.Oct 18 02:49:22.423: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:22.423: TPLUS: Sending AV service=shell
.Oct 18 02:49:22.423: TPLUS: Sending AV start_time=1539830842
.Oct 18 02:49:22.423: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:22.423: TPLUS: Sending AV cmd=do-exec sh run | i tacacs <cr>
.Oct 18 02:49:22.423: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:22.423: TPLUS(000000F7)/1/READ/32F8D84: timed out, clean up
.Oct 18 02:49:22.423: TPLUS(000000F7)/1/32F8D84: Processing the reply packet
.Oct 18 02:49:22.423: TPLUS(000000F7)/1/READ/5803C9C: timed out
.Oct 18 02:49:22.423: TPLUS: Sending AV task_id=372
.Oct 18 02:49:22.423: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:22.423: TPLUS: Sending AV service=shell
.Oct 18 02:49:22.423: TPLUS: Sending AV start_time=1539830842
.Oct 18 02:49:22.423: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:22.423: TPLUS: Sending AV cmd=show running-config <cr>
.Oct 18 02:49:22.423: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:22.423: TPLUS(000000F7)/1/READ/5803C9C: timed out, clean up
.Oct 18 02:49:22.423: TPLUS(000000F7)/1/5803C9C: Processing the reply packet
.Oct 18 02:49:30.351: TPLUS(000000F7)/1/READ/52D72B0: timed out
.Oct 18 02:49:30.351: TPLUS: Sending AV task_id=373
.Oct 18 02:49:30.356: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:30.356: TPLUS: Sending AV service=shell
.Oct 18 02:49:30.356: TPLUS: Sending AV start_time=1539830850
.Oct 18 02:49:30.356: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:30.356: TPLUS: Sending AV cmd=do-exec wr <cr>
.Oct 18 02:49:30.356: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:30.356: TPLUS(000000F7)/1/READ/52D72B0: timed out, clean up
.Oct 18 02:49:30.356: TPLUS(000000F7)/1/52D72B0: Processing the reply packet
.Oct 18 02:49:30.356: TPLUS(000000F7)/1/READ/32F8C54: timed out
.Oct 18 02:49:30.356: TPLUS: Sending AV task_id=374
.Oct 18 02:49:30.356: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:30.356: TPLUS: Sending AV service=shell
.Oct 18 02:49:30.356: TPLUS: Sending AV start_time=1539830850
.Oct 18 02:49:30.356: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:30.356: TPLUS: Sending AV cmd=write <cr>
.Oct 18 02:49:30.356: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:30.356: TPLUS(000000F7)/1/READ/32F8C54: timed out, clean up
.Oct 18 02:49:30.356: TPLUS(000000F7)/1/32F8C54: Processing the reply packet
.Oct 18 02:49:34.540: TPLUS(000000F7)/1/READ/52D70C4: timed out
.Oct 18 02:49:34.540: TPLUS: Sending AV task_id=375
.Oct 18 02:49:34.540: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:34.540: TPLUS: Sending AV service=shell
.Oct 18 02:49:34.540: TPLUS: Sending AV start_time=1539830854
.Oct 18 02:49:34.540: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:34.540: TPLUS: Sending AV cmd=do-exec term mon <cr>
.Oct 18 02:49:34.540: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:34.540: TPLUS(000000F7)/1/READ/52D70C4: timed out, clean up
.Oct 18 02:49:34.540: TPLUS(000000F7)/1/52D70C4: Processing the reply packet
.Oct 18 02:49:34.545: TPLUS(000000F7)/1/READ/4D5C964: timed out
.Oct 18 02:49:34.545: TPLUS: Sending AV task_id=376
.Oct 18 02:49:34.545: TPLUS: Sending AV timezone=NZDT
.Oct 18 02:49:34.545: TPLUS: Sending AV service=shell
.Oct 18 02:49:34.545: TPLUS: Sending AV start_time=1539830854
.Oct 18 02:49:34.545: TPLUS: Sending AV priv-lvl=15
.Oct 18 02:49:34.545: TPLUS: Sending AV cmd=terminal monitor <cr>
.Oct 18 02:49:34.545: TPLUS: Accounting request created for 247(lastresort)
.Oct 18 02:49:34.545: TPLUS(000000F7)/1/READ/4D5C964: timed out, clean up
.Oct 18 02:49:34.545: TPLUS(000000F7)/1/4D5C964: Processing the reply packet
.Oct 18 02:49:46.593: TPLUS(000000FD)/1/READ/32F5374: timed out
.Oct 18 02:49:46.593: TPLUS: Authentication start packet created for 253(adminPF)
.Oct 18 02:49:46.593: TPLUS(000000FD)/1/READ/32F5374: timed out, clean up
.Oct 18 02:49:46.598: TPLUS(000000FD)/1/32F5374: Processing the reply packet
.Oct 18 02:49:46.598: TPLUS: Queuing AAA Authentication request 253 for processing
.Oct 18 02:49:46.598: TPLUS: processing authentication start request id 253
.Oct 18 02:49:46.598: TPLUS: Authentication start packet created for 253(adminPF)
.Oct 18 02:49:46.598: TPLUS: Using server 10.21.250.212
.Oct 18 02:49:46.598: TPLUS(000000FD)/1/IDLE/32F5374: got immediate connect on new 1
.Oct 18 02:49:46.598: TPLUS(000000FD)/1/WRITE/32F5374: Started 120 sec timeout
.Oct 18 02:49:46.598: TPLUS(000000FD)/1/WRITE: wrote entire 44 bytes request

[mediaworksnz]

I can also confirm I have telnet connectivity on TCP port 49. Networking is all good.

I have reset the shared secret numerous times. No problems there. The config on the TACACS server is good i.e .identical to all my other switches.

xxx#show tacacs

Tacacs+ Server - public :
Server address: 10.21.250.212
Server port: 49
Socket opens: 254
Socket closes: 251
Socket aborts: 0
Socket errors: 0
Socket Timeouts: 11
Failed Connect Attempts: 21
Total Packets Sent: 180
Total Packets Recv: 0
Expected Replies: 0

Tacacs+ Server - public :
Server address: 10.21.132.28
Server port: 49
Socket opens: 162
Socket closes: 164
Socket aborts: 0
Socket errors: 1
Socket Timeouts: 14
Failed Connect Attempts: 21
Total Packets Sent: 35
Total Packets Recv: 1
Expected Replies: 0

[paul driver]

Hello

Is it just this device having the problems -  and is it running the same ios as the others?

When you eventually get onto the switch can you test connection to the tacacs from it

Try-

test aaa group TACACS_Group server 10.21.250.212 user password

test aaa group TACACS_Group server 10.21.132.28 user password

or

test aaa group tacacs+ user password legacy

ping 10.21.250.212 source vlan 10
ping 10.21.132.28 source vlan 10

telnet 10.21.250.212 49

telnet 10.21.132.28 49

re-adding the server key
removing the tacacs group server TACACS_Group

[paul driver]

Hello

Just realised you have already tested connectivity and server key - so apologies

What debugging have you enabled?

debug tacacs
debug aaa authentication
debug ip tcp transaction
debug aaa authorization

[mediaworksnz]

Hello, the log output above is from the following:

debug tacacs
debug aaa authentication