Tacacs not working for 5508

yprasannas · ‎10-25-2011

Tacacs not working for 3 new 5508 WLC's...working fine for 6 old 4400 WLC's.

before 7.116 code upgrade...I remember 5508 was working on and off and now they are not.

Same configs on SW, WLC and ACS.

Debug on WLC gives..below message when Tacacs is attempted..

*aaaQueueReader: Oct 25 09:20:41.700: tplus_processAuthRequest: memory alloc failed for tplus

Any pointers for troubleshooting? Not sure why statistics show zero...?? Radius is working for users.

(wlc03) >show tacacs auth statistics

Authentication Servers:

Server Index..................................... 1

Server Address................................... 10.3.121.21

Msg Round Trip Time.............................. 0 (msec)

First Requests................................... 0

Retry Requests................................... 0

Accept Responses................................. 0

Reject Responses................................. 0

Error Responses.................................. 0

Restart Responses................................ 0

Follow Responses................................. 0

GetData Responses................................ 0

Encrypt no secret Responses...................... 0

Challenge Responses.............................. 0

Malformed Msgs................................... 0

Bad Authenticator Msgs........................... 0

Timeout Requests................................. 0

Unknowntype Msgs................................. 0

Other Drops...................................... 0

Server Index..................................... 2

--More-- or (q)uit

Server Address................................... 10.3.121.22

Msg Round Trip Time.............................. 0 (msec)

First Requests................................... 0

Retry Requests................................... 0

Accept Responses................................. 0

Reject Responses................................. 0

Error Responses.................................. 0

Restart Responses................................ 0

Follow Responses................................. 0

GetData Responses................................ 0

Encrypt no secret Responses...................... 0

Challenge Responses.............................. 0

Malformed Msgs................................... 0

Bad Authenticator Msgs........................... 0

Timeout Requests................................. 0

Unknowntype Msgs................................. 0

Other Drops...................................... 0

(wlc03) >show tacacs summary

Authentication Servers

Idx Server Address Port State Tout

--- ---------------- ------ -------- ----

1 10.3.121.21 49 Enabled 5

2 10.3.121.22 49 Enabled 5

Authorization Servers

Idx Server Address Port State Tout

--- ---------------- ------ -------- ----

1 10.3.121.21 49 Enabled 30

2 10.3.121.22 49 Enabled 5

Accounting Servers

Idx Server Address Port State Tout

--- ---------------- ------ -------- ----

1 10.3.121.21 49 Enabled 5

We can ping the TACACS servers...

Stephen Rodriguez · ‎10-25-2011

Looks like a memory issue can you pull

show memory stat

show buffers

show process memory

HTH,

Steve

----------------------------------------------------------------------------------------------------------

Please remember to rate helpful posts or to mark the question as answered so that it can be found later.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

yprasannas · ‎10-25-2011

>show memory statistics

System Memory Statistics:

Total System Memory............: 1028820992 bytes

Used System Memory.............: 458424320 bytes

Free System Memory.............: 570396672 bytes

Bytes allocated from RTOS......: 21939008 bytes

Chunks Free....................: 29 bytes

Number of mmapped regions......: 45

Total space in mmapped regions.: 212779008 bytes

Total allocated space..........: 12015112 bytes

Total non-inuse space..........: 9923896 bytes

Top-most releasable space......: 133800 bytes

Total allocated (incl mmap)....: 234718016 bytes

Total used (incl mmap).........: 224794120 bytes

Total free (incl mmap).........: 9923896 bytes

show buffers

Pool[00]: 16 byte chunks

chunks in pool: 50000

chunks in use: 19030

bytes in use: 304480

bytes requested: 90479 (214001 overhead bytes)

Pool[01]: 64 byte chunks

chunks in pool: 40000

chunks in use: 14519

bytes in use: 929216

bytes requested: 566395 (362821 overhead bytes)

Pool[02]: 128 byte chunks

chunks in pool: 20000

chunks in use: 7726

bytes in use: 988928

bytes requested: 672853 (316075 overhead bytes)

Pool[03]: 256 byte chunks

chunks in pool: 4000

chunks in use: 808

bytes in use: 206848

bytes requested: 154777 (52071 overhead bytes)

Pool[04]: 1024 byte chunks

--More-- or (q)uit

chunks in pool: 15300

chunks in use: 11645

bytes in use: 11924480

bytes requested: 4945714 (6978766 overhead bytes)

Pool[05]: 2048 byte chunks

chunks in pool: 1000

chunks in use: 189

bytes in use: 387072

bytes requested: 355272 (31800 overhead bytes)

Pool[06]: 4096 byte chunks

chunks in pool: 1000

chunks in use: 36

bytes in use: 147456

bytes requested: 102479 (44977 overhead bytes)

Raw Pool:

chunks in use: 186

bytes requested: 156052303

show process memory

Name Priority BytesInUse BlocksInUse Reaper

cslStoreManager (240/ 7) 0 0 ( 0/ 0)%

System Reset Task (240/ 7) 0 0 ( 0/ 0)%

reaperWatcher ( 3/ 96) 0 0 ( 0/ 0)% I

osapiReaper ( 10/ 94) 0 0 ( 0/ 0)% I

TempStatus (240/ 7) 424 1 ( 0/ 0)% I

pktDebugSocketTask (255/ 1) 0 0 ( 0/ 0)%

LICENSE AGENT (240/ 7) 2228 85 ( 0/ 0)% I

emWeb ( 7/ 95) 1235795 20743 ( 0/ 0)% T 300

webJavaTask (240/ 7) 0 0 ( 0/ 0)%

fmcHsTask (100/ 60) 0 0 ( 0/ 0)%

apstatEngineTask (240/ 7) 0 0 ( 0/ 0)%

rrcEngineTask (240/ 7) 0 0 ( 0/ 0)%

spectrumDataTask (255/ 1) 1614480 12 ( 0/ 0)%

spectrumNMSPTask (255/ 1) 28808 3 ( 0/ 0)%

wipsTask (240/ 7) 0 0 ( 0/ 0)%

tsmTask (255/ 1) 0 0 ( 0/ 0)%

cids-cl Task (240/ 7) 0 0 ( 0/ 0)%

ethoipSocketTask ( 7/ 95) 0 0 ( 0/ 0)%

ethoipOsapiMsgRcv (240/ 7) 0 0 ( 0/ 0)%

--More-- or (q)uit

envCtrollerStatus (240/ 7) 0 0 ( 0/ 0)%

rfidTask (240/ 7) 0 0 ( 0/ 0)%

idsTrackEventTask (239/ 8) 0 0 ( 0/ 0)%

DHCP Server (240/ 7) 0 0 ( 0/ 0)%

bcastReceiveTask (240/ 7) 0 0 ( 0/ 0)%

ProcessLoggingTask (240/ 7) 0 0 ( 0/ 0)%

CDP Main (240/ 7) 3100 13 ( 0/ 0)%

sntpMainTask (240/ 7) 0 0 ( 0/ 0)%

sntpReceiveTask (240/ 7) 0 0 ( 0/ 0)%

cdpSocketTask (240/ 7) 0 0 ( 0/ 0)%

grouping Task (255/ 1) 0 0 ( 0/ 0)%

dot11a (255/ 1) 63 3 ( 0/ 0)%

rrm Socket Task ( 1/ 97) 35024 1 ( 0/ 0)%

rrm Socket Task (255/ 1) 35024 1 ( 0/ 0)%

dot11a (255/ 1) 0 0 ( 0/ 0)%

grouping Task (255/ 1) 0 0 ( 0/ 0)%

dot11b (255/ 1) 105 5 ( 0/ 0)%

rrm Socket Task (255/ 1) 35024 1 ( 0/ 0)%

dot11b (255/ 1) 0 0 ( 0/ 0)%

rrm Socket Task (255/ 1) 35024 1 ( 0/ 0)%

apfPmkCacheTimer (240/ 7) 0 0 ( 0/ 0)%

Apf Guest (240/ 7) 0 0 ( 0/ 0)%

RLDP Schedule Task (240/ 7) 0 0 ( 0/ 0)%

--More-- or (q)uit

apfMsConnTask_5 (175/ 32) 0 0 ( 0/ 0)%

apfMsConnTask_4 (175/ 32) 0 0 ( 0/ 0)%

apfMsConnTask_6 (175/ 32) 0 0 ( 0/ 0)%

apfMsConnTask_7 (175/ 32) 0 0 ( 0/ 0)%

apfMsConnTask_3 (175/ 32) 0 0 ( 0/ 0)%

apfMsConnTask_2 (175/ 32) 0 0 ( 0/ 0)%

apfLbsTask (240/ 7) 0 0 ( 0/ 0)%

apfMsConnTask_0 (175/ 32) 0 0 ( 0/ 0)%

apfMsConnTask_1 (175/ 32) 0 0 ( 0/ 0)%

apfProbeThread (200/ 22) 0 0 ( 0/ 0)%

apfOrphanSocketTas (240/ 7) 0 0 ( 0/ 0)%

apfRogueDetectorTh (175/ 32) 0 0 ( 0/ 0)%

apfRogueTask (240/ 7) 0 0 ( 0/ 0)%

apfOpenDtlSocket (175/ 32) 0 0 ( 0/ 0)%

apfRLDP (175/ 32) 424 1 ( 0/ 0)%

apfRLDPRecv (175/ 32) 0 0 ( 0/ 0)%

apfReceiveTask (175/ 32) 0 0 ( 0/ 0)%

mmMfpTask (175/ 32) 0 0 ( 0/ 0)%

mmMobility (240/ 7) 1272 3 ( 0/ 0)%

mmSSHPeerRegister (240/ 7) 0 0 ( 0/ 0)%

mmListen (180/ 30) 99920 227 ( 0/ 0)%

tplusTransportThre (201/ 22) 0 0 ( 0/ 0)%

radiusCoASupportTr (201/ 22) 0 0 ( 0/ 0)%

--More-- or (q)uit

EAP Framework (240/ 7) 0 0 ( 0/ 0)%

aaaQueueReader (225/ 13) 3518 12 ( 0/ 0)%

radiusRFC3576Trans (201/ 22) 0 0 ( 0/ 0)%

radiusTransportThr (201/ 22) 0 0 ( 0/ 0)%

pemReceiveTask (240/ 7) 0 0 ( 0/ 0)%

iappSocketTask (240/ 7) 0 0 ( 0/ 0)%

ccxRmTask (230/ 11) 0 0 ( 0/ 0)%

ccxS69Task (240/ 7) 424 1 ( 0/ 0)%

ccxDiagTask (240/ 7) 0 0 ( 0/ 0)%

ccxL2RoamTask (240/ 7) 240424 3 ( 0/ 0)%

dot1xSocketTask (240/ 7) 0 0 ( 0/ 0)%

Dot1x_NW_MsgTask_7 (240/ 7) 0 0 ( 0/ 0)%

Dot1x_NW_MsgTask_6 (240/ 7) 0 0 ( 0/ 0)%

Dot1x_NW_MsgTask_2 (240/ 7) 0 0 ( 0/ 0)%

Dot1x_NW_MsgTask_3 (240/ 7) 0 0 ( 0/ 0)%

Dot1x_NW_MsgTask_4 (240/ 7) 0 0 ( 0/ 0)%

Dot1x_NW_MsgTask_5 (240/ 7) 0 0 ( 0/ 0)%

Dot1x_NW_MsgTask_1 (240/ 7) 0 0 ( 0/ 0)%

Dot1x_NW_MsgTask_0 (240/ 7) 424 1 ( 0/ 0)%

dot1xMsgTask (240/ 7) 0 0 ( 0/ 0)%

locpTxServerTask (220/ 15) 408 2 ( 0/ 0)%

locpRxServerTask (200/ 22) 428043 1961 ( 0/ 0)%

capwapSocketTask ( 72/ 70) 303104 148 ( 0/ 0)%

--More-- or (q)uit

spamApTask6 (118/ 53) 25929 63 ( 0/ 0)%

spamApTask7 ( 53/ 78) 24233 59 ( 0/ 0)%

spamApTask5 (118/ 53) 23445 61 ( 0/ 0)%

spamApTask4 (118/ 53) 23513 58 ( 0/ 0)%

spamApTask3 (118/ 53) 19569 48 ( 0/ 0)%

spamApTask2 ( 53/ 78) 23809 58 ( 0/ 0)%

spamApTask1 ( 53/ 78) 22961 56 ( 0/ 0)%

spamApTask0 ( 78/ 68) 39189 106 ( 0/ 0)%

spamReceiveTask (120/ 52) 2204024 252 ( 0/ 0)%

spamSocketTask ( 32/ 85) 0 0 ( 0/ 0)%

Image License brok (240/ 7) 0 0 ( 0/ 0)% I

Image License brok (240/ 7) 28 1 ( 0/ 0)% I

IPC Main Thread (240/ 7) 0 0 ( 0/ 0)% I

License Client Lib (240/ 7) 96 1 ( 0/ 0)% I

sshpmLscScepTask (100/ 60) 0 0 ( 0/ 0)%

License Client Lib (240/ 7) 96 1 ( 0/ 0)% I

sshpmLscTask (100/ 60) 25783 1739 ( 0/ 0)%

sshpmReceiveTask (175/ 32) 6697 66 ( 0/ 0)%

sshpmMainTask (100/ 60) 208440 358 ( 0/ 0)%

mfpKeyRefreshTask (255/ 1) 0 0 ( 0/ 0)%

mfpEventTask (255/ 1) 0 0 ( 0/ 0)%

mfpTrapForwardTask (255/ 1) 0 0 ( 0/ 0)%

clientTroubleShoot (100/ 60) 2841248 4 ( 0/ 0)%

--More-- or (q)uit

loggerMainTask (200/ 22) 0 0 ( 0/ 0)%

debugMainTask (200/ 22) 0 0 ( 0/ 0)%

dot3ad_lac_task (240/ 7) 32901 3 ( 0/ 0)%

gccp_t (240/ 7) 5864 5 ( 0/ 0)%

dot1dTimer (240/ 7) 0 0 ( 0/ 0)% T 300

dot1dRecv (250/ 3) 0 0 ( 0/ 0)%

uart_session (240/ 7) 0 0 ( 0/ 0)%

StatsTask (240/ 7) 0 0 ( 0/ 0)%

fdbTask (240/ 7) 0 0 ( 0/ 0)%

broffu_SocketRecei (100/ 60) 13 1 ( 0/ 0)%

SNMPProcMon (240/ 7) 0 0 ( 0/ 0)% T 300

RMONTask ( 71/ 71) 0 0 ( 0/ 0)% I

SNMPTask (240/ 7) 61089 1064 ( 0/ 0)%

DHCP Socket Task (240/ 7) 0 0 ( 0/ 0)%

DHCP Proxy Task (240/ 7) 0 0 ( 0/ 0)%

dhcpClientTimerTas (240/ 7) 0 0 ( 0/ 0)%

DHCP Client Task (240/ 7) 0 0 ( 0/ 0)% T 600

BootP (240/ 7) 0 0 ( 0/ 0)% T 300

TransferTask (240/ 7) 848 2 ( 0/ 0)% I

osapiTimer (100/ 60) 13024 2 ( 0/ 0)% T 300

nim_t (100/ 60) 2447 3 ( 0/ 0)%

dtlArpTask ( 7/ 95) 98436 3 ( 0/ 0)%

dtlTask (100/ 60) 41089 20 ( 0/ 0)%

--More-- or (q)uit

dtlDataLowTask ( 7/ 95) 0 0 ( 0/ 0)%

sysapiprintf (240/ 7) 22657 3 ( 0/ 0)%

osapiBsnTimer ( 95/ 62) 0 0 ( 0/ 0)%

fp_main_task (240/ 7) 153068796 26868 ( 0/ 0)%

Stephen Rodriguez · ‎10-25-2011

can you also grab a

show run-config no-ap

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

yprasannas · ‎10-25-2011

reboot fixed the tacacs....which is not good....

Other WLC's we redid the tacacs and they are also fine...FYI:- 7.116 code upgrade started Tacacs issue.

George Stefanick · ‎10-25-2011

OMG -- I had to do this very same thing ... VERY ODD ...

"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
___________________________________________________________

yprasannas · ‎11-08-2011

Looks like this is classified as a bug..It says it will be fixed in 7.2 code??

CSCsg87862 WLC Console/Telnet/SSH/HTTP/HTTPS Hang When AAA server is unresponsive

This might be also impacting Radius....

I have been pulling my hair with Radius issues for over a year and Tacacs for several months. ACS reboot has been fixing Radius..weird..Many TAC cases have been opened.

It is time for a hair cut...no need to pull hair!

yprasannas · ‎11-08-2011

Got a confirmation that this bug does not affect Radius...will check with AAA team.