Communications problems between WLC and RADIUS (ISE)

[schulcz]

Hi All,

I tried to setup a Guest Portal on ISE for WLCs.

Configuration:
2 x WLC in HA (AP SSO) deployment
2 x ISE-VM-K9 in small deployment model

A made a configuration process, now I run tests, but not works as good as expected. We have problems with associations.

The association works basically, but sometimes the client can only join the many attempts. According to WLC radius statistics, radius timeouts can be seen when problems occur at the client connection. (For example, it sends 5 requests, of which 3 runs to timeout) I enabled client debug on WLC, and analyzed logs after that by WLC debug analyzer, it's always says radius timeout problems: Client expiration timer code set for 10 seconds. The reason: No response from radius server for mac filtering request

I tried to add only 1 radius server instead of 2 to the WLAN in WLC, the problem is unchanged.

Examining the ISE "Live logs" in the above case, all 5 requests arrive and none are dropped.

It's like not getting RADIUS ACCEPT on the controller.

WLC mgmt and ISE mgmt (added to the controller as radius) are in the same L2 domain.

ISE sw: 2.4.0.357 (patch 8)

WLC sw: 8.5.140.0

What could be the problem, what do I have to check?

Thanks!

[Ben Walters]

Are you able to post the config for the RADIUS servers on the WLC? 

Also, do you use RADIUS for anything else on the WLC? 802.1x on other WLANs or for admin authentication, if yes do they work as expected? 

[schulcz]

Sure, this is the radius config and statistics on WLC:

(Cisco Controller) >show radius summary 

Vendor Id Backward Compatibility................. Disabled
Call Station Id Case............................. lower
Accounting Call Station Id Type.................. Mac Address
Auth Call Station Id Type........................ AP's Radio MAC Address:SSID
Extended Source Ports Support.................... Enabled
Aggressive Failover.............................. Disabled
Keywrap.......................................... Disabled
Fallback Test:
    Test Mode.................................... Passive
    Probe User Name.............................. cisco-probe
    Interval (in seconds)........................ 300
MAC Delimiter for Authentication Messages........ hyphen
MAC Delimiter for Accounting Messages............ hyphen
RADIUS Authentication Framed-MTU................. 1300 Bytes

Authentication Servers

Idx  Type  Server Address    Port    State     Tout  MgmtTout  RFC3576  IPSec - state/Profile Name/RadiusRegionString
---  ----  ----------------  ------  --------  ----  --------  -------  -------------------------------------------------------
1  * N     10.17.0.31        1812    Enabled   5     5         Enabled   Disabled - /none
2  * N     10.17.0.32        1812    Enabled   5     5         Enabled   Disabled - /none

--More-- or (q)uit

Accounting Servers

Idx  Type  Server Address    Port    State     Tout  MgmtTout  RFC3576  IPSec - state/Profile Name/RadiusRegionString
---  ----  ----------------  ------  --------  ----  --------  -------  -------------------------------------------------------
1  * N     10.17.0.31        1813    Enabled   5     5         N/A       Disabled - /none
2  * N     10.17.0.32        1813    Enabled   5     5         N/A       Disabled - /none

(Cisco Controller) >show radius auth detailed 1

Radius Auth Server Index.........................  1
 PAC Provisioning ...............................  Disabled
Auth Server Type................................. 0

(Cisco Controller) >show radius auth detailed 2

Radius Auth Server Index.........................  2
 PAC Provisioning ...............................  Disabled
Auth Server Type................................. 0

(Cisco Controller) >show radius acct detailed 1

Radius Acct Server Index.........................  1
 PAC Provisioning ...............................  Disabled

(Cisco Controller) >show radius acct detailed 2

Radius Acct Server Index.........................  2
 PAC Provisioning ...............................  Disabled

(Cisco Controller) >

(Cisco Controller) >show radius auth statistics 
Authentication Servers:

Server Index..................................... 1
Server Address................................... 10.17.0.31
Msg Round Trip Time.............................. 12 (msec)
First Requests................................... 6039
Retry Requests................................... 3847
Accept Responses................................. 6017
Reject Responses................................. 0
Challenge Responses.............................. 0
Malformed Msgs................................... 0
Bad Authenticator Msgs........................... 0
Pending Requests................................. 0
Timeout Requests................................. 3869
Consecutive Drops ............................... 0
Unknowntype Msgs................................. 0
Other Drops...................................... 0


Server Index..................................... 2
Server Address................................... 10.17.0.32
Msg Round Trip Time.............................. 0 (msec)

--More-- or (q)uit
First Requests................................... 0
Retry Requests................................... 0
Accept Responses................................. 0
Reject Responses................................. 0
Challenge Responses.............................. 0
Malformed Msgs................................... 0
Bad Authenticator Msgs........................... 0
Pending Requests................................. 0
Timeout Requests................................. 0
Consecutive Drops ............................... 0
Unknowntype Msgs................................. 0
Other Drops...................................... 0

Authentication Total:

First Requests................................... 6039
Retry Requests................................... 3847
Accept Responses................................. 6017
Reject Responses................................. 0
Challenge Responses.............................. 0

(Cisco Controller) >show radius acct stat       

Global Accounting Info:

  Accounting Interim update sent count........... 0

                                                   Accounting Info per Servers:

  Accounting Server Index........................ 1
    Server Address............................... 10.17.0.31
    Msg Round Trip Time.......................... 3 (msec)
    First Requests............................... 5367
    Retry Requests............................... 2933
    Accounting Responses......................... 5358
    Malformed Msgs............................... 0
    Bad Authenticator Msgs....................... 0
    Pending Requests............................. 0
    Timeout Requests............................. 2942
    Unknowntype Msgs............................. 0
    Other Drops.................................. 0


  Accounting Server Index........................ 2

--More-- or (q)uit
    Server Address............................... 10.17.0.32
    Msg Round Trip Time.......................... 0 (msec)
    First Requests............................... 0
    Retry Requests............................... 0
    Accounting Responses......................... 0
    Malformed Msgs............................... 0
    Bad Authenticator Msgs....................... 0
    Pending Requests............................. 0
    Timeout Requests............................. 0
    Unknowntype Msgs............................. 0
    Other Drops.................................. 0



(Cisco Controller) >show radius coa stat
COA Servers:

Server Index..................................... 1
Server Address................................... 10.17.0.31
Disconnect-Requests.............................. 0
COA-Requests..................................... 221
Retransmitted Requests........................... 0
Malformed Requests............................... 0
Bad Authenticator Requests....................... 0
Other Drops...................................... 0
Sent Disconnect-Ack.............................. 0
Sent Disconnect-Nak.............................. 0
Sent CoA-Ack..................................... 221
Sent CoA-Nak..................................... 0
Server Index..................................... 2
Server Address................................... 10.17.0.32
Disconnect-Requests.............................. 0
COA-Requests..................................... 11
Retransmitted Requests........................... 0
Malformed Requests............................... 0
Bad Authenticator Requests....................... 0
Other Drops...................................... 0

--More-- or (q)uit
Sent Disconnect-Ack.............................. 0
Sent Disconnect-Nak.............................. 0
Sent CoA-Ack..................................... 11
Sent CoA-Nak..................................... 0

(Cisco Controller) >ping 10.17.0.31 management 100

Send count=100, Receive count=100 from 10.17.0.31, Packet size = 84

(Cisco Controller) >ping 10.17.0.31 management 100 2000

Send count=100, Receive count=100 from 10.17.0.31, Packet size = 2000

(Cisco Controller) >

Currently we do not use radius for any other purposes, only for Guest "Authentication" on Guest Portal by AUP on ISE.

[Arne Bier]

What does the IP routing path between WLC and ISE nodes look like?  Is there something in the path that could vary the routing of the IP packets, to perhaps cause them to take an alternative return path?  Firewalls?  Load balancer?

In general this should be the most basic setup - but your WLC stats are quite revealing, because they show timeouts (delayed responses from ISE).  Not that many unanswered requests - most of them eventually get responded to by ISE (by the looks of the WLC stats).

is the ISE VM node running the correct spec (8 cores/ 16GB RAM etc) ?  Sorry, I have to ask these dumb questions. :(

Failing that - perhaps there is a dodgy physical connection between WLC and the switch?  

ISE does have one feature that might throw a spanner in the works - not sure if this will drop your requests or send Access-Reject ... I don't know - I could be wrong ... With Guest MAB you don't reject any requests (you tell ISE to CONTINUE to Authorization stage - but you probably do that already)

[schulcz]

There is no firewall or loadbalancer between them. WLC mgmt interface and ISE mgmt interface is in the same vlan/subnet, so the packets didn't need to reach the gateway (firewall) at all. (We use ASA fws for filtering and routing functions)

ISE runs as a virtual appliance by VMware on UCS server. WLCs and servers connected directly to the switch stack (3850x) in the same vlan/subnet.

But what you wrote make me think.. Is it possible ISE send radius responses on more than one interface? We use a separate interface for Guest Portal, and WLC mgmt address isn't reachable from that subnet because of the filters on firewall. Can I edit which interface should be used for radius communication by ISE?

ISE has 6 cores and 16GB RAM, but the load currectly isn't so hard: CPU is around 20%, RAM is around 70% (average cpu load is 11% for this virtual machine seen on ESX)

[Arne Bier]

You cannot disable the listening/processing of RADIUS on a per interface basis - If Policy Services is enabled, then each interface that is up and enabled will be listening on those UDP ports configured.  The only flexibility we have is to turn on/off guest portal processing per interface.

[schulcz · Accepted solution]

Finally, we found the source of the error and found that the error hadn't related with WLC and/or ISE.

Packets missed in the port channel between phisycal switch and ESXi (UCS). The configuration was good, static aggregation was configured on both side, load balancing was set to IP hash, and all ports was set to active state in the vswitch, but not work as good as expected.

After we removed port channel configuration from both side, it works good, I didn't seen any timeout on WLC ever since.

[hslai]

I would suggest to check the response time of all requests between WLC and ISE. ISE RADIUS auth reports have a column on response time and not ISE RAIDUS acct reports so good to start with the auth reports. In case that is not giving you any clue, then next will be to take packet captures. In case you need further help in troubleshooting this, please open a TAC case with Cisco support.