ISE slow to gather attributes during 2.2 -> 3.1 migration

Josh Morris · ‎12-05-2022

I am slowly migrating NADs from my 2.2 to 3.1P3 deployment. I have only gotten a few NADs in but ran into something strange during my latest round. I've seen this both with 3850s and 4510R+E. When I change my radius config and clear all sessions, I immediately see all endpoints authenticating against the new ISE deployment. The problem is that it appears no attributes are making it through initially to make intelligent profiling decisions. Even simple things like Cisco APs and IP Phones are getting classified as 'Cisco-Device'. This is not good for the phones since this means they do not join the VOICE domain. If I wait a few minutes for ISE to gather attributes and clear all authentication sessions again, they work as expected.

I am running 4 PSNs, load balanced, with the following profilers: HTTP, RADIUS, NMAP, AD. This has not been an issue in the 2.2 environment and I haven't seen it in my 3.1 testing thus far.

Here is a snippet of the relevant config.

n-k1swidf2#show run | sec aaa
aaa new-model
aaa group server tacacs+ vtygroup
 server-private 10.2.1.14 key xxx
aaa group server radius ISE_RADIUS
 server name ISE_PSN_VS
aaa authentication login vtylogin group vtygroup local enable
aaa authentication login consolelogin local
aaa authentication dot1x default group ISE_RADIUS
aaa authorization network default group ISE_RADIUS 
aaa authorization network cts-list group ISE_RADIUS 
aaa authorization auth-proxy default group ISE_RADIUS 
aaa accounting update newinfo periodic 420
aaa accounting dot1x default start-stop group ISE_RADIUS
aaa accounting exec default start-stop group vtygroup
aaa accounting commands 0 default start-stop group vtygroup
aaa accounting commands 1 default start-stop group vtygroup
aaa accounting commands 15 default start-stop group vtygroup
aaa accounting network default start-stop group ISE_RADIUS
aaa accounting system default start-stop group ISE_RADIUS
aaa server radius dynamic-author
 client 10.2.1.14 server-key xxx
 client 10.2.2.10 server-key xxx
 client 10.2.2.11 server-key xxx
aaa session-id common
n-k1swidf2#
n-k1swidf2#
n-k1swidf2#
n-k1swidf2#show run | sec radius
aaa group server radius ISE_RADIUS
 server name ISE_PSN_VS
aaa server radius dynamic-author
 client 10.2.1.14 server-key xxx
 client 10.2.2.10 server-key xxx
 client 10.2.2.11 server-key xxx
ip radius source-interface Vlan4000 
radius-server attribute 6 on-for-login-auth
radius-server attribute 6 support-multiple
radius-server attribute 8 include-in-access-req
radius-server attribute 25 access-request include
radius-server attribute 31 mac format ietf upper-case
radius-server attribute 31 send nas-port-detail mac-only
radius-server dead-criteria time 10 tries 3
radius-server deadtime 10
radius server ISE_PSN_VS
 address ipv4 10.2.1.14 auth-port 1812 acct-port 1813
 pac key xxx
n-k1swidf2#
n-k1swidf2#
n-k1swidf2#show run | sec device-sensor
device-sensor filter-list cdp list cdp-list
 tlv name device-name
 tlv name address-type
 tlv name capabilities-type
 tlv name version-type
 tlv name platform-type
device-sensor filter-list lldp list lldp-list
 tlv name system-name
 tlv name system-description
 tlv name system-capabilities
device-sensor filter-list dhcp list dhcp-list
 option name host-name
 option name default-ip-ttl
 option name requested-address
 option name parameter-request-list
 option name class-identifier
 option name client-identifier
device-sensor filter-spec dhcp include list dhcp-list
device-sensor filter-spec lldp include list lldp-list
device-sensor filter-spec cdp include list cdp-list
device-sensor accounting
device-sensor notify all-changes
n-k1swidf2#
n-k1swidf2#
n-k1swidf2#
n-k1swidf2#show run int g1/1
Building configuration...

Current configuration : 929 bytes
!
interface GigabitEthernet1/1
 switchport access vlan 42
 switchport mode access
 switchport voice vlan 74
 ip device tracking maximum 10
 logging event link-status
 authentication control-direction in
 authentication event fail action next-method
 authentication event server dead action authorize vlan 42
 authentication event server dead action authorize voice
 authentication event server alive action reinitialize 
 authentication host-mode multi-auth
 authentication open
 authentication order mab dot1x
 authentication priority dot1x mab
 authentication port-control auto
 authentication periodic
 authentication timer reauthenticate server
 authentication violation restrict
 mab
 dot1x pae authenticator
 dot1x timeout quiet-period 300
 dot1x timeout tx-period 10
 dot1x timeout ratelimit-period 300
 service-policy input QoS-Input-Policy
 service-policy output QoS-Host-Port-Output-Policy
end

Arne Bier · ‎12-05-2022

You need to validate that the DHCP Snooping is working on the VLANs you expect. Do a few show ip dhcp snooping commands to check that. Have you also dumped the ip device-sensor cache to see if CDP/LLDP/DHCP is popuplating? If there is nothing in the cache, then ISE will get zero profiling data.

Device Tracking should also be setup to allow the MAC to IP binding (which is handy in the absence of VLANs on which you don't use DHCP Snooping). Create a separate profile for trunks

show ip dhcp snooping
show ip dhcp snooping binding
show device-sensor cache all
show device-tracking database

Josh Morris · ‎12-06-2022

Thanks. All these things seem to check out properly. The device-sensor cache is cool though, I didn't know about that one.

n-l1swidf8#show device-sensor cache interface g1/0/27
Device: 2834.a283.bfbd on port GigabitEthernet1/0/27
----------------------------------------------------------------------------
Proto Type:Name                       Len Value                       Text
CDP     28:secondport-status-type       7 00 1C 00 07 00 02 00        .......
CDP      6:platform-type               23 00 06 00 17 43 69 73 63 6F  ....Cisco
                                          20 49 50 20 50 68 6F 6E 65   IP Phone
                                          20 38 38 36 31               8861
CDP      5:version-type                33 00 05 00 21 73 69 70 38 38  ...!sip88
                                          78 78 2E 31 34 2D 31 2D 31  xx.14-1-1
                                          2D 30 30 30 31 2D 31 32 35  -0001-125
                                          2E 6C 6F 61 64 73           .loads
CDP      4:capabilities-type            8 00 04 00 08 00 00 04 90     .......^P
CDP      2:address-type                17 00 02 00 11 00 00 00 01 01  .........
                                          01 CC 00 04 0A 2A 4F 9E     .L...*O.
CDP      1:device-name                 19 00 01 00 13 53 45 50 32 38  ....SEP28
                                          33 34 41 32 38 33 42 46 42  34A283BFB
                                          44                          D

n-l1swidf8#show ip dhcp snooping binding interface g1/0/27
MacAddress          IpAddress        Lease(sec)  Type           VLAN  Interface
------------------  ---------------  ----------  -------------  ----  --------------------
28:34:A2:83:BF:BD   10.42.79.158     467705      dhcp-snooping   78    GigabitEthernet1/0/27
08:92:04:85:92:6D   10.42.47.51      512097      dhcp-snooping   46    GigabitEthernet1/0/27
Total number of bindings: 2

n-l1swidf8#show device-tracking database int g1/0/27
portDB has 4 entries for interface Gi1/0/27, 4 dynamic 
Codes: L - Local, S - Static, ND - Neighbor Discovery, ARP - Address Resolution Protocol, DH4 - IPv4 DHCP, DH6 - IPv6 DHCP, PKT - Other Packet, API - API created
Preflevel flags (prlvl):
0001:MAC and LLA match     0002:Orig trunk            0004:Orig access           
0008:Orig trusted trunk    0010:Orig trusted access   0020:DHCP assigned         
0040:Cga authenticated     0080:Cert authenticated    0100:Statically assigned   


    Network Layer Address                   Link Layer Address Interface  vlan  prlvl age    state     Time left        
DH4 10.42.79.158                            2834.a283.bfbd     Gi1/0/27   78    0025  61s    REACHABLE 249 s try 0(467673 s)
DH4 10.42.47.51                             0892.0485.926d     Gi1/0/27   46    0025  1081mn STALE     try 4 512065 s        
ND  FE80::BA0E:B2D9:95BC:1B95               0892.0485.926d     Gi1/0/27   46    0005  1083mn STALE     try 0 24141 s         
ND  FE80::F82F:38F7:F95A:A984               0892.0485.926d     Gi1/0/27   46    0005  1202mn STALE     try 0 15809 s

I wonder if there could be any session related issue? My order of operations is as follows:

Verify NAD is in new ISE deployment
Save Config
Migrate Tacacs/Test
Add new ISE dynamic authors, remove old
Add new radius server and pac key. This automatically removes the old server, so I don't remove it manually
Pac downloads
perform manual clear authentication session
All endpoints show up on New ISE with little-to-no attributes
Wait a few minutes for attributes to get populated (maybe due to nmap?)
perform another manual clear authentication session, watch devices get profiled correctly

Should I add some steps around clearing device-tracking, dhcp snooping, device-sensor cache as well?

hslai · ‎12-06-2022

Since RADIUS probe enabled but not SNMP query probe, ISE will rely on device-sensor data through RADIUS accounting updates. As such, I would suggest two things:

Take packet captures of RADIUS accounting exchanges and verify the network devices are sending device-sensor attributes to ISE.
Check ISE RADIUS Settings and ensure the setting for Ignore repeated accounting updates with N seconds is not too aggressive. You may also check ISE RADIUS accounting reports and ISE local store logs.

Josh Morris · ‎12-07-2022

Thanks, I will look into packet captures. But your second suggestion led me to checking Radius and profiling settings and I did find a different in how I'm handling CoA. Please see the screenshot below. 2.2 on left, 3.1 on right. I'm guess I need COA enabled with Reauth in 3.1 along with enabling the endpoint attribute filter?

Edit: In reading Craig Hyps profiling guide from 2012, the recommended option in this case is Port Bounce. My only hesitation with doing this instead of ReAuth is that I have a bunch of PoE devices like phones, APs, security cameras. I feel like Port Bounce could cause unnecessary issues.

Arne Bier · ‎12-07-2022

Apparent ISE will "downgrade" the Port Bounce to a re-auth if (and only if) it detects that the Interface on which the Endpoint is located, also contains a phone. I have never tested this. And I wonder whether it simply does this if there are multiple endpoints on an interface (irrespective of type). So in theory it should be safe to send a bounce to those ports - but if the reauth happens it may not have the desired effect (i.e. it may not trigger the endpoint to perform another DHCP etc.).

hslai · ‎12-10-2022

@Josh Morris I, too, usually use ReAuth instead of Port Bounce.

Do check the affected endpoints in ISE context visibility and see if they have these CDP/LLDP attributes there.