HP Thin Client running Windows Embedded cant profile with DHCP attributes

[BTinNC]

My company has a large population of HP Thin Clients that are not joined to our AD domain and thus cannot do dot1x because they do not have certificates.

We decided to do profiling for these devices. We profile for a couple attributes, two of those being DHCP attributes.

About 90% of our thin clients profile as expected, but the other 10% refuse to work. We have to statically assign them to an identity group to get them to authenticate properly.

Through a LOT of troubleshooting we narrowed it down to the DHCP request that the thin client was sending was not being received by the policy node. A TAC engineer looked over the switch config, IP helper address config, etc and said everything appeared to be setup correctly.

The only explanation was that it appears these specific thin clients were not completing the DHCP process before the switch reassigned the VLAN on the port. So when the dhcp request was sent out, the thin client was already in our "guest" vlan which does not forward dhcp to the ISE.

This is very odd as we have so many thin clients which work properly. It is only a small handful that do not. We have not been able to narrow down to anything specific yet. They are running Windows Embedded Standard 7 and a majority of them are HP t5740's. I dont see any Windows or HP updates available for these units and am not sure if there are any registry hacks available to speed up the DHCP process. 

Has anyone ever came across something similar to this?

[jan.nielsen]

Doesn't sound like a normal situation, i would need to see some more info, could you post your interface config of a switch port ?

Jan

[BTinNC]

This is a Cisco WS-C3560G-48PS running 12.2(55)SE10

C3560-IPSERVICESK9-

Here is the standard port config. This is the actual port one of the failing devices is connected to:

interface GigabitEthernet0/11
description Access Port ()
switchport access vlan 300
switchport mode access
switchport voice vlan 500
ip arp inspection limit rate 1000
ip access-group ACL_ALLOW in
srr-queue bandwidth share 10 10 60 20
queue-set 2
priority-queue out
authentication event fail action next-method
authentication event server dead action authorize vlan 300
authentication event server dead action authorize voice
authentication event server alive action reinitialize
authentication host-mode multi-auth
authentication order mab dot1x
authentication priority dot1x mab
authentication port-control auto
authentication violation restrict
mab
mls qos trust device cisco-phone
mls qos trust dscp
snmp trap mac-notification change added
snmp trap mac-notification change removed
dot1x pae authenticator
dot1x timeout tx-period 10
spanning-tree portfast
end

The VLAN 300 has the helper address for all three of our PSN's. 

[jan.nielsen]

Well, you are doing mab before dot1x, which means the second a packet arrives on that port it will be trying to authenticate with ise, and if you are using guest functions on wired, with vlan change, thats gonna happen since it's probably your "default" rule for any unknown mac adresses. Maybe some of you thin clients are sending out packets before they try to do dhcp, in which case ise would have changed the vlan before they send the dhcp packets.

Have you tried using these settings :

authentication order dot1x mab

dot1x timeout tx-period 5

[BTinNC]

I have tried changing the auth order already and it didnt make a difference.

I have not tried bumping the dot1x timeout down to 5 seconds.

What I had been doing previously is starting up the thin client with the switch port shutdown.

Starting Windows Network Monitor on the Thin Client.

As soon as I no-shut the port, the very first packet I see get sent is a DHCP Request and it contains all the attributes I am profiling for.

[jan.nielsen]

Hmm, so maybe try to enable the packet capture function on the ise server that you have entered as ip helper, and verify if ise is getting the packet or not ? Also, is the mac address in ise as unknown or is it not there at all?

The dhcp response is never sent to the ise server anyways, so it shouldn't matter if the client actually finishes the dhcp request or not. 

[BTinNC]

After a lot more troubleshooting it seems that this problem is more widespread than I initially thought.
The devices I thought were working properly aren't. It turns out that the "working" devices were being profiled before auth was enabled on the ports. We had the ip-helper addresses added to the switch several days before we started enabling the switch port configs. So I believe the PSN's had been collecting data on the endpoints before they were being auth'd.

I verified on numerous switches, different models and IOS rev's that DHCP requests are not making it to the PSN using the config from above.

Watching the endpoint debug logs and trying to guesstimate when things are occuring, I would guess that the VLAN is being changed to our guest vlan before the dhcp packet request is sent.

These endpoints are connected to a L2 or L3 switch that trunks the guest VLAN to a different router to handle the guest traffic.

If I add a guest-vlan L3 interface on the L2/L3 switch that my endpoint is connected to, dhcp requests get sent to the ISE as expected. Unfortunately because of our design and security concern, we can't leave this change in place. 

[jan.nielsen]

Are you sure the default acl you have on your interface is allowing the dhcp request ?

[BTinNC]

I have been reading every blog and forum I can find on this topic and just came across one discussing the default ACL when I saw your reply.

Here is the default ACL applied to this interface.

Extended IP access list ACL_ALLOW
10 permit ip any any (1183349 matches)

I wish there was a debug log that showed every step ISE was taking when it transitioned through each Authz policy so I could correlate the endpoints actions with ISE's actions. After reviewing one of the Endpoint debug logs, I can see what I think is ISE changing the access port to the guest vlan really fast.

[jan.nielsen]

Don't know what to tell you, it sounds like ISE is doing what i would expect. Profiling does not happen instantly, so your first authentication, will match your guest redirect rule, if you are changing vlans in that authz profile, the switch will almost instantly change vlans. Is guest access on wired really something you need?...i have many customers asking for it initally, but when we get talking about it, and all the weird issues you can run into with it especially with vlan change and open mode, they realise it's not worth the trouble.

[BTinNC]

It is very possible that I dont understand the mechanics of how ISE is working with the switch.

When I look at my Authz policies, here is the basic list. 1 is a default rule. 2-5 are custom MAB lists that we have for various devices. None of them change VLAN's. 6-7 are the profiling policies where it is looking for specific attributes. 8-9 are the Dot1x policies. 10 is the Guest Portal. 11 is where the endpoints get reassigned to a different VLAN if there is no match in the above rules.

The policy uses "First Matched Rule applies".

So what I am not able to figure out is when the endpoint comes online and the DHCP request is sent, why is it not seen by the PSN to be matched on rules 6 or 7 before it changes VLAN when it processes rule 11?

1: Wireless Black List
2: MAB List for Phones
3: MAB List for Printers
4: MAB List for Misc Devices
5: MAB List for Computers
6: Profling Policy for Thin Clients
7: Profiling Policy for Printers
8: Dot1x - Wired
9: Dot1x - Wireless
10: Guest Portal
11: Guest - Wired
12: Guest - Wireless

[jan.nielsen · Accepted solution]

It's pretty obvious to me, you endpoint is not getting profiled before it's authenticted, which means you are not matching the profiling conditions set in rules 6-7, which means it will match rule 11 (i think, not having seen your actual rules). What i would expect to happen, is that the endpoint gets profiled, if ISE receives the attributes for that endpoint via ex. dhcp helper forwarding. What then should happen is, that it should issue a CoA to the switch, which will cause the switch to re-authenticate that endpoint, which now should have the profiling attributes you are trying to match. However if the DHCP packet never reaches ise, it won't work. This is why i think you should do a packet trace on the ise server, to see if the packets actually reach ise. If they don't you will probably need to find another way to profile, or enable dhcp helper on your guest vlan. Have you looked at the endpoint attributes maybe after 30 secs? Do they change ?

[BTinNC]

Thank you for the great help Jan.

This is what I suspect as well.

What I am trying to figure out is why the DHCP request is not being received by the ISE PSN. I can confirm through a packet capture that the endpoint sends the dhcp request as soon as the port becomes active. The problem is that it seems like the endpoint is already in the guest vlan assigned in rule 11 before the request is sent.

I have the ip-helper assigned to the VLAN that the endpoint is in before it changes.

The L3 switch that the endpoint is attached to trunks the guest VLAN to a router, and that router is the DHCP server for the guest VLAN, so assigning the ip-helper address on that router does not forward the dhcp packets to the ISE PSN.