ThinkPad Universal USB-C Smart Dock with multiple MACs

[LuensmannIT]

Hi Community,

our client hardware department is ordering a lot of the new ThinkPad Universal USB-C Smart Docks which seems to bring a couple of problems with them regarding ISE / 802.1x.
These smart docks have a Microsoft Azure Sphere connection build in to be able to update firmware and doing maintenance without having a client connected to the dock. To accomplish this the docks have a burned in IoT MAC on the ethernet interface (and also a Wi-Fi interface for the same reason but this is out of scope in the moment).

We are using authentication host-mode multi-domain on our ports and now run into trouble when a notebook is connected to the dock as we are seeing the burned in IoT MAC from the dock itself and the MAC from the notebook (via pass-through) at the same time.

I know we could switch to authentication host-mode multi-auth to allow more then one MAC in the DATA domain, but we don’t want to allow this behavior in general.

So, my questions:

The client hardware department is saying that they are unable to deactivate the IoT interface, but they may be ok with just not using this interface. Is there a possibility to block specific OUIs (in this case the Lenovo IoT MACs) before even triggering the violation in the DATA domain if multi-domain is set? For example, with port-security? Does anyone have experience with this?

If we must bring the IoT interface online to enable firmware upgrades and doing maintenance without having a client connected to the dock, how about dynamic VLAN assignment? Even if we would allow the IoT MAC to connect to the same client VLAN as the client behind the dock, there is more than one client VLAN at a location.
As far as I know and regarding the docs I have read, there is no way to have multiple clients behind the same physical port to be in different VLAN?

Do you have advice to solve this problem?

Scenario 1:
Switchport <-> IP Phone (Voice VLAN) <-> IoT Dock (IoT VLAN) – No client
 Should be ok, IoT Dock is like a single normal client.

Scenario 2:
Switchport <-> IP Phone (Voice VLAN) <-> IoT Dock (IoT VLAN) – Customer A (VLAN A)
 Does not work if dock and Customer A need to be in two different VLANs, correct?
 Would work if we allow the IoT Dock to be in the same VLAN as Customer A and enable authentication host-mode multi-auth, correct?

Scenario 3:
Switchport <-> IP Phone (Voice VLAN) <-> IoT Dock (VLAN A) – Customer B (VLAN B)
We are offering shared desk environment where different customers can attach to the dock in the same building.
 So even if we put the IoT Dock in the Customer A client VLAN (instead of IoT VLAN) it would work when a Customer A is attaching to the dock, but it wouldn’t work if a Customer B is attaching to the dock, correct?

Thanks for your time in best regards

[Arne Bier]

Oh boy - thanks for highlighting this - I have not seen one of these docs but I can see how this can cause issues.

There is no way you can be selective about which MAC addresses result in an Auth (MAB) event to ISE, since the switch's job is to send all of these "unknown MACs" (MAC address without a Session) to the RADIUS server. And as you quite rightly pointed out, multi-auth is the gold standard for most of us, but on occasion we must use multi-auth to check each and every MAC address because we expect there to be more than one in the DATA domain.

I have not tried Port Security + NAC (I believe these two are mutually exclusive). 

I have never tried returning a different VLAN to multiple MAC addresses in the DATA domain on the same interface - that would turn that interface into a quasi-trunk - I would assume this is not allowed. The VOICE domain is already a special case because MAC addresses in that domain are tagged with 802.1Q - but an interface in switchport mode access does not expect/support tagged frames in the DATA domain.

Using SDA would definitely solve your issues, since we can put all the endpoints in the same VLAN (as long as they are in the same VN/VRF) and then use SGTs to distinguish one type of endpoint from another. But you would not implement SDA just to solve your docking station issues.

I remember reading about Private VLANs back when I studied my CCNA R&S ... LOL. That concept never took off. And I doubt it would work in combination with NAC. Have a look perhaps.

I think you might end up having to bite the bullet, and go with multi-auth (you can use a MAC prefix rule or MAC OUI rule in ISE to catch all the dock endpoints). But I do share your concern, that having a Lenovo dock on the same VLAN as your employee devices is a bit risky. 

[LuensmannIT]

Hi Arne,

thanks for you detailed reply.

SDA is indeed out of scope for us at the moment as we are still using a lot of “legacy” devices and I’m trying to find a solution for up to 120 locations.  I will have a look into private VLANs … so far, I was only thinking about this technique in the data center environment.

I will get a couple of these docks, do some testing and keep the community informed about my findings and possible solutions we might find.

[andrewswanson]

There is a "Multi-auth Per User VLAN assignment" feature that is supported on some platforms e.g. 9k. I haven't used it myself and has some drawbacks with ARP/ multicast but could be worth looking into.

hth

Andy

[LuensmannIT]

Great, I will have a look at it.
9k is still in early adoption on our side, but it seems 3650 are also supported and maybe it’s only necessary to switch to a higher major version to support this.

I found three Scenarios in the following document: https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3650/software/release/16-9/configuration_guide/sec/b_169_sec_3650_cg/configuring_ieee_802_1x_port_based_authentication.html#concept_4399A67822B44467858A3DD4B5613E1A

Scenario one does not perfectly match as there is no hub, but the rest is fine.
Your mentioned limitations are also listed, but we are ok with these limitations so far.

I will try it and post the results.
Thanks!

[Leo Laohoo]

We are rolling out Lenovo to replace our end-of-lease HP laptops.  We are seeing similar issues when the Lenono laptops are paired with Lenovo docking stations. 

The issue goes away if the Lenovo laptops are paired with our HP G2 docking stations (which came with our HP laptops).  Can you try if you can get an HP G5 docking station and try?

Alternatively, each Lenovo laptop we get comes with a USB-C-to-Ethernet dongle.  Would the behaviour be the same if the ethernet cable is connected to this dongle?

[LuensmannIT]

I’m afraid I will not be able to convince our client hardware department to change the docks to another manufacturer.

USB-C-to-Ethernet dongles are seldom used, but I will try to catch one and do a test.

[LuensmannIT]

Hi Community,
as promised, I did a couple of tests.

With multi-domain enabled I did not find a steady working solution.

With the older switch series like the WS-C3560V2-48PS with version 15.0(2)SE11 the NAC process is going into a loop, never leaving the notebook in and producing a lot of syslog events flooding the system.

With the newer switch series like the WS-C3650-24PS with version 03.06.09E or C9300L-24P-4G with version 17.09.03 there is no NAC process loop and no flooding with syslog messages. If you connect the docking station to the switch while the notebook is already attached to the docking station, the notebook can authenticate and get access to the network, while the docking station itself is blocked. If you connect the docking station to the switch while NOT having the notebook attached and connect the notebook later (> 30 sec) the docking station can authenticate and get access to the network, while the notebook itself is blocked.

So, it seems I must use multi-auth instead of multi-domain.

The older switch series like the WS-C3560V2-48PS with version 15.0(2)SE11 are indeed not able to support the "Multi-auth Per User VLAN assignment" feature. Whoever authenticate first gets the intended vlan assignment, the second machine will be placed into the same vlan.

With the newer switch series like the WS-C3650-24PS, even with the old version 03.06.09E or the actual C9300L-24P-4G with version 17.09.03 are both able to support the "Multi-auth Per User VLAN assignment" feature and it works great.

Sooooo, if I agree to use multi-auth, I could be more or less happy, but during my tests I found a – in my opinion – quite severe security issue:

The docking station does not signal the switch if the notebook is detached from the dock and the ethernet link will stay up.
The switch and of course the ISE is still thinking that there is an active session for the notebook on the switch port. If you attach a second notebook to the same docking station shortly after detaching the first one, the docking station is using the same mac address for the second notebook as for the first notebook. Since the switch and ISE never noticed that the first notebook got detached and the second notebook is using the same mac address, the second notebook gets the same ip address from dhcp as the first one and don’t need to authenticate and get authorized. It has access to the network as long as the reauthentication timer does not time out. I was able to reproduce the issue with all three switch models and software versions. It also doesn’t matter if the docking station is directly attached to the switch or there is an VoIP telephone in between.

Do you have any idea to mitigate this behavior?

[andrewswanson]

Thanks for the update.

The documentation for the dock states that "MAC address Pass Through" is supported when the dock is connected to a LAN.

Although the laptop connects to the hub via USB, does the dock driver installed on the laptop give any option/settings for this?

Andy

[andrewswanson]

"MAC address Pass Through" seems to be supported with selected notebooks/docks:

https://support.lenovo.com/us/en/solutions/ht503574-mac-address-pass-through-feature-of-selected-lenovo-usb-based-docking-solutions-thinkpad

If "MAC address Pass Through" isn't do-able in your environment, does the dock support LLDP?

If so, it could be worth checking whether the dock informs the connecting switch of any notebook disconnects.

[LuensmannIT]

According to our client hardware department "MAC address Pass Through" is enabled on all notebooks / docks and the (recent) dock drivers are installed on every device. They took a quick look into the settings but didn't found a suitable option. Due to holiday season we have to postpone further investigation.

We are already engaged with lenovo to find a solution for this problem.

[Leo Laohoo]

I'd be using a different docking station. 

The workaround is highly management- or user intensive.  It is better to use a docking station that does not have it's own MAC address.

[LuensmannIT]

i fully agree with you. As just mentioned in the reply to andrew swanson we are already engaged with lenovo regarding this problem.
But nevertheless i was hoping that there may be a special option on the switch side to mitigate this problem that i'm just not aware about.

[thomas]

There is a reason why we built multi-auth.  8-) 

While originally it was to handle chained mini-switches and VMware, this is effectively the same problem.

Any time you have an effectively permanent L2 link (IP phone or VMware host), there is no way for the switch to detect the change of client unless they are courteous enough to provide the network with an EAP-Logoff message to tell the network "this MAC has gone away". You could ask Lenovo to implement this but don't hold your breath.

Ultimately you may need to choose what you want to authenticate - docking stations or workstations and only perform network authentication for those endpoints. Since the docking stations are relying on your generosity with MAB, deny them. Only allow 802.1X with a username+password or digital certificate from a properly configured supplicant on the properly provisioned Windows workstation.

To handle the lack of EAP Logoff, you may need to configure a shorter than preferred Session Timeout (1 hour instead of 8 hours ?) for your workstation RADIUS sessions to be regularly re-authenticated by the currently docked computer.

[LuensmannIT]

Yes it’s comparable to VMware and chained mini switches but until now we had all our vms under own control and denied all unmanaged / mini switches which worked well to far. Also all types of phones our customers are using are EAP Logoff capable. 

Reducing the reauthentication timers leads to problems with a lot of our notebooks using certificate based authentication. We / our client software / hardware department never figured out why exactly this is the case, just assumed that it seems to be a driver or windows authentication problem when windows / the notebook just forget how to provide the certificate after already provided it a few minutes ago.

As already mentioned we raised an escalation with Lenovo which includes the request to implement EAP Logoff. IT security already decided that the docking station will be denied so it’s only about getting the notebook constantly and Safe into the network.

Thanks for the advices.