Wired Dot1x / MAB devices stop communicating

[bturner]

We are rolling out 802.1x to an Industrial Manufacturer.  Our maintenance windows are rather tight for the production floor, and we have a deadline for full TrustSec deployment by April 2020.

For these reasons we are deploying in what we are calling monitor mode, but is actually closed mode with a default ISE permit.  This essentially allows us to capture all the unknown devices without "Blocking" traffic.

We are on ISE 2.4 patch 7, and are running dot1x / mab order with dot1x priority, and a 5 second timer.  This has worked well in the Admin areas, but we have started to run in to a some devices that essentially stop communicating after the initial do1x phase.

Today I found one that was profiled as Linux 2.6, it was a keytracer keypad that essentially checks keys in and out of a box.  If i put authentication open on the port and bounce it, the device works without issues.  However if I remove authentication open and bounce the port it authenticates with MAB, but stops responding to packets.  Wireshark shows it sends out an arp for its default gateway, and upon receiving no response it simply never talks again no matter how much traffic you send its way.

My thought was to add a preauth ACL that allowed ARP, to prevent the device from essentially "Hanging" waiting on an arp response.  I created the ACL and realized ARP was a layer 2 protocol and thus couldn't exist inside an IP ACL.  However when I applied the ACL to the port and the permit DACL on the policy, the problem cleared up.  Wireshark showed that a "fail" response from the switch was enough to kick the device out of its hung state.  Not sure why an IP ACL that can't contain an ARP, can trigger the switch to send a fail response to arp, but it did, and I thought I had found a work around.

When I tried the same thing on our Omron PLC's, I found that they still didn't work after the change.  And it doesn't seem to be all Omron PLC's just some of them, and specifically the "secure external" port that they expose for data collection to the enterprise network.

The only way I have right now to resolve the Omron PLC issue is to have Authentication Open on the port.  So we are having to create a database of ports that we have turned this on for, and would really like to see if anyone else is having similar issues with deploying closed mode to IOT like devices?

Is there some other way to allow ARP in Pre-Auth, or some other solution I haven't thought of yet?  I need to close these loop holes by the end of the year.

At this point I am considering writing a container app that will detect this fail condition and insert the authentication open command dynamically, but I find it hard to believe I am the only one facing this challenge.  

My initial thought was maybe our certificate is triggering some sort of protected mode on the PLC, however authentication open still sends the EAPOL messages, so it must be something the PLC is trying to do that we are blocking... Similar to the KeyTracer.

[dilnaazhum · Accepted solution]

Hello,

I have not tried dot1x feature on Linux machine, in our production enviornment we eventually allowed linux devices

with MAB. Did you tried with bouncing of Port /reauthentication later ?

We observed this, not frequently, It happens after Windows patch and GPO is causing this type of issue, so alternatively, we allow the devices (which are reported) with authenticcation open / MAB  and perform "gpupdate /force on end user devices", later it automatically start authenticate with dot1x.

I suggest to analyse the pattern of cause as it behaves same with Canon printers , runs with MAB still need bounce of interface, so it requires upto date database (IP address, Switch and port details) especially in Production area.

Give a try with different authentication mode and remove Port-security configuration if it is applied, also remove server reauthentication commands from switchport, as I know PLC devices may have multiple users behind the domain connectivity..

Thanks and regards

Mali 

[bturner]

Yes we did try bouncing the port.  The key is that the port has to be in authentication open when it bounces for the device to communicate.  If any other setting, it will not talk until you turn Authentication Open back on, and bounce the port.  These devices appear to have a broken network stack that is not able to recover from an abnormal connection process.

I think I am going to recommend we go with authentication open, and let ISE restrict the devices we do not recognize unless someone has a magic solution that I have not yet thought of.

In the case of a plant floor you may be dealing with conveyor belt PLC's or who knows what IOT device that has in the past never been tested to the standards of modern network equipment.  Unfortunately they will not replace $$$ equipment because your security project is at a standstill.

[dilnaazhum]

Hello,

Authentication open cause problem for intruder to connect their machine without any security..

Try with this configuration and observe for couple of days ..

 description **** Device details ****
 switchport mode access
 switchport block unicast
 ip arp inspection limit rate 2048
 authentication event server dead action authorize vlan "ID"
 authentication event server alive action reinitialize
 authentication host-mode multi-host
 authentication order dot1x mab 
 authentication priority dot1x mab 
 authentication port-control auto
 authentication violation restrict
 mab
 snmp trap mac-notification change added
 snmp trap mac-notification change removed
 snmp ifindex persist
 dot1x pae authenticator
 dot1x timeout tx-period 10
 spanning-tree portfast
 spanning-tree bpduguard enable
 ip dhcp snooping limit rate 2048

Thanks and regards

Mali

[dilnaazhum]

Hello,

Note on DHCP snoop command, you can enable it if switch enabled with DHCP snooping feature..

ip arp inspection limit rate 2048

ip dhcp snooping limit rate 2048

rest commands not cause communication issue..

Regards

Mali

[Jason Kunst]

Authentication open can still be secured by limiting the ports communicating as a base configuration using the initial ACL

If you’re having funny client behavior I would never advocate for closed mode. This is only for highly secure environments using dot1x perhaps on windows machines

[Jamesseibert11]

old post, but feel a few notes could be added to original issue:

if using static IP on the devices you can run in to issues if arp inspection is on.

DHCP snooping enables device-tracking on the vlan

you could run in to issues with mac timeouts vs arp timeouts vs device-tracking timeouts.

on the interface configs for mab devices 

int gi x/x      

    authentication control-direction in

could be helpful to have devices reply to ping to 'wake up' Linux and other types of devices are very quite and tend to time out.

if the mab device won't be moved I often will update ports to be mab then dot1x to avoid the failed dot1x log messages.

int gi x/x

   authentication order mab dot1x
   authentication priority mab dot1x

sh authentication sess int gi x/x detail

is useful for getting insight into why something is occurring.