Nexus 92160YC switch drops its dot1x configuration when authentication failure occurs; goes into dumb state.

[jonathan.e.brown]

I have a Nexus 92160YC switch on NX-OS version 9.3.1.  It's configured to use a radius server (RHEL 7 server running freeradius) for 802.1x MAC Authentication Bypass (MAB).  

When it's initially setup it appears to work fine for devices that are authorized to connect to the switch as defined in the radius server's configuration.  The switch will indeed assign the appropriate VLAN as long as the device passes authentication/authorization.  However, once an unauthorized device is connected to the switch, the switch will drop several radius/dot1x configuration lines from its configuration.

The following lines get dropped:

radius-server host <X.X.X.11> key 7 <Shared Secret Key> authentication accounting
aaa group server radius RadTest
    server <d.e.f.11>
    deadtime 1
    source-interface vlan9

ip radius source-interface vlan9

The following lines remain:

feature dot1x

aaa authentication dot1x default group RadTest

Once those global config lines are removed there is nothing I can do sans reboot on the failed switch. Doing a ‘show run all’ or any show radius type commands always return nothing. Like there is nothing configured. If you try to re-add these lines, even using bogus IPs, group, interface, etc results in a ‘failed to commit radius config’. Tried using bogus data just in case something in the switch still had the “real” config. The only way to recover at this point is to reboot the switch. After reboot, it loads the saved config and works again, port will authenticate and everything works as desired until you change hosts on the port or plug in a host who is not included as an authorized device.

Using the "test server aaa radius <IP> <username> <password>" command does not duplicate these results.  I have not been able to find any known Cisco bug that relates to these issues.  

I am using the following statements at the interface level:

dot1x port-control auto

dot1x max-req 1

dot1x re-authentication

dot1x timeout quiet-period 10

dot1x timeout tx-period 5

dot1x timeout supp-timeout 5

dot1x pae authenticator

dot1x mac-auth-bypass

dot1x host-mode single-host

switchport access vlan 99 <- vlan 99 is my blackhole vlan

no cdp enable

speed 1000

[Sergiu.Daniluk]

Hi @jonathan.e.brown 

The behavior described is either a very serious security issue (which I doubt), or a misconfig somewhere.

Have you verified the radius server, to make sure you do not have any unnecessary configuration which can potentially remove the config on the switch?

Also, check the "show accounting log" on Nexus switch and see which user removed the mentioned configuration. This show command preserves the logs after reload, so if not a lot of changes have been performed in the meantime, you should see the logs during the time of the issue.  "show logging nvram" might help as well.

Cheers,

Sergiu

[jonathan.e.brown]

Hi @Sergiu.Daniluk,

We've discovered that the issue involves FIPS.  The RADIUS server is configured to use MD5 for the conversation between the RADIUS server and the Nexus 92160YC switch.  The Nexus 92160YC switch had "fips mode enable" in its running config.  If we disable FIPS on the switch then 802.1x behaves as expected.

At this point in time we're researching what we need to do to configure the RADIUS server to operate using FIPS approved algorithms.

Thanks for your comments.  

[Sergiu.Daniluk]

Hi @jonathan.e.brown 

Thank you as well for sharing your finding. Definitely was not expecting this. Very interesting scenario indeed. So basically FreeRadius on RHEL is not supporting FIPS mode? Please share what would be the final solution on the FreeRadius to enable FIPS mode. You made me super curious ^_^

Stay safe!

Sergiu

[jonathan.e.brown]

@Sergiu.Daniluk ,

The RADIUS server we're using is running on a hardened Redhat 6.8 image.  So it has FIPS enabled on it as well.  When I had first installed the freeRADIUS RPM package on the server I found that the service wouldn't start.  After a bit of troubleshooting I discovered that if I turned off FIPS on the RADIUS server then the service would actually start.  This was a problem for us as we are required per IA policy to have FIPS enabled.  So I had to figure out how to get the service to start while leaving FIPS enabled.  After further troubleshooting I discovered that if I disabled all of the encryption modules that freeRADIUS attempts to load; such as EAP, EAP-TLS, etc., but left MD5 enabled then the service would run and I was still able to leave FIPS enabled on the server. 

But as you likely know, MD5 is not an approved FIPS encryption standard.  Which begs the question, how on earth can freeRADIUS run using MD5 on a server that has FIPS enabled?  Well, I also learned that freeRADIUS implements it's own internal MD5 encryption module and therefore is able to function without the FIPS kernel module getting in the way.

So essentially, I have a freeRADIUS server that is only configured to use MD5 but also has FIPS enabled. 

So now it's time to configure the switch.  It didn't occur to me to check whether FIPS was enabled on the switch.  So the freeRADIUS  server wants to talk the switch using MD5 while the switch itself wants to use FIPS approved algorithms.  There's a conflict there and things go to crap.  So if I disable FIPS on the switch everything is happy.

Now I've got the IA security folks to deal with.  They want FIPS enabled on the switch, but they also want 802.1x.  At this point in time they get one or the other.

My next task is to figure out how to get the RADIUS server to run with FIPS and NOT using MD5 but rather an algorithm that is actually FIPS approved.  After I do that, I can re-enable FIPS on the switch.

We are in the process of upgrading our hardened Redhat image to version 7.  I've been given a hardened Redhat 7 image to play with and I've found that I can in fact start the freeRADIUS service (raddb) without having to disable any algorithms.  So that's something.