Cisco ACS 5.1 802.1x auth fails on LAN when WLAN connected

[at2885]

I am running Cisco ACS 5.1 802.1x with certificate based authentication for Wired and Wireless connections. The issue that I am having is that when a user comes in from home with their laptop the wireless connection works, they pass the authentication and have network access fine. But when the plug the laptop into a docking station the LAN connection fails and gets put in the Auth Failure Vlan. 

A reboot of the phone/ shut/no shut fixes this, but I really need to find a resolution

This is an intermittent fault and only effects users with both LAN and WLAN enabled.

Running ACS 5.1.0.44, all Cisco 3750s - c3750-ipservicesk9-mz.122-55.SE.

Certificates are issues by group policy and only using computer authentication.

any help would be greatly appreciated

Thanks

[at2885]

I forgot to mention, we are running Mitel 3300 MCD 5, with Mitel 5330 phones. The problem we are having with a Laptop plugged into the back of a phone,

[at2885]

After a long TAC case with Cisco we discovered that the Mitel phone was not sending the EAPoL-Logoff packet so the switch still thought that the device off the back of the phone was connected.

There are no EAPoL-Logoff messages seen on switch when laptop is disconnected/port is shut down.

http://www.cisco.com/en/US/docs/solutions/Enterprise/Security/TrustSec_1.99/Dot1X_Deployment/Dot1x_Dep_Guide.html#wp386903

This feature is supported by most IP phones -  I do not know if Mitel phones support that but we cannot see this message in the debugs you sent.

As a workaround we can configure inactivity timer (by default it is infinity):

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.2_55_se/commmand/reference/cli1.html#wp11888691

This did resolve all our issues,

Aaron

[lewiscf77]

Hi Aaron, just a quick question. Was this resolved or are you still using the workaround? We are seeing the same / similar problem.

Craig

[at2885]

Hi Craig

I am afraid I am still using the workaround and have had to on subsequent deployments as well, the limitation is on the Mitel side so until they address the issue it maybe the best option. I came across someone else that had a similar issue on my travels and they addresses it by using error disable recovery. Something like, errdisable recovery cause security-violation do deal with it, the downside to this is I think the port drops so if you are using a POE handset it will re-boot, but depending on you the size of your organisation this maybe between that a lot of re-auth request.

Anything else on this please just let me know.

Aaron

[lewiscf77]

Aaron, many thanks for the incredibly quick response. We have spent a considerable time looking at this. We had been advised that this issue was resolved with a later phone firmware version. :-(

We will implement the workout, as it sounds like the other one won’t help us as we are running POE switches.

Craig

[at2885]

No problem at all. I came across this about 3 years ago now and I am sure they said something similar then. I work for a Mitel and Cisco partner so managed to get both involved in the troubleshooting at the time, but have not investigated since. What MCD release are you on? I had loads of other issues on pre MCD 4 as well.

  I have used the re-auth timer a few times now on separate deployments and never had any issues so for now that's a safe bet

Aaron

[yangsonggui]

Hi  aaron.tunnicliff :

I have a similiar problem with your, below is the detail. 

I am confused some failure authentication session could not disappeared even this failure MAC address did not find in the MAC address table or it did not connect to the switch. kindly hope you give me some adivise about this issue, thanks!

I am running Cisco ACS( Version : 5.4.0.46.0a)  802.1x with certificate based authentication for Wired connections. the issus is i found some authentication failed messages in some switch port. when I troubleshooting in ACS, it is an error: "22056 Subject not found in the applicable identity store(s). : Authentication failed ". but I could not find the MAC address on this port.  the authentication failed message should disappeared after 60 seconds normally it the device pull out the cable. but i found the authentication failed session always in the switch and the ACS. 

for example:

in the port Gi1/0/15, there has an Avaya phone and a PC authentication success, but there has another MAC address failed. it was strange the this port did not connect any other device. so i am so confused about this situation. i tried to add one command :"authentication timer inactivity 30", but it seem like no use.

switch#show authe se | inc Gi1/0/15
Gi1/0/15   90b1.1c9b.d9c4  dot1x    DATA     Authz Success  0A19F5820001536935ED8383
Gi1/0/15   24d9.214e.39be  dot1x    VOICE    Authz Success  0A19F5820001452D31ECA0FD
Gi1/0/15   8c70.5a29.39be  dot1x    DATA     Authz Failed        0A19F582000150163568626F
switch#show mac add | inc Gi1/0/15
 100    90b1.1c9b.d9c4    STATIC      Gi1/0/15
 300    24d9.214e.39be    STATIC      Gi1/0/15

switch#show run int Gi1/0/15
Building configuration...

Current configuration : 540 bytes
!
interface GigabitEthernet1/0/15
 switchport access vlan 100
 switchport mode access
 switchport voice vlan 300
 duplex full
 authentication event server dead action reinitialize vlan 100
 authentication event server dead action authorize voice
 authentication host-mode multi-auth
 authentication order dot1x mab
 authentication port-control auto

 authentication timer inactivity 30
 mab
 dot1x pae authenticator
 dot1x timeout tx-period 10
 storm-control broadcast level 5.00
 spanning-tree portfast
 spanning-tree bpduguard enable

switch module: WS-C3750X-48PF-S

switch IOS: c3750e-universalk9-mz.150-2.SE4.bin

[at2885]

Hello

This is my default port config,

 description 802.1x Voice and Data
 switchport mode access
 switchport voice vlan 100
 srr-queue bandwidth share 10 10 60 20
 srr-queue bandwidth shape 10 0 0 0
 priority-queue out
 authentication event fail action authorize vlan 112
 authentication event server dead action authorize vlan 1
 authentication event server dead action authorize voice
 authentication event no-response action authorize vlan 112
 authentication event server alive action reinitialize
 authentication host-mode multi-domain
 authentication order mab dot1x
 authentication port-control auto
 authentication timer inactivity 3600
 mab
 mls qos trust cos
 auto qos voip trust
 dot1x pae authenticator
 dot1x timeout tx-period 3
 spanning-tree portfast

I notice that you are using Authentication host-mode multi-auth, I would typically use this if I had a L2 switch of a normal switch port

multi-auth—Multiauthentication allows one authentication on a voice VLAN and multiple authentications on the data VLAN

This does not explain why you are seeing an additional MAC, does it show in the mac address table at all?

Maybe try swapping over to use multi-domain and see if that helps?

I have also ran into many bugs in the past so I would rule that out either..

Aaron

[yangsonggui]

Thanks! Aaron.

the additional MAC could not found int the mac address table. and i have check the switch port, it just connect one Avaya phone and the end user laptop, no other device. I monitored the port, it did not have the additional MAC but it suddenly appeared and I have no idear about it. and the most important it this failure authentication session could not clear until I manually "clear authentication session mac x.x.x.x". 

[at2885]

Hello

Is this just one phone\port in a fully operational deployment or are you still trialling it on a few users?

I think the next thing to do is clear the authentication on this port and unplug both devices. The run, debug authentication all, reconnect the devices and see if it happens again. Then send post the logs.

Do you see the failed mac address in your ACS logs?

Aaron

[Peter Koltl]

Strange, because I experienced that as soon as WLAN connection is established, the PC stops running 802.1X on the LAN NIC.

[yangsonggui]

Hi Aaron:

our office have 5 floores in the office building, and we have 4 or 5 switches stacked every floor. and we find several users have this porblem in every floor.

all of the failed mac address in the ACS is the same error message "22056 Subject not found in the applicable identity store(s). : Authentication failed". 

it if difficult for me to debug in the switch.

do you know this command "authentication mac-move permit"?  I am not sure whether this command could fix the problem or not.

Enabling MAC Move

MAC move allows an authenticated host to move from one port on the switch to another.

Beginning in privileged EXEC mode, follow these steps to globally enable MAC move on the switch. This procedure is optional.

	Command	Purpose
Step 1	configure terminal	Enter global configuration mode.
Step 2	authentication mac-move permit	Enable MAC move on the switch.
Step 3	end	Return to privileged EXEC mode.
Step 4	show running-config	Verify your entries.
Step 5	copy running-config startup-config	(Optional) Save your entries in the configuration file.

This example shows how to globally enable MAC move on a switch:

Switch(config)# authentication mac-move permit

[at2885]

Hello

Yes I do tend to use mac-move permit if laptop users are using wired, in a hot-desking sort of situation it allows a mac address to appear on multiple switch ports.

In the ACS logs, does it tell you what subject is being offered up that is not found? 

Thanks

Aaron