Avaya 9650 phones not clearing failed dot1x authentication sessions (ISE 2.0.1)

[kuzminsk1]

Hi Guys

We have just implemented Cisco ISE 2.0.1 and configured mab and dot1x policies. Unfortunately, there is one problem we are really struggling with.

Avaya 9650 phones are connecting directly to the switch and windows 7 clients are connected to the Avaya phones in pass-through mode. Avaya phones are set to "802.1x passthrough /w Logoff". This is a settling that sends a notification to the switch when a device is disconnected from the phone. This setting works ok for all successful mab and dot1x authentications. The windows 7 clients' authentication sessions dissapear from the switch as soon as they are disconnected from the back of the phone.

The phones are configured as Non-Cisco Ip Telephony Devices in the ISE authorization policy.

The problem begins when a Windows 7 client with a miss-configured / missing supplicant is connected to the avaya phone. The following takes place:

Dot1x policy blocks access in ISE as per auth policy. They are profiled and authenticated using MAB.

On the switch, Status is "Running" for about 60 seconds, then it changes to "Authz Failed"

Interface MAC Address Method Domain Status Session ID
Fa0/5 3c97.0eee.9f89 dot1x UNKNOWN Running 0A1204180000018F14CD2B02
Fa0/5 001b.4f24.2f36 mab VOICE Authz Success 0A1204180000018D14CAFCA4
Fa0/6 b4b0.1795.b092 mab VOICE Authz Success 0A1204180000016914580713
Fa0/17 2884.fa6f.79f5 mab DATA Authz Success 0A1204180000017314B8C721

BWY_01_A07#show auth sess

Interface MAC Address Method Domain Status Session ID
Fa0/5 3c97.0eee.9f89 N/A DATA Authz Failed 0A1204180000018F14CD2B02
Fa0/5 001b.4f24.2f36 mab VOICE Authz Success 0A1204180000018D14CAFCA4
Fa0/6 b4b0.1795.b092 mab VOICE Authz Success 0A1204180000016914580713
Fa0/17 2884.fa6f.79f5 mab DATA Authz Success 0A1204180000017314B8C721

The problem is that no matter when you unplug the unauthorised Windows 7 client from the back of the Avaya phone, the failed session does not clear. Furtermore, it continues in a cycle of Running / Authz Failed indefinitely (at least 8 hours confirmed) after the device has long been disconnected. This means the device that has been disconnected a long time ago is popping up in the ISE logs every few mins.

The only ways to resolve this are rebooting the phone or running "clear auth session" from the switch CLI. This problem does not occur when the Win 7 clients are connected directly to the switch.

http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Security/TrustSec_1-99/IP_Tele/IP_Telephony_DIG.html

https://downloads.avaya.com/elmodocs2/one-X_Deskphone_Edition/R1.5/output/16_300698_4/admn0710.html

Has anyone got any experience of ISE deployments with Avaya phones and Win 7 clients (with native supplicants using PKI and computer authentication)?

FULL PORT CONFIG, using show run all CLI:

interface FastEthernet0/5
description Avaya Phone Connection
switchport
switchport access vlan 6
switchport mode access
no switchport nonegotiate
no switchport protected
no switchport block multicast
no switchport block unicast
switchport voice vlan 16
no ip arp inspection trust
ip arp inspection limit rate 15 burst interval 1
ip arp inspection limit rate 15
carrier-delay 4
no shutdown
power inline consumption 15400
power inline auto max 15400
power inline police
srr-queue bandwidth share 1 70 25 5
srr-queue bandwidth shape 30 0 0 0
authentication control-direction both
authentication event fail retry 2 action next-method
authentication event server alive action reinitialize
authentication host-mode multi-auth
no authentication open
authentication linksec policy should-secure
authentication order mab dot1x
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer restart 60
authentication timer reauthenticate 3600
authentication timer inactivity 5                    /////////////////i've tried multiple timers here, but it doesnt seem to make a difference
authentication violation restrict
no authentication fallback
mab radius
mls qos cos 0
mls qos trust dscp
mls qos dscp-mutation Default DSCP Mutation Map
snmp trap mac-notification change added
snmp trap mac-notification change removed
no snmp trap link-status
dot1x pae authenticator
dot1x timeout quiet-period 60                    /////////////////i've tried multiple timers here, but it doesnt seem to make a difference
dot1x timeout server-timeout 0
dot1x timeout tx-period 30                    /////////////////i've tried multiple timers here, but it doesnt seem to make a difference
dot1x timeout supp-timeout 30
dot1x timeout ratelimit-period 0
dot1x max-req 2
dot1x max-reauth-req 2
dot1x timeout start-period 30
dot1x timeout held-period 60                    /////////////////i've tried multiple timers here, but it doesnt seem to make a difference
dot1x timeout auth-period 30                    /////////////////i've tried multiple timers here, but it doesnt seem to make a difference
dot1x max-start 3
cdp tlv location
cdp tlv server-location
cdp tlv app
spanning-tree portfast disable
spanning-tree portfast trunk
spanning-tree portfast
spanning-tree port-priority 3
spanning-tree cost 3
ip igmp snooping tcn flood

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 26 WS-C2960-24PC-L 12.2(55)SE10 C2960-LANBASEK9-M

!

Any help would be greatly appreciated..

Thanks

M

[kuzminsk1]

So we have narrowed down the problem.

The phones a supposed to send an EAPOL frame to teh switch when a client is physically disconnected from a phone.

In the event of a successful dot1x authentication followed by a client disconnection from phone, the EAPOL frame is sent by the phone and correctly processed by the switch on its current IOS firmware. I believe this rules out the argument about the switch not processing EAPOL notifications correctly due to a particular IOS version (we have seen the same behaviour on 3 x IOS versions so far).

The same successfully authenticated dot1x client does not generate an EAPOL frame when it is disconnected from the phone. i.e. the phone acknowledges the physical disconnection and sends the EAPOL frame to the switch.

Unauthenticated dot1x client disconnections do not get acknowledged by the phone. The switch then believes they are still connected to the phone.

[kuzminsk1]

Hi

Long story short, Avaya phones are unable to send the EAPOL frame the switch is expecting when the client to disconnects from the phones hub. After capturing detailed evidence of the problem at Avaya's request, Avaya support formally refused to help me, citing incompatibility of the older [in full support at the time] firmware.

So...we had to use a workaround. Since the only real problem the dead sessions created was the clients inability to unplug from one phone and plug into another due to mac address duplication in the dot1x table, we introduced the following command in the global configure of the the access switches:

authentication mac-move permit

(Please check the syntax)

In order to get rid of the dead sessions we created a kron scheduler task for clear authentication  sessions to run every night. Since not that many people in BAU are unplugging from one phone and plugging into the next, this solution was accepted by the business.

If you use the scheduler task, make sure you rest all of the sleeping devices like printers etc.

As for the comment about random sessions timing out/hanging, we had that problem until we set the authentication method next command to keep trying the next authentication method (mab and dot1x) so any failed session attempts to RE-authenticate every 30 seconds or so 

[#4]

HI, 

Let me answer this for you. 

What i understand that when computer does a MAB authentication as it is not having correct dot1x  policy configured on computer. 

Now even if you configure EAPOL logoff. what happens This works for dot1x authentication only.

For MAB authentication you need to configure attribute 28 ( inactivity timer)  on radius for MAB. 

Which means that when MAB authentication occurs  the radius sends out a timeout to switch for inactivity. which you can check as per your requirement .

Else switch would take the default timers that are configured on switch for MAB timeout. 

The 802.1x design documents has it ; 

https://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Security/TrustSec_1-99/IP_Tele/IP_Telephony_DIG.html

====================================================

wording from the documents 

Proxy EAPoL-Logoff

If your switch or phone does not support CDP Enhancement for Second Port Disconnect, Proxy EAPoL-Logoff can provide a partial solution for 802.1X-authenticated data devices. Proxy EAPoL-Logoff enables the phone to transmit an EAPoL-Logoff message on behalf of the data device when the phone detects that an 802.1X device has unplugged from behind the phone. The phone substitutes the MAC address of the data device, so the proxy EAPoL-Logoff message is indistinguishable from an actual EAPoL-Logoff from the data device itself. The switch immediately clears the session as soon as it receives the Logoff message.

To support this feature, your phone must be capable of sending proxy EAPoL-Logoff messages. All Cisco IP phones and some third-party phones provide this functionality. No special functionality is required from the switch because the EAPoL-Logoff message is fully supported as per the IEEE standard.

Although effective for 802.1X-authenticated endpoints, Proxy EAPoL-Logoff does not work for MAB or WebAuth, because these authentication methods do not use EAP to authenticate. Another method, such as the inactivity timer, must be used to ensure that MAB sessions are appropriately cleared.

Inactivity Timer

If your switch or phone does not support CDP Enhancement for Second Port Disconnect, the inactivity timer can provide a partial solution for disconnected data devices. When the inactivity timer is enabled, the switch monitors the activity from authenticated endpoints. When a device disconnects, the inactivity timer counts down. When the timer expires, the switch removes the authenticated session. The inactivity timer applies to 802.1X and MAB sessions.

---

Note The current implementation of WebAuth uses a different, IP Device Tracking-based inactivity timer than 802.1X and MAB. This timer clears the WebAuth authorization state, but it does not clear the entire session state. The only way to clear a WebAuth session behind a phone is to use the CDP Enhancement for Second Port Disconnect.

Regards 

Faisal Khan 

Faisalkhn@hotmail.com

[gabriel.barrios]

same problem here. When a user leaves the wired connection and comes back, it may appear the issue. 

We also have the BIOS setting to switch from wired-to-wireless or vice versa enabled (it disables wifi when wired connection is present).

still looking, 

did you find the solution?

thanks!!

[Ben Jenkins]

Also seeing this issue not just on Avaya 9650 but all Avaya models that we have.  We dont just see this for failed devices, we see random successfully authenticated devices have this issue.  Mainly when the users place the laptop in Hibernate mode.  I have seen accounting stop messages sent from switch to ISE with a response, then a minute later the switch starts sending RADIUS requests to ISE even though the laptop is not connected!

Not seeing this with Cisco IP Phones (CDP Enhanced) so looking like Avaya issue at the mo.

Anyone with a fix for this would be awesome as its stopping our deployment.  

Also added inactivity timers to switch port but doesn't do anything!