Is there anyone using wired 802.1x in production?

[jimmie25h69]

I have 802.1x configured with PEAP and vlan assignment using the MS supplicant. I have hardcoded Machine Auth because Remote Desktop does not work with User Auth. (see my other posts) I have figured out how to change the Microsoft supplicant to PEAP with a vb script. I have a catalyst 6509 enabled with 802.1x enabled on module 8 and 9 with about 60 actual PCs authenticated between to 2 modules. At this point I am testing ACS redundancy (2 ACS SE) and any potential ACS load issues before campus deployment.

My problem:

1. If I reset module 8 or 9 the switch reloads. I guess it is overwhelmed by all the 802.1x requests. I am not too concerned about this right now because this type of product quality is very common in this new era. I have not reset another module in this switch that does not have 802.1x enabled. Though, I have reset modules in the past.

2. My main concern is the amount of time it takes for all the ports on a given module to finally 802.1x authenticate. To avoid problem (1.), I disabled/enabled all ports on module 8. It takes about 8-12 minutes before all 30+ ports are authenticated. This behavior is the same after the reload.

It takes about 10 seconds for one supplicant to authenticate. It appears the switch is serializing the logins in a loop until all are authenticated. I calculate 384 ports * 10 seconds = 1.06 hour to authenticate after reload (best case).

Please share your experiences or ideas.

[s-doyle]

The time delay that you are seeing is just the way the 802.1x authentication works and there is nothing we can do about it

[chilinh]

We have dot1x enabled in production. Over thousand of supplicants. Terrible!! I found last week that while we reset one of the modules; the switch crashed. Because we have 2 Sup in one box, the switch did not reload but failed over to the standby SUP.

We have CatOS 8.4.1 and 8.4.5 in our environment; 2 ACSes for redundant purpose.

I did not perceive the dealy you mention.We found that a lot of supplicants could not be authenticated.

Below is the trace that I found when switches crashed. Do you see the same output?

pantree port fast start set to default for ports 6/29,6/34.

QRDCN05ACC01> (enable) set vlan 461 System reset on software watchdog is disabled

TLB Exception (load/instruction fetch) occurred on Sep 26 2005 16:15:00

Software version = 8.4(1)

Process ID #4c, Name = Backend_SM

process stack top = 3ff1b170, stack pointer = 3ff1b0e8

cause = 00000008

TLB Exception (load/instruction fetch) exception happened

EPC: 210274A0

Traceback:

210274A0

210274A0

Stack content:

sp+00: 00000006 00000006 21026C38 202ECAB0

sp+10: 267AD970 22F9B290 22FA0000 00000006

sp+20: 0000001D 00000030 00000005 21027230

sp+30: 0000012C 00000000 20B47BDC 20B47BDC

sp+40: 00000000 00000000 00000000 00000000

sp+50: 00000000 00000000 00000000 00000000

sp+60: 3FF1B150 20B4A2A0 20B47BDC 20B47BDC

sp+70: 20B47BDC 20B47BDC 00000007 20B47BDC

sp+80: 00000000 20B4A250 20B47BDC 20B47BDC

sp+90: 20B47BDC 20B47BDC 20B47BDC 20B47BDC

sp+A0: 20B47BDC 20B47BDC 20B47BDC 20B47BDC

sp+B0: 20B47BDC 20B47BDC 20B47BDC 20B47BDC

sp+C0: 20B47BDC 20B47BDC 20B47BDC 20B47BDC

sp+D0: 20B47BDC 20B47BDC 20B47BDC 20B47BDC

sp+E0: 20B47BDC 20B47BDC 20B47BDC 20B47BDC

sp+F0: 20B47BDC 20B47BDC 20B47BDC 20B47BDC

Register content:

Status: 3400FC23 Cause: 00800008

AT: 22830000

V0: 00000001 V1: 267AD970

A0: 00000006 A1: 0000001D

A2: 0000001C A3: 22FA0000

T0: 23C00BC0 T1: 3FFFF070

T2: 00000001 T3: 00000007

T4: 00007080 T5: 00000000

T6: 00800000 T7: F03FFFFF

S0: 22F9B290 S1: 00000000

S2: 00000006 S3: 0000001D

S4: 00000005 S5: 0000001C

S6: 22FA0000 S7: 0000000D

T8: FFFFFFFF T9: 4B34A6A4

K0: 30409001 K1: 215016E8

GP: 2283AC70 SP: 3FF1B0E8

S8: 00000007 RA: 2102742C

HIGH: 0000001A LOW: 0355485E

BADVADDR: 00000002 ERR EPC: A3A3A3A3

Total download memory used = 3989996

crash info filename is bootflash:crashinfo_050926-161503

Opening crash info file bootflash:crashinfo_050926-161503

Time took to write crashinfo = 00:05.09

crashinfo finished

[jimmie25h69]

It sounds like we have a similar configuration and similar problem. I also found that "a lot of supplicants could not be authenticated". I think it is because of a timeout that appears to be within the 6509 chassis. This is what I did to expose the timeout.

I enabled security logging debug. I disabled/enabled (1) port to see the normal output. I incremented by (1) port until I reached (4) ports. At (4) ports there is a TIMEOUT entry. 802.1x retries and the port eventually authenticates. BUT when I authenticate up to 48 ports most ports do not authenticate. I believe the MS supplicant quits trying at some point. Give it a try and let me know if you get the timeouts. What hardware do you have 6000 or 4000 series? I have a case open.

set logging level security 7

2005 Sep 24 09:36:49 CDT -05:00 %SECURITY-7-DOT1X_BACKEND_STATE:DOT1X: backend state for port 8/41 is TIMEOUT

[jimmie25h69]

8.4(5) fixed both.

bug ID CSCeh95025 - 802.1x simultaneous authentications fail

[brford]

What version of ACS are you using? What Catalyst operating system and Supervisor are you using? What version of Catalyst OS are you using? Which Microsoft operating system(s) are you using and approximately what service pack are you running?

What are you configured to do after you authenticate a device? Are you changing VLANs?

[jimmie25h69]

What version of ACS are you using?

ACS SE 1112

Cisco Secure ACS 3.3.2.2

Appliance Management Software 3.3.2.1

Appliance Base Image 3.3.1.6

CSA build 4.0.1.543.2 (Patch: 4_0_1_543)

And

ACS SE 1111

Cisco Secure ACS 3.3.2.2

Appliance Management Software 3.3.2.1

Appliance Base Image 3.3.1.1-HP

CSA build 4.0.1.543.2 (Patch: 4_0_1_543)

What Catalyst operating system and Supervisor are you using?

8.4(5), SUP2/PFC2

Crashed once when 802.1x was disabled on a port after upgrading to 8.4(5)

"set port dot1x 7/26 port-control force-authorized"

Possibly related to bugs CSCei80863 and CSCsc02053

What version of Catalyst OS are you using?

WS-C6509 Software, Version NmpSW: 8.4(5)

Copyright (c) 1995-2005 by Cisco Systems

NMP S/W compiled on Aug 3 2005, 12:01:19

Which Microsoft operating system(s) are you using and approximately what service pack are you running?

All are XP SP2. updates are current within 30-45 days. The PCs are PEAP with registry keys AuthMode=2, SupplicantMode=3

What are you configured to do after you authenticate a device?

Authenticating a PC only once unless port state changes.

Dynamic vlan assignment

Are you changing VLANs?

yes