Hi guys,
I am not sure if I am hitting IOS bug CSCtx61557
according to the bug tool this is the info:
-----------------------------
crash after authc result 'success' from 'dot1x' for client (Unknown MAC)
CSCtx61557
Description
Symptoms: The switch crashes after logging "success" from "dot1x" for client
(Unknown MAC).
Conditions: The symptom is observed with the following conditions:
1. A switchport is configured with both of the following:
authentication event server dead action authorize...
authentication event server alive action reinitalize
2. The radius server was down previously, and a port without traffic (for
example: a hub with no devices attached) was authorized into the inaccessible
authentication bypass (IAB) VLAN without an associated MAC address.
3. The radius server becomes available again, and a dot1x client
attempts to authenticate.
Workaround: There is no workaround.
--------------------------------------------------------------
I am running the following IOS on my 4500X-16 SFP+:
cat4500e-universalk9.SPA.03.05.03.E.152-1.E3.bin
-------------------------------------------------------------
This is what I configured, and what happened:
HOSTNAME(config)#aaa group server radius rad_eap
HOSTNAME(config-sg-radius)# server name ACS1
HOSTNAME(config-sg-radius)# server name ACS2
HOSTNAME(config-sg-radius)# server name ACS3
HOSTNAME(config-sg-radius)#$ication login default group radius local
HOSTNAME(config)#aaa authentication login CONSOLE local
HOSTNAME(config)#aaa authentication enable default group radius enable
HOSTNAME(config)#aaa authentication ppp default local group radius
HOSTNAME(config)#aaa authentication dot1x default group radius
HOSTNAME(config)#aaa authorization exec default if-authenticated
HOSTNAME(config)#aaa authorization network default group radius
HOSTNAME(config)#aaa accounting update newinfo
HOSTNAME(config)#aaa accounting dot1x default start-stop group radius
HOSTNAME(config)#aaa accounting network default start-stop group
eption to IOS Thread:
Frame pointer 897BAE38, PC = 1C03EECC
IOSD-EXT-SIGNAL: Aborted(6), Process = Exec
-Traceback= 1#49176b00b95a50f3145e3825de17d470 c:1C008000+36ECC c:1C008000+3BE50 c:1C008000+3BF48 :1F679000+201A18C :1F679000+31CEE2C :1F679000+2C22958 :1F679000+2C293E4 :1F679000+1166260 :1F679000+2C3C20C
Fastpath Thread backtrace:
-Traceback= 1#49176b00b95a50f3145e3825de17d470 uld:1F224000+2DE8 uld:1F224000+2DE4 iosd_unix:1C3ED000+186A0 pthread:1AA69000+6450
Auxiliary Thread backtrace:
-Traceback= 1#49176b00b95a50f3145e3825de17d470 pthread:1AA69000+BB8C pthread:1AA69000+BB6C c:1C008000+F61E4 iosd_unix:1C3ED000+21270 pthread:1AA69000+6450
Buffered messages: (last 8192 bytes only)
6 left the port-channel Port radius
HOSTNAME(config)#aaa accounting system default start-stop group radius
HOSTNAME(config)#
HOSTNAME(config)#
HOSTNAME(config)#no authentication logging verbose
HOSTNAME(config)#
HOSTNAME(config)#
HOSTNAME(config)#login block-for 300 attempts 5 within 60
-channel1
*Aug 28 01:08:47.873 UTC: %C4K_IOSINTF-5-LMPHWSESSIONSTATE: Lmp HW session DOWN on slot 11 port 12.
*Aug 28 01:08:48.056 UTC: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 172.16.5.98 port 514 started - CLI initiated
*Aug 28 01:08:48.571 UTC: %FASTHELLO-2-FH_DOWN: Fast-Hello interface Te2/1/12 lost dual-active detection capability
*Aug 28 01:08:49.099 UTC: %PIM-5-DRCHG: DR change from neighbor 0.0.0.0 to 172.16.250.61 on interface Vlan250
*Aug 28 01:15:08.753 UTC: %C4K_IOSINTF-5-LMPHWSESSIONSTATE: Lmp HW session UP on slot 11 port 1.
*Aug 28 01:15:24.759 UTC: %VSLP-5-VSL_UP: Ready for control traffic
*Aug 28 01:15:27.760 UTC: %VSLP-5-RRP_ROLE_RESOLVED: Role resolved as ACTIVE by VSLP
*Aug 28 01:15:27.760 UTC: %EC-5-BUNDLE: Interface TenGigabitEthernet2/1/1 joined port-channel Port-channel2
*Aug 28 01:15:28.049 UTC: %C4K_REDUNDANCY-6-DUPLEX_M
<Thu Aug 28 01:18:32 2014> Message from sysmgr: Reason Code:[2] Reset Reason:Service [iosd] pid:[6813] terminated abnormally [6].
Details:
--------
Service: IOSd service
Description: IOS daemon
Executable: /tmp/sw/mount/cat4500e-universalk9.SPA.152-1.E.pkg//usr/binos/bin/iosd
Started at Wed Aug 27 22:27:48 2014 (647795 us)
Stopped at Thu Aug 28 01:18:32 2014 (115506 us)
Uptime: 2 hours 50 minutes 44 seconds
Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago
PID: 6813
Exit code: signal 6 (no core)
CWD: /var/sysmgr/work
PID: 6813
UUID: 512
FAILURE: syslogd shutdown
I had a ICMP ping going, and it was not affected, as the Standby VSS chassis kicked in and took over, while the previous active chassis reloaded.
--------------------------------------------------------
2nd time it happened:
Now this time, I had waited until the previous active chassis was back up and running and came back up as Standby hot.
once again I pasted the same config, and bang, It happened a second time on the second chassis which was acting now as Active supervisor.
And once again, the ICMP continuous ping was not interrupted, as the other chassis remained up, while the "new" active crashed after configuring the same configs in a slight different order.
HOSTNAME(config)#radius server ACS2
HOSTNAME(config-radius-server)#$5.22 auth-port 1812 acct-port 1813
HOSTNAME(config-radius-server)# timeout 1
HOSTNAME(config-radius-server)# key 0 XXXX
HOSTNAME(config-radius-server)#!
HOSTNAME(config-radius-server)#radius server ACS3
HOSTNAME(config-radius-server)#$xxxx auth-port 1812 acct-port 1813
HOSTNAME(config-radius-server)# timeout 1
HOSTNAME(config-radius-server)# key 0 xxxxxxx
HOSTNAME(config-radius-server)#
HOSTNAME(config-radius-server)#aaa group server radius rad_eap
HOSTNAME(config-sg-radius)# server name XXXX
HOSTNAME(config-sg-radius)# server name XXXX
HOSTNAME(config-sg-radius)# server name XXXX
HOSTNAME(config-sg-radius)#
HOSTNAME(config-sg-radius)#
PER-3-S
Exception to IOS Thread:
Frame pointer 89455E38, PC = 1CC27ECC
IOSD-EXT-SIGNAL: Aborted(6), Process = Exec
-Traceback= 1#e495ba4f9346cc1496eecd01ebf1814a c:1CBF1000+36ECC c:1CBF1000+3BE50 c:1CBF1000+3BF48 :20276000+201B18C :20276000+31D0DA8 :20276000+2C24800 :20276000+2C2B28C :20276000+11671B0 :20276000+2C3E0B4
Fastpath Thread backtrace:
-Traceback= 1#e495ba4f9346cc1496eecd01ebf1814a iosd_unix:1CFD6000+1C230 iosd_unix:1CFD6000+1C284 iosd_unix:1CFD6000+18854 pthread:1B653000+6450
Auxiliary Thread backtrace:
-Traceback= 1#e495ba4f9346cc1496eecd01ebf1814a pthread:1B653000+BB8C pthread:1B653000+BB6C c:1CBF1000+F61E4 iosd_unix:1CFD6000+21270 pthread:1B653000+6450
Buffered messages: (last 8192 bytes only)
INTF-5-TRANSCEIVERINSERTED: Slot=11 Port=3: Transceiver hasW-9(config-sg-radius)#
HOSTNAME(config-sg-radius)#no authentication logging verbose
HOSTNAME(config)#
HOSTNAME(config)#
HOSTNAME(config)#login block-for 300 attempts 5 within 60
been inserted
*Aug 28 01:26:03.864 UTC: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=11 Port=4: Transceiver has been inserted
*Aug 28 01:26:03.864 UTC: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=11 Port=5: Transceiver has been inserted
*Aug 28 01:26:03.864 UTC: %C4K_IO
<Thu Aug 28 01:28:10 2014> Message from sysmgr: Reason Code:[2] Reset Reason:Service [iosd] pid:[6770] terminated abnormally [6].
Details:
--------
Service: IOSd service
Description: IOS daemon
Executable: /tmp/sw/mount/cat4500e-universalk9.SPA.152-1.E3.pkg//usr/binos/bin/iosd
Started at Thu Aug 28 01:13:52 2014 (60006 us)
Stopped at Thu Aug 28 01:28:10 2014 (993041 us)
Uptime: 14 minutes 18 seconds
Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago
PID: 6770
Exit code: signal 6 (no core)
CWD: /var/sysmgr/work
------------------------------------------------
are these the symptoms related to CSCtx61557 ?
I have tested this in a test environment, where no ACS was reachable!
Thanks
Colin
