3560 Crashed on Putting Dot1x Configuration

[Muhammad Kashif Siddiqui]

I have about 300 sites which are having more than 500 switches. I am in process to deploy Dot1x. I am configuring all the switches remotely one by one. While putting the configurations sometimes switch gets crash. This is all random. Like there is a possibility that one switch crashes on one site and one on another. I am using different IOS in different switches. I am unable to find out any exact bug which can hit under my environment.

One the switch come back after reload it gives below message in "show version"

!

System returned to ROM by bus error at PC 0x2AF251C, address 0x0
System restarted at 11:00:46 UAE Tue Mar 24 2015
System image file is "flash:/c3560-ipbasek9-mz.122-55.SE4.bin"

!

IOS : 12.2(55)SE5

IOS : 12.2(55)SE4

IOS : 12.2(35)SE5

interface FastEthernet0/x
 switchport access vlan 101
 switchport mode access
 switchport voice vlan 102
 ip access-group auth-default-acl in
 authentication event server dead action authorize vlan 101
 authentication host-mode multi-auth
 authentication order mab dot1x
 authentication priority dot1x mab
 authentication port-control auto
 mab
 dot1x pae authenticator
 dot1x timeout quiet-period 5
 dot1x timeout tx-period 5
 dot1x max-req 3
 dot1x max-reauth-req 3
 spanning-tree portfast

aaa group server tacacs+ SECURE
 server a.b.c.d
 server w.x.y.z

!
aaa authentication login CONSOLE local
aaa authentication login SECURE group tacacs+ local
aaa authentication enable default group tacacs+ enable
aaa authentication dot1x default group radius
aaa authorization console
aaa authorization config-commands
aaa authorization exec default group tacacs+ local
aaa authorization exec CONSOLE local
aaa authorization commands 15 NW group tacacs+ local none
aaa authorization network default group radius
aaa accounting dot1x default start-stop group radius
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+

!

aaa server radius dynamic-author
 client a.b.c.d server-key 7 <removed>
 client w.x.y.z server-key 7 <removed>
 port 3799
 auth-type all

radius-server attribute 44 extend-with-addr
radius-server attribute 8 include-in-access-req
radius-server dead-criteria time 3 tries 1
radius-server host a.b.c.d auth-port 1812 acct-port 1813
radius-server key 7 <removed>
radius-server vsa send accounting
radius-server vsa send authentication
 

[Leo Laohoo]

12.2(55)SE IOS didn't "mature" until 12.2(55)SE8.  So try to the latest version of 12.2(55)SE10.

[Muhammad Kashif Siddiqui]

Hi Leo,

Thanks for your reply. Your statement is based on experience but unfortunately I will not be able to simply state this to Customer. I would highly appreciate if you can share some document to prove this.

Thanks/Regards

[Leo Laohoo]

I would highly appreciate if you can share some document to prove this.

If you want a list of known bugs and security vulnerabilities your fleet's IOS is running on, use Cisco Bug Toolkit.

[Muhammad Kashif Siddiqui]

Hi Leo,

Once again thanks for your reply. Can you check the below part taken from Crashinfo...Do u have any idea what it is stating ?

---- Partial decode of process block ----

Pid 306: Process "LLDP Protocol "
stack 0x46CCD5C  savedsp 0x3DC8B74
Flags: analyze prefers_new
Status     0x00000000 Orig_ra   0x00000000 Routine    0x00000000 Signal 0
Caller_pc  0x00000000 Callee_pc 0x00000000 Dbg_events 0x00000000 State  0
Totmalloc  513148     Totfree   110776     Totgetbuf  0      
Totretbuf  0          Edisms    0x0        Eparm 0x0      
Elapsed    0x1B0      Ncalls    0x878      Ngiveups 0x0      
Priority_q 4          Ticks_5s  1          Cpu_5sec   0        Cpu_1min 11
Cpu_5min   1          Stacksize 0x2328     Lowstack 0x2328   
Ttyptr     0x3DB27FC  Mem_holding 0x652C     Thrash_count 0
Wakeup_reasons      0x0FFFFFFF  Default_wakeup_reasons 0x0FFFFFFF
Direct_wakeup_major 0x00000000  Direct_wakeup_minor 0x00000000

Preempted processes context:

[Leo Laohoo]

Can you check the below part taken from Crashinfo...Do u have any idea what it is stating ?

Post the crashinfo file.  I am not certain which of the crashinfo file you've opened up.  

I still stand with my initial recommendation that the most stable IOS is still 12.2(55)SE10.  

[Muhammad Kashif Siddiqui]

Dear Leo,

I am attaching crashinfo's of two different switches.

[Leo Laohoo]

mez-sw7-crashinfo.txt is dated 12 March 2015.  The other cannot be determine how recent/old it is.  

[Muhammad Kashif Siddiqui]

Yes, you are right these are around 3 weeks old. For the second one I am attaching the show tech-support which includes the Crashinfo also.

[Leo Laohoo]

System returned to ROM by bus error at PC 0x2AF251C, address 0x0

Crash is caused by an IOS bug.  Only Cisco can decipher what caused this bug.  

[Charles Hill]

Here is a bug that affects 12.2(55)SE5

CSCuc47072

System crash on 802.1x ports

Symptom:
3560 hangs and creates crash dump during dot1x authentication.
As a result the 3560 becomes in accessible via ssh or console.
Only recourse is to power cycle the switch

Conditions:
3560 running 12.2(55)SE5
RADIUS server is ISE 1.1.1

Workaround:
None

[Muhammad Kashif Siddiqui]

Hi Cehill,

Thanks for your response. Can you share the details of fixed IOS version of 3560 for this bug as when I am opening this bug it is not showing any IOS which Cisco built for 3560. I can see the bug is duplicated with another bug.

I am attaching two images for details.

Thanks/Regards

[Charles Hill]

I attached a snapshot of the bug, unfortunately it shows no known fixed release.

I would follow Leos advice of 12.2(55)SE10.