WLC SRE module constantly crashes

[the-lebowski]

Hello

We are having an ongoing issue with WLC 7.2.111.3 running on an SRE module inside a 2911.  The WLC is rebooting a couple times a day, we have RMA'd the 2911 and the SRE module and the problem still remains.  We have two SSIDs, one for employees doing radius auth and a guest network doing web-auth.  We are using a single LLWAP and nothing out of the ordinary.

Cisco TAC has been helpful but even they can't seem to figure out what the problem is. The logs give little information as they are cleared when the WLC reboots and if I am not consoled into it logging the session I don't see any message of it going down.

This what I see on the console when it reboots, has anyone had problems or see this before? 

.............

(Cisco Controller) >

(Cisco Controller) >

(Cisco Controller) >Oops: 0000 [#1]

PREEMPT

Modules linked in: sshquicksec(P) wlc_dtl(F) wlc_fp(P) wlc_af(P) cisco_quack(P)

CPU:    0

EIP:    0060:[<c0104c9d>]    Tainted: PF      VLI

EFLAGS: 00010087   (2.6.21_mvlcge500-octeon-mips64_octeon_v2_be #1)

EIP is at dump_trace+0x5a/0x93

eax: 00000000   ebx: 6e692f6e   ecx: 6e692f6e   edx: c034ef56

esi: 3a786165   edi: 6e692000   ebp: 6e692ffd   esp: dfdb1058

ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068  preempt:00000003

Process softirq-net-rx/ (pid: 6, ti=dfdb0000 task=dfec3830 task.ti=dfdb1000)

Stack: 0000007a c034ea4d dfec3830 dfdb1000 dfdb1ffd c0104cf1 c03a6910 c034ea4d

       00000001 c0119a0c c0105371 c034ea4d c01053f3 dfdb1098 c0136670 c034e1ca

       c035664c 00000053 c030b89c 00000001 c0119a0c c0119a0c 00000000 00000000

Call Trace:

[<c0104cf1>] show_trace_log_lvl+0x1b/0x36

[<c0119a0c>] vprintk+0x11/0x2a1

[<c0105371>] show_trace+0xf/0x11

[<c01053f3>] dump_stack+0x12/0x14

[<c0136670>] add_preempt_count+0x41/0xbe

[<c0119a0c>] vprintk+0x11/0x2a1

[<c0119a0c>] vprintk+0x11/0x2a1

[<c01bff1f>] vsnprintf+0x43a/0x476

[<c010655b>] do_IRQ+0x7a/0xe7

[<c0119a0c>] vprintk+0x11/0x2a1

[<c0119cb7>] printk+0x1b/0x1f

[<c013666b>] add_preempt_count+0x3c/0xbe

[<c0104847>] common_interrupt+0x23/0x28

[<c0104847>] common_interrupt+0x23/0x28

..................

BUG: unable to handle kernel paging request at virtual address 6e692034

printing eip:

c0104cac

*pde = 00000000

Oops: 0000 [#2]

PREEMPT

Modules linked in: sshquicksec(P) wlc_dtl(F) wlc_fp(P) wlc_af(P) cisco_quack(P)

CPU:    0

EIP:    0060:[<c0104cac>]    Tainted: PF      VLI

EFLAGS: 00010046   (2.6.21_mvlcge500-octeon-mips64_octeon_v2_be #1)

EIP is at dump_trace+0x69/0x93

eax: 00000000   ebx: 6e692ffd   ecx: c03a6910   edx: c034ef56

esi: 50f8458d   edi: 6e692000   ebp: 6e692ffd   esp: dfdb0ef0

ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068  preempt:00000003

Process softirq-net-rx/ (pid: 6, ti=dfdb0000 task=dfec3830 task.ti=dfdb1000)

Stack: c01c002d c034f0ea 00000000 00000000 c034f0ea c0104cf1 c03a6910 c034f0ea

       dfdb10bb 00000018 c0104dac c034f0ea c034f0ea dfdb1020 dfdb1058 00000002

       c04298c8 c0104c9d dfdb1020 c0104f9d c034f0ea 00000010 dfec39d8 00000006

Call Trace:

[<c01c002d>] snprintf+0x1f/0x22

[<c0104cf1>] show_trace_log_lvl+0x1b/0x36

[<c0104dac>] show_stack_log_lvl+0xa0/0xad

[<c0104c9d>] dump_trace+0x5a/0x93

[<c0104f9d>] show_registers+0x1e4/0x2e0

[<c01049b7>] kernel_thread_helper+0x7/0x10

[<c01051af>] die+0x116/0x1ed

[<c0113c7b>] do_page_fault+0x4a7/0x578

=======================

---------------------------

| preempt count: c01137d4 ]

| 24-level deep critical section nesting:

----------------------------------------

.. [<c0115d25>] .... task_rq_lock+0x12/0x19

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<c021c365>] .... iegbe_xmit_frame+0x298/0x948

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<c01050db>] .... die+0x42/0x1ed

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<c01196cf>] .... release_console_sem+0x51/0x1cf

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<c0119a7b>] .... vprintk+0x80/0x2a1

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

.. [<00000000>] .... _stext+0x3feff000/0x14

.....[<00000000>] ..   ( <= _stext+0x3feff000/0x14)

Code: 00 00 eb 10 8b 4c 24 18 89 f2 8b 44 24 1c ff 51 08 83 c3 04 39 fb 76 23 39 eb 73 1f 8b 33 89 f0 e8 96 33 02 00 85 c0 74 e8 eb d9 <8b> 4f 34 85 c9 74 1d 89 cb e8 18 ad 00 00 eb b9 8b 4c 24 18 ba

EIP: [<c0104cac>] dump_trace+0x69/0x93 SS:ESP 0068:dfdb0ef0

note: softirq-net-rx/[6] exited with preempt_count 2

Clocksource tsc unstable (delta = 28120727222 ns)

Time: acpi_pm clocksource has been installed.

...................

At this point looking to see if anyone has insight or thoughts on what could be causing this.

TIA

[Stephen Rodriguez]

There actually is a known issue with the SRE and 7.2 code.  one of my customers was having the same issue.

CSCua08891, the fix is in 7.3

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered