Catalyst C6506-E crash system

Dmitry Stolyarov · ‎10-31-2013

Hi, all!

I used C6506-E as L3 switch

sh mod
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  2   48  CEF720 48 port 1000mb SFP              WS-X6748-SFP       SAL1513A2DC
  3   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL10403HLQ
  6    2  Supervisor Engine 720 (Active)         WS-SUP720-3BXL     SAD090502EW
Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  2  c89c.1dc1.4dd8 to c89c.1dc1.4e07   2.4   12.2(18r)S1  12.2(33)SXI5 Ok
  3  0019.558c.1170 to 0019.558c.119f   2.5   12.2(14r)S5  12.2(33)SXI5 Ok
  6  0011.21a1.882c to 0011.21a1.882f   4.1   8.1(3)       12.2(33)SXI5 Ok
Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  2  Distributed Forwarding Card WS-F6700-DFC3BXL   SAL1540RJP2  5.3    Ok
  3  Centralized Forwarding Card WS-F6700-CFC       SAL1515B8RF  4.1    Ok
  6  Policy Feature Card 3       WS-F6K-PFC3BXL     SAD090104F1  1.4    Ok
  6  MSFC3 Daughterboard         WS-SUP720          SAD090202L7  2.2    Ok
Mod  Online Diag Status
---- -------------------
  2  Pass
  3  Pass
  6  Pass
#sh ver
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(33)SXI5, RELEASE SOFTWARE (fc2)

Today received to log server messages and my switch down:

Oct 31 02:52:44 172.24.0.1 1047148: 18w1d: %SYS-SP-2-MALLOCFAIL: Memory allocation of 12000 bytes failed from 0x417BA04C, alignment 0

Oct 31 02:52:44 172.24.0.1 1047149: Pool: Processor Free: 22024 Cause: Memory fragmentation

Oct 31 02:52:44 172.24.0.1 1047150: Alternate Pool: None Free: 0 Cause: No Alternate pool

Oct 31 02:52:44 172.24.0.1 1047151: -Process= "RFSS server global process", ipl= 0, pid= 86

Oct 31 02:52:44 172.24.0.1 1047152: -Traceback= 408B9030 408BE54C 417BA054 408A3A80 408EB780 408EBB2C 40B46188 40B46690 40B48704 408A3B84 408A3B70

Oct 31 02:52:44 172.24.0.1 1047153: 18w1d: %SYS-SP-2-CFORKMEM: Process creation of RFSS worker process failed (no memory). -Process= "RFSS server global process", ipl= 0, pid= 86

Oct 31 02:52:44 172.24.0.1 1047154: -Traceback= 408EB990 408EBB2C 40B46188 40B46690 40B48704 408A3B84 408A3B70

Oct 31 03:04:14 172.24.0.1 1047605: 18w2d: %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 30 seconds [6/0]

Oct 31 03:04:44 172.24.0.1 1047606: 18w2d: %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 60 seconds [6/0]

Oct 31 03:05:14 172.24.0.1 1047607: 18w2d: %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 90 seconds [6/0]

Oct 31 03:05:52 172.24.0.1 1047608: 18w2d: %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 31 seconds [6/0]

Oct 31 03:06:15 172.24.0.1 1047609: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (11/5),process = SP Error Detection Process.

Oct 31 03:06:15 172.24.0.1 1047610: -Traceback= 408DC5AC 40CCFB98 40CCFCF8 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:15 172.24.0.1 1047611: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (4004)msecs, more than (2000)msecs (15/5),process = SP Error Detection Process.

Oct 31 03:06:15 172.24.0.1 1047612: -Traceback= 408E13AC 408DC7D0 40CCFB98 40CCFCF8 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:15 172.24.0.1 1047613: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (6004)msecs, more than (2000)msecs (20/5),process = SP Error Detection Process.

Oct 31 03:06:15 172.24.0.1 1047614: -Traceback= 408DC5A8 40CCFB98 40CCFCF8 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:15 172.24.0.1 1047615: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (8004)msecs, more than (2000)msecs (25/5),process = SP Error Detection Process.

Oct 31 03:06:15 172.24.0.1 1047616: -Traceback= 408DC6E4 40CCFB98 40CCFCF8 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:15 172.24.0.1 1047623: 18w2d: %SYS-SP-2-MALLOCFAIL: Memory allocation of 576 bytes failed from 0x416A1FB4, alignment 0

Oct 31 03:06:15 172.24.0.1 1047624: Pool: Processor Free: 384 Cause: Not enough free memory

Oct 31 03:06:15 172.24.0.1 1047625: Alternate Pool: None Free: 0 Cause: No Alternate pool

Oct 31 03:06:15 172.24.0.1 1047626: -Process= "Spanning Tree", ipl= 0, pid= 288

Oct 31 03:06:15 172.24.0.1 1047627: -Traceback= 408B9030 408C5C70 416A1FBC 40807764 40803CDC 40803E7C 40803F4C 40801F34 40802338 408030C4 40803244 40804880 407DB6F4 407DC070 407DD0EC 408A3B84

Oct 31 03:06:15 172.24.0.1 1047630: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (11/5),process = SP Error Detection Process.

Oct 31 03:06:15 172.24.0.1 1047631: -Traceback= 408DC56C 40CCFB98 40CCFCF8 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:17 172.24.0.1 1047660: 18w2d: %CPU_MONITOR-SP-2-NOT_RUNNING: CPU_MONITOR messages have not been sent for 30 seconds [SP Error Detection Process 93%/0% (00:00:30.704 99%/0%)] [SP Error Detection Process 00:00:30.268] [LTL MGR 00:00:30.268] [SCP async: LCP#2 00:00:30.268]

Oct 31 03:06:17 172.24.0.1 1047661: 18w2d: %CPU_MONITOR-SP-2-NOT_RUNNING_TB: CPU_MONITOR traceback: 408DC6BC 40CCFB98 40CCFCF8 40C63338 40D2404C 40C6348C 41639FE4 41638424

Oct 31 03:06:17 172.24.0.1 1047662: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (32004)msecs, more than (2000)msecs (58/5),process = SP Error Detection Process.

Oct 31 03:06:17 172.24.0.1 1047663: -Traceback= 408DC7C4 40CCFB98 40CCFCF8 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:17 172.24.0.1 1047664: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (34004)msecs, more than (2000)msecs (61/5),process = SP Error Detection Process.

Oct 31 03:06:17 172.24.0.1 1047665: -Traceback= 408DC714 40CCFB98 40CCFCF8 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:17 172.24.0.1 1047690: 18w2d: %CPU_MONITOR-SP-2-NOT_RUNNING: CPU_MONITOR messages have not been sent for 60 seconds [SP Error Detection Process 93%/0% (00:01:00.704 99%/0%)] [SP Error Detection Process 00:01:00.268] [LTL MGR 00:01:00.268] [SCP async: LCP#2 00:01:00.268]

Oct 31 03:06:17 172.24.0.1 1047691: 18w2d: %CPU_MONITOR-SP-2-NOT_RUNNING_TB: CPU_MONITOR traceback: 408DC630 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424

Oct 31 03:06:17 172.24.0.1 1047692: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (60004)msecs, more than (2000)msecs (104/5),process = SP Error Detection Process.

Oct 31 03:06:17 172.24.0.1 1047693: -Traceback= 408DC770 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:17 172.24.0.1 1047694: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (62004)msecs, more than (2000)msecs (106/5),process = SP Error Detection Process.

Oct 31 03:06:17 172.24.0.1 1047695: -Traceback= 408DC5C8 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:19 172.24.0.1 1047722: 18w2d: %CPU_MONITOR-SP-2-NOT_RUNNING: CPU_MONITOR messages have not been sent for 90 seconds [SP Error Detection Process 93%/0% (00:01:30.704 99%/0%)] [SP Error Detection Process 00:01:30.268] [LTL MGR 00:01:30.268] [SCP async: LCP#2 00:01:30.268]

Oct 31 03:06:19 172.24.0.1 1047723: 18w2d: %CPU_MONITOR-SP-2-NOT_RUNNING_TB: CPU_MONITOR traceback: 408DC4CC 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424

Oct 31 03:06:19 172.24.0.1 1047724: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (90004)msecs, more than (2000)msecs (146/5),process = SP Error Detection Process.

Oct 31 03:06:19 172.24.0.1 1047725: -Traceback= 408DC514 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:19 172.24.0.1 1047726: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (92004)msecs, more than (2000)msecs (149/5),process = SP Error Detection Process.

Oct 31 03:06:19 172.24.0.1 1047727: -Traceback= 408DC56C 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:19 172.24.0.1 1047728: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (94004)msecs, more than (2000)msecs (152/5),process = SP Error Detection Process.

Oct 31 03:06:19 172.24.0.1 1047729: -Traceback= 408DC56C 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:19 172.24.0.1 1047730: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (96004)msecs, more than (2000)msecs (156/5),process = SP Error Detection Process.

Oct 31 03:06:19 172.24.0.1 1047731: -Traceback= 408DC79C 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:19 172.24.0.1 1047732: 18w2d: %SYS-SP-2-MALLOCFAIL: Memory allocation of 872 bytes failed from 0x40CC8384, alignment 0

Oct 31 03:06:19 172.24.0.1 1047733: Pool: Processor Free: 5832 Cause: Memory fragmentation

Oct 31 03:06:19 172.24.0.1 1047734: Alternate Pool: None Free: 0 Cause: No Alternate pool

Oct 31 03:06:19 172.24.0.1 1047735: -Process= "Pool Manager", ipl= 0, pid= 10

Oct 31 03:06:19 172.24.0.1 1047736: -Traceback= 408B9030 408C2974 40CC838C 408E861C 408E8864 408A3B84 408A3B70

Oct 31 03:06:19 172.24.0.1 1047737: 18w2d: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 97 seconds [2/0]

Oct 31 03:06:19 172.24.0.1 1047738: 18w2d: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 97 seconds [3/0]

Oct 31 03:06:19 172.24.0.1 1047739: 18w2d: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 97 seconds [6/1]

Oct 31 03:06:19 172.24.0.1 1047740: 18w2d: SP: IPC: push event list failed. Cannot send MSG: ptr: 0x44C60384, flags: 0x108, retries: 0, seq: 0x0, refcount: 1, rpc_result = 0x0, data_buffer = 0x50C69A00, header = 0x928E0A8, data = 0x928E0C8 || HDR: src: 0x10000, dst: 0x2160007, index: 0, seq: 0, sz: 14, type: 0, flags: 0x4, ext_flags: 0x0, hi: 0x2200ECF9, lo: 0x1 || DATA: 00 00 00 17 00 00 00 00 00 00 00 00 00 61

Oct 31 03:06:19 172.24.0.1 1047741: 18w2d: SP: -Traceback= 4044D3C0 4042EED4 40E8D178 40E8D3F4 40E9F714 40E9F78C 40E9FCF0 413FE59C 413FD7F8 40B8C28C 41215768 41215684 408A3B84 408A3B70

Oct 31 03:06:19 172.24.0.1 1047742: 18w2d: SP: icc_send_request_internal: ipc_send_rpc_blocked failed, result 4

Oct 31 03:06:20 172.24.0.1 1047743: 18w2d: SP: -Traceback= 40E8D234 40E8D3F4 40E9F714 40E9F78C 40E9FCF0 413FE59C 413FD7F8 40B8C28C 41215768 41215684 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047744: 18w2d: SP: icc_req_debug_send_message: icc_send_request failed

Oct 31 03:06:20 172.24.0.1 1047745: 18w2d: SP: icc_req_debug_get_pid_info: failed to get the process stack on the peer after blocking rpc call failed

Oct 31 03:06:20 172.24.0.1 1047746: 18w2d: %RPC-SP-2-FAILED_USERHANDLE: Failed to send RPC request online_diag_sp_request:diag_event_logger_add_entry

Oct 31 03:06:20 172.24.0.1 1047747: -Traceback= 40B8C2A8 41215768 41215684 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047754: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047755: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047756: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047757: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047758: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:06:20 172.24.0.1 1047759: -Traceback= 40E96930 40E96E0C 41050A3C 4104D368 4104CAFC 410431E8 41042E74 41042C40 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047760: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (20/15),process = SP Error Detection Process.

Oct 31 03:06:20 172.24.0.1 1047761: -Traceback= 408DC5A8 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047788: 18w2d: %CPU_MONITOR-SP-2-NOT_RUNNING: CPU_MONITOR messages have not been sent for 30 seconds [SP Error Detection Process 13%/0% (00:00:30.492 99%/0%)] [SP Error Detection Process 00:00:29.600] [L2 MAC oob sync sp task 00:00:29.600] [L2 Aging Task 00:00:29.600]

Oct 31 03:06:20 172.24.0.1 1047789: 18w2d: %CPU_MONITOR-SP-2-NOT_RUNNING_TB: CPU_MONITOR traceback: 408DC668 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424

Oct 31 03:06:20 172.24.0.1 1047790: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (30004)msecs, more than (2000)msecs (72/15),process = SP Error Detection Process.

Oct 31 03:06:20 172.24.0.1 1047791: -Traceback= 408DC770 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047792: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (32004)msecs, more than (2000)msecs (75/15),process = SP Error Detection Process.

Oct 31 03:06:20 172.24.0.1 1047793: -Traceback= 408DC770 40CCFB98 40CCFD30 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047814: 18w2d: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 53 seconds [2/0]

Oct 31 03:06:20 172.24.0.1 1047815: 18w2d: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 53 seconds [3/0]

Oct 31 03:06:20 172.24.0.1 1047816: 18w2d: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 53 seconds [6/1]

Oct 31 03:06:20 172.24.0.1 1047817: 18w2d: %SYS-SP-2-MALLOCFAIL: Memory allocation of 1200 bytes failed from 0x4012A158, alignment 0

Oct 31 03:06:20 172.24.0.1 1047818: Pool: Processor Free: 34672 Cause: Memory fragmentation

Oct 31 03:06:20 172.24.0.1 1047819: Alternate Pool: None Free: 0 Cause: No Alternate pool

Oct 31 03:06:20 172.24.0.1 1047820: -Process= "Spanning Tree", ipl= 0, pid= 288

Oct 31 03:06:20 172.24.0.1 1047821: -Traceback= 408B9030 408C2974 4012A160 4012A6F8 4137821C 41378260 4080D46C 4080D8F4 40805FEC 407DCD08 407DD0E4 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047836: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:06:20 172.24.0.1 1047837: 18w2d: SP: WAVL_INIT for source_tree failed

Oct 31 03:06:20 172.24.0.1 1047838: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:06:20 172.24.0.1 1047885: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047886: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047887: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047888: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047889: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047890: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047891: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047892: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047893: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047894: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047895: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047896: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047897: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047898: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047899: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047900: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047901: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047902: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047903: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047904: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047905: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047906: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047907: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047908: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047909: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047910: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:20 172.24.0.1 1047911: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:06:20 172.24.0.1 1047912: 18w2d: SP: -Traceback= 410CC244 410BDE7C 410BF9EC 410892C4 410928EC 4107B340 41079DF8 408A3B84 408A3B7003:04:43.272 FW[Mod 06]: scp_fpoe_req: memory allocation error, subopcode=10, count=73

Oct 31 03:06:20 172.24.0.1 1047913: 03:04:43.552 FW[Mod 06]: scp_fpoe_req: memory allocation error, subopcode=10, count=135

Oct 31 03:06:20 172.24.0.1 1047914: 03:04:43.684 FW[Mod 06]: scp_fpoe_req: memory allocation error, subopcode=10, count=141

Oct 31 03:06:21 172.24.0.1 1047915: 03:04:43.832 FW[Mod 06]: scp_fpoe_req: memory allocation error, subopcode=10, count=152

Oct 31 03:06:22 172.24.0.1 1047916: 18w2d: %ERR_DET-SP-5-ERR_DET_LOW_MEM: Very low memory, dump debuginfo, local 30724, io 46223920

Oct 31 03:06:35 172.24.0.1 1047917: 18w2d: SP: MLS-MSC ASSERTION FAILED "mscdb_scref_oif_insert(sc, &mac, vlan)" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10368

Oct 31 03:06:35 172.24.0.1 1047918: 18w2d: SP: -Traceback= 410CC340 410BB8FC 410BBD78 41087648 410927F4 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:35 172.24.0.1 1047919: 18w2d: SP: MLS-MSC ASSERTION FAILED "mscdb_scref_oif_insert(sc, &mac, vlan)" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10368

Oct 31 03:06:35 172.24.0.1 1047920: 18w2d: SP: -Traceback= 410CC340 410BB8FC 410BBD78 41087648 410927F4 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:35 172.24.0.1 1047921: 18w2d: SP: MLS-MSC ASSERTION FAILED "mscdb_scref_rpf_insert(sc, &mac, vlan)" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10357

Oct 31 03:06:35 172.24.0.1 1047922: 18w2d: SP: -Traceback= 410CC340 410BBEB8 41087648 410927F4 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:06:38 172.24.0.1 1047923: 18w2d: %SCHED-SP-3-THRASHING: Process thrashing on watched boolean 'STP-Flush-Running'. -Process= "SpanTree Flush", ipl= 5, pid= 120

Oct 31 03:06:39 172.24.0.1 1047924: -Traceback= 408ECF64 408ED040 40D9C044 40EE131C 40EE7250 40EE92F4 40EF80A0 40EE3530 4133A0B4 416A1C74 408A3B84 408A3B70

Oct 31 03:06:39 172.24.0.1 1047927: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:06:39 172.24.0.1 1047928: 18w2d: %SCHED-SP-3-STUCKMTMR: Sleep with expired managed timer 5295AF60, time 0x29336A860 (00:00:00 ago). -Process= "L2 PI_E consistency checker", ipl= 5, pid= 256

Oct 31 03:06:39 172.24.0.1 1047929: -Traceback= 408ECBD8 408ED040 40D9C044 40EE131C 40EE9518 40EE9830 408A3B84 408A3B70

Oct 31 03:06:39 172.24.0.1 1047930: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:06:40 172.24.0.1 1047931: -Traceback= 40D9BC90 40D9C2B0 40EE131C 40EE9518 40EE9830 408A3B84 408A3B70

Oct 31 03:06:40 172.24.0.1 1047932: 18w2d: SP: Dumping srcb info sman_setup_search

Oct 31 03:06:40 172.24.0.1 1047933: 18w2d: SP: se_flags: 50460100

Oct 31 03:06:40 172.24.0.1 1047934: 18w2d: SP: se_data: 0x00040000000000000000014000000000

Oct 31 03:06:40 172.24.0.1 1047935: 18w2d: SP: se_mask: 0x0004000000000000000007C000210000

Oct 31 03:06:40 172.24.0.1 1047936: 18w2d: SP: sew_data: 0x00040000000000000000000000013000

Oct 31 03:06:40 172.24.0.1 1047937: 18w2d: SP: sew_mask: 0x00040000000000000000000000013000

Oct 31 03:06:42 172.24.0.1 1047938: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:06:42 172.24.0.1 1047939: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:06:42 172.24.0.1 1047940: 18w2d: SP: Total SMAN interrupt=36547337, Total SE failures since last good one 1

Oct 31 03:06:42 172.24.0.1 1047941: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:06:43 172.24.0.1 1047942: -Traceback= 40D9BC90 40D9C2B0 40EE131C 40EE9518 40EE9830 408A3B84 408A3B70

Oct 31 03:11:46 172.24.0.1 1048227: Pool: Processor Free: 88304 Cause: Memory fragmentation

Oct 31 03:11:46 172.24.0.1 1048228: Alternate Pool: None Free: 0 Cause: No Alternate pool

Oct 31 03:11:46 172.24.0.1 1048229: -Process= "IP ACL XDR LC Background", ipl= 0, pid= 341

Oct 31 03:11:46 172.24.0.1 1048230: -Traceback= 408B9030 408C2974 40EB06B8 40EB1028 40EB1604 408A3B84 408A3B70

Oct 31 03:18:28 172.24.0.1 1052062: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:18:28 172.24.0.1 1052063: 18w2d: SP: -Traceback= 410CC244 410BBB6C 410BBD78 41087648 410927F4 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:18:28 172.24.0.1 1052064: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:18:28 172.24.0.1 1052065: 18w2d: SP: -Traceback= 410CC244 410BBEB8 41087648 410927F4 4107B340 41079DF8 408A3B84 408A3B7003:16:55.351 FW[Mod 06]: scp_fpoe_req: memory allocation error, subopcode=10, count=28

Oct 31 03:18:28 172.24.0.1 1052066: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:18:28 172.24.0.1 1052067: 18w2d: SP: -Traceback= 410CC244 410BBB6C 410BBD78 41087648 410927F4 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:18:28 172.24.0.1 1052068: 18w2d: SP: MLS-MSC ASSERTION FAILED "gce" in msc_sc2vdb_link() at ../const/native-sp/mls_msc_utils.c:10340

Oct 31 03:18:28 172.24.0.1 1052069: 18w2d: SP: -Traceback= 410CC244 410BBEB8 41087648 410927F4 4107B340 41079DF8 408A3B84 408A3B70

Oct 31 03:18:34 172.24.0.1 1052181: -Traceback= 40E96930 40E96E0C 41050A3C 4104D368 4104CAFC 410431E8 41042E74 41042C40 408A3B84 408A3B70

Oct 31 03:18:34 172.24.0.1 1052182: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:19:06 172.24.0.105 1827: 3w3d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3904 from FULL to DOWN, Neighbor Down: Dead timer expired

Oct 31 03:19:08 172.24.0.116 1075: 17w5d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3912 from FULL to DOWN, Neighbor Down: Dead timer expired

Oct 31 03:19:09 172.24.0.115 567: 3w5d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3907 from FULL to DOWN, Neighbor Down: Dead timer expired

Oct 31 03:19:09 172.24.0.26 23: Oct 30 23:17:36.552: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on GigabitEthernet0/0.3914 from FULL to DOWN, Neighbor Down: Dead timer expired

Oct 31 03:19:12 172.24.0.113 2028: 12w4d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3902 from FULL to DOWN, Neighbor Down: Dead timer expired

Oct 31 03:19:12 172.24.0.108 1757: 9w3d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3905 from FULL to DOWN, Neighbor Down: Dead timer expired

Oct 31 03:19:13 172.24.0.118 398: 28w0d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3918 from FULL to DOWN, Neighbor Down: Dead timer expired

Oct 31 03:19:14 172.24.0.114 1717: 10w6d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3909 from FULL to DOWN, Neighbor Down: Dead timer expired

Oct 31 03:19:14 172.24.0.110 21855: 32w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3908 from FULL to DOWN, Neighbor Down: Dead timer expired

Oct 31 03:20:01 172.24.0.115 568: 3w5d: %PIM-5-NBRCHG: neighbor 10.1.0.25 DOWN on interface Vlan3907 DR

Oct 31 03:20:01 172.24.0.115 569: 3w5d: %PIM-5-DRCHG: DR change from neighbor 10.1.0.25 to 10.1.0.26 on interface Vlan3907

Oct 31 03:20:04 172.24.0.105 1828: 3w3d: %PIM-5-NBRCHG: neighbor 10.1.0.13 DOWN on interface Vlan3904 DR

Oct 31 03:20:04 172.24.0.105 1829: 3w3d: %PIM-5-DRCHG: DR change from neighbor 10.1.0.13 to 10.1.0.14 on interface Vlan3904

Oct 31 03:20:06 172.24.0.1 1052183: -Traceback= 40E96930 40E96E0C 41050A3C 4104D368 4104CAFC 410431E8 41042E74 41042C40 408A3B84 408A3B70

Oct 31 03:20:06 172.24.0.1 1052184: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:20:06 172.24.0.1 1052185: -Traceback= 40E96930 40E96E0C 41050A3C 4104D368 4104CAFC 410431E8 41042E74 41042C40 408A3B84 408A3B70

Oct 31 03:20:06 172.24.0.1 1052186: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:20:06 172.24.0.1 1052187: -Traceback= 40E96930 40E96E0C 41050A3C 4104D368 4104CAFC 410431E8 41042E74 41042C40 408A3B84 408A3B70

Oct 31 03:20:06 172.24.0.1 1052188: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:20:06 172.24.0.1 1052189: -Traceback= 40E96930 40E96E0C 41050A3C 4104D368 4104CAFC 410431E8 41042E74 41042C40 408A3B84 408A3B70

Oct 31 03:20:06 172.24.0.1 1052190: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:20:07 172.24.0.115 570: 3w5d: %PIM-5-NBRCHG: neighbor 10.1.0.25 UP on interface Vlan3907

Oct 31 03:20:07 172.24.0.115 571: 3w5d: %PIM-5-DRCHG: DR change from neighbor 10.1.0.26 to 10.1.0.25 on interface Vlan3907

Oct 31 03:20:07 172.24.0.105 1830: 3w3d: %PIM-5-NBRCHG: neighbor 10.1.0.13 UP on interface Vlan3904

Oct 31 03:20:07 172.24.0.105 1831: 3w3d: %PIM-5-DRCHG: DR change from neighbor 10.1.0.14 to 10.1.0.13 on interface Vlan3904

Oct 31 03:20:08 172.24.0.1 1052191: -Traceback= 40E96930 40E96E0C 41050A3C 4104D368 4104CAFC 410431E8 41042E74 41042C40 408A3B84 408A3B70

Oct 31 03:20:08 172.24.0.1 1052192: 18w2d: %SCHED-SP-3-STUCKMTMR: Sleep with expired managed timer 5295AF60, time 0x293418D70 (00:00:00 ago). -Process= "L2 PI_E consistency checker", ipl= 5, pid= 256

Oct 31 03:20:08 172.24.0.1 1052193: -Traceback= 408ECBD8 408ED040 40D9C044 40EE131C 40EE9518 40EE9830 408A3B84 408A3B70

Oct 31 03:20:08 172.24.0.1 1052194: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:08 172.24.0.1 1052195: -Traceback= 40D9BC90 40D9C2B0 40EE131C 40EE9518 40EE9830 408A3B84 408A3B70

Oct 31 03:20:08 172.24.0.1 1052196: 18w2d: SP: Dumping srcb info sman_setup_search

Oct 31 03:20:08 172.24.0.1 1052197: 18w2d: SP: se_flags: 50460100

Oct 31 03:20:08 172.24.0.1 1052198: 18w2d: SP: se_data: 0x00040000000000000000014000000000

Oct 31 03:20:08 172.24.0.1 1052199: 18w2d: SP: se_mask: 0x0004000000000000000007C000210000

Oct 31 03:20:08 172.24.0.1 1052200: 18w2d: SP: sew_data: 0x00040000000000000000000000013000

Oct 31 03:20:08 172.24.0.1 1052201: 18w2d: SP: sew_mask: 0x00040000000000000000000000013000

Oct 31 03:20:08 172.24.0.1 1052202: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:20:08 172.24.0.1 1052203: -Traceback= 40E96930 40E96E0C 41050A3C 4104D368 4104CAFC 410431E8 41042E74 41042C40 408A3B84 408A3B70

Oct 31 03:20:08 172.24.0.1 1052204: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:20:15 172.24.0.1 1052416: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:20:17 172.24.0.26 24: Oct 30 23:18:44.459: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on GigabitEthernet0/0.3914 from LOADING to FULL, Loading Done

Oct 31 03:20:17 172.24.0.118 399: 28w0d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3918 from LOADING to FULL, Loading Done

Oct 31 03:20:17 172.24.0.110 21856: 32w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3908 from LOADING to FULL, Loading Done

Oct 31 03:20:17 172.24.0.105 1832: 3w3d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3904 from LOADING to FULL, Loading Done

Oct 31 03:20:17 172.24.0.115 572: 3w5d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3907 from LOADING to FULL, Loading Done

Oct 31 03:20:17 172.24.0.108 1758: 9w3d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3905 from LOADING to FULL, Loading Done

Oct 31 03:20:18 172.24.0.114 1718: 10w6d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3909 from LOADING to FULL, Loading Done

Oct 31 03:20:18 172.24.0.113 2029: 12w4d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3902 from LOADING to FULL, Loading Done

Oct 31 03:20:18 172.24.0.116 1076: 17w5d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.8 on Vlan3912 from LOADING to FULL, Loading Done

Oct 31 03:20:18 172.24.0.1 1052417: -Traceback= 40E96930 40E96E0C 41050A3C 4104D368 4104CAFC 410431E8 41042E74 41042C40 408A3B84 408A3B70

Oct 31 03:20:18 172.24.0.1 1052418: 18w2d: %ICC-SP-2-NOMEM: No memory available for grp_copy

Oct 31 03:20:29 172.24.0.1 1052453: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:29 172.24.0.1 1052454: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:29 172.24.0.1 1052455: 18w2d: %C6K_PROCMIB-SP-3-IPC_TRANSMIT_FAIL: Failed to send process statistics update : error code = no memory available

Oct 31 03:20:29 172.24.0.1 1052456: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:29 172.24.0.1 1052457: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.1.0.54 on Vlan3914 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052458: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.1.0.70 on Vlan3918 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052459: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.10 on Vlan3904 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052460: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.21 on Vlan3902 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052461: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.15 on Vlan3908 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052462: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.23 on Vlan3907 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052463: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.13 on Vlan3905 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052464: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.22 on Vlan3909 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052465: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.1.0.46 on Vlan3912 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052466: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:29 172.24.0.1 1052467: 18w2d: %L2_API-SP-5-SENDFAIL: ICC Send failed. Reason EARL_NO_RESOURCE

Oct 31 03:20:29 172.24.0.1 1052468: 18w2d: %SYS-SP-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x408E6CC0, alignment 0

Oct 31 03:20:29 172.24.0.1 1052470: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:29 172.24.0.1 1052471: 18w2d: %RPC-2-FAILED_USERHANDLE: Failed to send RPC request sea_console_rp_cli_request:sea_console_send_rp_buffer_rpc

Oct 31 03:20:29 172.24.0.1 1052472: -Traceback= 423B28B8 4298B63C 416690F0 416690DC

Oct 31 03:20:29 172.24.0.1 1052473: 18w2d: %RPC-2-RETRY: Recovered from RPC send failure for request c6k_power_sp:c6k_pwr_get_fru_present. Resending request.

Oct 31 03:20:29 172.24.0.1 1052474: -Traceback= 4200D31C 4212C9C4 41AB5CE4 41AB57D8 414CC3BC 414A21E8 414932BC 414BC94C 416690F0 416690DC

Oct 31 03:20:29 172.24.0.1 1052475: 18w2d: %XDR-SP-3-XDRNOMEM: XDR failed to allocate memory during xdr window message creation.

Oct 31 03:20:29 172.24.0.1 1052476: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.19 on Vlan10 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052477: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.1.2 on Vlan10 from LOADING to FULL, Loading Done

Oct 31 03:20:29 172.24.0.1 1052478: 18w2d: %SCHED-SP-3-STUCKMTMR: Sleep with expired managed timer 5295AF60, time 0x29341A4E0 (00:01:43 ago). -Process= "L2 PI_E consistency checker", ipl= 5, pid= 256

Oct 31 03:20:29 172.24.0.1 1052479: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:29 172.24.0.1 1052480: 18w2d: %XDR-SP-3-XDRNOMEM: XDR failed to allocate memory during xdr window message creation.

Oct 31 03:20:29 172.24.0.1 1052481: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:29 172.24.0.1 1052482: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:30 172.24.0.1 1052483: 18w2d: %SYS-3-LOGGER_DROPPED: System dropped 2262 console debug messages.

Oct 31 03:20:31 172.24.0.1 1052484: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:31 172.24.0.1 1052485: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:31 172.24.0.1 1052486: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:31 172.24.0.1 1052487: 18w2d: SP: Total SMAN interrupt=36556280, Total SE failures since last good one 3

Oct 31 03:20:31 172.24.0.1 1052488: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:32 172.24.0.1 1052489: -Traceback= 40D9BC90 40D9C2B0 40EE131C 40EE9518 40EE9830 408A3B84 408A3B70

Oct 31 03:20:33 172.24.0.1 1052490: 18w2d: SP: Dumping srcb info sman_setup_search

Oct 31 03:20:33 172.24.0.1 1052491: 18w2d: SP: se_flags: 50460100

Oct 31 03:20:33 172.24.0.1 1052492: 18w2d: SP: se_data: 0x00040000000000000000014000000000

Oct 31 03:20:33 172.24.0.1 1052493: 18w2d: SP: se_mask: 0x0004000000000000000007C000210000

Oct 31 03:20:33 172.24.0.1 1052494: 18w2d: SP: sew_data: 0x00040000000000000000000000013000

Oct 31 03:20:33 172.24.0.1 1052495: 18w2d: SP: sew_mask: 0x00040000000000000000000000013000

Oct 31 03:20:33 172.24.0.1 1052496: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:33 172.24.0.1 1052497: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:34 172.24.0.1 1052498: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:34 172.24.0.1 1052499: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:34 172.24.0.1 1052500: 18w2d: SP: Total SMAN interrupt=36556280, Total SE failures since last good one 4

Oct 31 03:20:34 172.24.0.1 1052501: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:36 172.24.0.1 1052502: -Traceback= 40D9BC90 40D9C2B0 40EE131C 40EE9518 40EE9830 408A3B84 408A3B70

Oct 31 03:20:36 172.24.0.1 1052503: 18w2d: SP: Dumping srcb info sman_setup_search

Oct 31 03:20:36 172.24.0.1 1052504: 18w2d: SP: se_flags: 50460100

Oct 31 03:20:36 172.24.0.1 1052505: 18w2d: SP: se_data: 0x00040000000000000000014000000000

Oct 31 03:20:36 172.24.0.1 1052506: 18w2d: SP: se_mask: 0x0004000000000000000007C000210000

Oct 31 03:20:36 172.24.0.1 1052507: 18w2d: SP: sew_data: 0x00040000000000000000000000013000

Oct 31 03:20:36 172.24.0.1 1052508: 18w2d: SP: sew_mask: 0x00040000000000000000000000013000

Oct 31 03:20:36 172.24.0.1 1052509: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:36 172.24.0.1 1052510: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:36 172.24.0.1 1052511: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:36 172.24.0.1 1052512: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:36 172.24.0.1 1052513: 18w2d: SP: WAVL_INIT for stats_wavl_tree failed

Oct 31 03:20:36 172.24.0.1 1052514: 18w2d: %XDR-SP-3-XDRNOMEM: XDR failed to allocate memory during xdr window message creation.

Oct 31 03:20:36 172.24.0.1 1052515: -Traceback= 4094C50C 4093DC9C 4093AB64 4094B450 4094B488 40949B34 4094A848 408A3B84 408A3B70

Oct 31 03:20:36 172.24.0.1 1052516: 18w2d: %COMMON_FIB-SP-3-NOMEM: Memory allocation failure for fib defnet source sb in Common CEF [0x40303D80] (fatal) (0 subsequent failures).

Oct 31 03:20:36 172.24.0.1 1052517: 18w2d: %COMMON_FIB-SP-4-DISABLING: Common CEF is being disabled due to a fatal error.

Oct 31 03:20:37 172.24.0.1 1052518: 18w2d: %OSPF-5-ADJCHG: Process 1, Nbr 172.24.1.1 on Vlan3915 from LOADING to FULL, Loading Done

Oct 31 03:20:37 172.24.0.1 1052519: 18w2d: %FIB-2-FIBDISABLE: Fatal error, slot 6/0 (6): CEF-Common: no memory

Oct 31 03:20:37 172.24.0.1 1052520: 18w2d: %XDR-SP-6-XDRLCDISABLEREQUEST: Client CEF push requested to be disabled.

Oct 31 03:20:37 172.24.0.1 1052521: -Traceback= 4094B1DC 40354C54 403578DC 40309860 4030F8C0 40307A24 40307BE8 408A3B84 408A3B70

Oct 31 03:20:37 172.24.0.1 1052522: 18w2d: SP: cfib_enqueue_object_delete: no memory for cfib_fibmsg_chunk

Oct 31 03:20:37 172.24.0.1 1052523: 18w2d: SP: cfib_enqueue_object_delete: no memory for cfib_fibmsg_chunk

Oct 31 03:20:37 172.24.0.1 1052524: 18w2d: SP: Total SMAN interrupt=36556280, Total SE failures since last good one 5

Oct 31 03:20:37 172.24.0.1 1052525: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:38 172.24.0.1 1052526: -Traceback= 40D9BC90 40D9C2B0 40EE131C 40EE9518 40EE9830 408A3B84 408A3B70

Oct 31 03:20:38 172.24.0.1 1052527: 18w2d: SP: Dumping srcb info sman_setup_search

Oct 31 03:20:38 172.24.0.1 1052528: 18w2d: SP: se_flags: 50460100

Oct 31 03:20:38 172.24.0.1 1052529: 18w2d: SP: se_data: 0x00040000000000000000014000000000

Oct 31 03:20:38 172.24.0.1 1052530: 18w2d: SP: se_mask: 0x0004000000000000000007C000210000

Oct 31 03:20:38 172.24.0.1 1052531: 18w2d: SP: sew_data: 0x00040000000000000000000000013000

Oct 31 03:20:38 172.24.0.1 1052532: 18w2d: SP: sew_mask: 0x00040000000000000000000000013000

Oct 31 03:20:39 172.24.0.1 1052538: 18w2d: SP: Total SMAN interrupt=36556280, Total SE failures since last good one 6

Oct 31 03:20:39 172.24.0.1 1052539: 18w2d: %EARL_L2_ASIC-SP-4-SRCH_ENG_FAIL: EARL L2 ASIC Search Engine has failed

Oct 31 03:20:40 172.24.0.1 1052540: -Traceback= 40D9BC90 40D9C2B0 40EE131C 40EE9518 40EE9830 408A3B84 408A3B70

Oct 31 03:20:40 172.24.0.1 1052541: 18w2d: SP: Dumping srcb info sman_setup_search

Oct 31 03:21:15 172.24.0.1 1052708: 18w2d: SP: Total SMAN interrupt=36556414, Total SE failures since last good one 19

Oct 31 03:22:40 172.24.0.1 1052946: 18w2d: Frames of RPC c6k_sp_environmental process (pid 400) on 6 (proc|slot) after blocking rpc call failed: 408ED018 413FDD24 408A3B84 408A3B70 IGMЁ

Oct 31 03:22:41 172.24.0.1 1052947: 18w2d: %RPC-2-RETRY: Recovered from RPC send failure for request c6k_sp_environmental:env_get_sensor_value_sp. Resending request.

Oct 31 03:22:41 172.24.0.1 1052948: -Traceback= 42015E10 4233F398 4233FC84 42340F84 4233E9A0 416690F0 416690DC

Oct 31 03:22:54 172.24.0.1 1052989: 18w2d: icc_send_request_internal: ipc_send_rpc_blocked failed, result 6

Oct 31 03:22:54 172.24.0.1 1052990: 18w2d: -Traceback= 425EAC08 425EADC8 425EB1E8 4296ABB8 4200D2F0 4212C9C4 41AB5CE4 41AB57D8 414CC3BC 414A21E8 414932BC 414BC94C 416690F0 416690DC

Oct 31 03:22:54 172.24.0.1 1052991: 18w2d: Frames of RPC sea_console_rp_cli_request process (pid 455) on 6 (proc|slot) after blocking rpc call failed: 408ED018 413FDD24 408A3B84 408A3B70 IGMаE г

Oct 31 03:22:55 172.24.0.1 1052992: 18w2d: %RPC-2-FAILED_USERHANDLE: Failed to send RPC request sea_console_rp_cli_request:sea_console_send_rp_buffer_rpc

Oct 31 03:22:55 172.24.0.1 1052993: -Traceback= 423B28B8 4298B63C 416690F0 416690DC

Help me deal with the problem.

Thanks in advance!

maxrodri · ‎10-31-2013

Hello Dmitry,

There are bunches or error on this log, however, there are few of them that might explain the issue:

Oct 31 03:06:15 172.24.0.1 1047630: 18w2d: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (11/5),process = SP Error Detection Process.

Oct 31 03:06:15 172.24.0.1 1047631: -Traceback= 408DC56C 40CCFB98 40CCFCF8 40C63338 40D2404C 40C6348C 41639FE4 41638424 408A3B84 408A3B70

What Causes %SYS-3-CPUHOG Messages?

http://www.cisco.com/en/US/products/hw/iad/ps397/products_tech_note09186a00800a6ac4.shtml

And

Oct 31 03:06:19 172.24.0.1 1047732: 18w2d: %SYS-SP-2-MALLOCFAIL: Memory allocation of 872 bytes failed from 0x40CC8384, alignment 0

Oct 31 03:06:19 172.24.0.1 1047733: Pool: Processor Free: 5832 Cause: Memory fragmentation

Oct 31 03:06:19 172.24.0.1 1047734: Alternate Pool: None Free: 0 Cause: No Alternate pool

Oct 31 03:06:19 172.24.0.1 1047735: -Process= "Pool Manager", ipl= 0, pid= 10

Oct 31 03:06:19 172.24.0.1 1047736: -Traceback= 408B9030 408C2974 40CC838C 408E861C 408E8864 408A3B84 408A3B70

What is Memory Allocation Failure?

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml#topic1

The main issue seems be related to the Malloc Failure present as this is the first error message present on the log. To make sure of that and what other process are involved on the issue, a TAC case can be raised to decode these tracebacks and narrow down the error to an specific process.

Leo Laohoo · ‎10-31-2013

I'm seeing a lot of CPU Hog errors. You may have to consider upgrading your IOS.

Dmitry Stolyarov · ‎11-01-2013

This IOS running at another catalyst C6506-E and works well over a year

Leo Laohoo · ‎11-01-2013

This IOS running at another catalyst C6506-E and works well over a year

As you wish. CPU Hogs, in some versions, don't just jump out when you power up the chassis. One of the main culprit of CPU Hogs is TIME. CPU Hogs happen when un-used processes are not "returned" to the pool properly. This can be small amount but compound this over time and you'll get CPU Hog.

You have but three choices:

1. Create a Cisco TAC;

2. Reboot your chassis; or

3. Upgrade your IOS and reboot your chassis.

Chances are Cisco TAC will give you options 2 and 3.