ME-3400 crashes when unplugging fiber link.

[mark_tegg]

Hi

As the subject suggests I have an issue with a ME-3400 that crashed when the fibre patch was pulled.

This is the reduced output from #show cpu proc

198       78753       213     369732 46.38%   37.30%   13.32%    0 IGMPSN L2MCM
213       21813       468      46608  11.23%     9.90%    3.69%    0 REP BPA/EPA Proc

Amongst other things this was pulled from #show log

Dec 15 11:01:38 WST: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for REP Cam Flush . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= D9BAA8 D9C1F4 1738F20 A6E77C A6524C
Dec 15 11:01:46 WST: %SYS-2-MALLOCFAIL: Memory allocation of 4120 bytes failed from 0x7ADECC, alignment 0
Pool: Processor  Free: 658564  Cause: Memory fragmentation
Alternate Pool: None  Free: 0  Cause: No Alternate pool
-Process= "REP Topology change", ipl= 0, pid= 157
-Traceback= D9BAA8 D9C1F4ex 17398D4 173BDF4 173C058 1A40F40 7ADED0 908B18 908BCC A6E77C A6524C
Dec 15 11:01:49 WST: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for REP Cam Flush . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= D9BAA8 D9C1F4 1738F20 A6E77C A6524C
Dec 15 11:01:59 WST: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for REP Cam Flush . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= D9BAA8 D9C1F4 1738F20 A6E77C A6524Cccl 0.00

Dec 15 11:02:09 WST: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for REP Cam Flush . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= D9BAA8 D9C1F4 1738F20 A6E77C A6524C
Dec 15 11:02:16 WST: %SYS-2-MALLOCFAIL: Memory allocation of 260 bytes failed from 0x1B8010, alignment 0
Pool: I/O  Free: 2312  Cause: Memory fragmentation
Alternate Pool: None  Free: 0  Cause: No Alternate pool
-Process= "IGMPSN L2MCM", ipl= 0, pid= 198
-Traceback= D9BAA8 D9C1F4 17398D4 173BDF4 173BF68 1B8014 1B8464 1B8BAC 1B7184 7AAF1C 7AD220 7D0474 7D8274 7CB0F8 7AEF18 7AFBCC
Dec 15 11:02:19 WST: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for REP Cam Flush . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= D9BAA8 D9C1F4 1738F20 A6E77C A6524C
Dec 15 11:02:23 WST: %HSRP-5-STATECHANGE: Vlan762 Grp 1 state Standby -> Active
Dec 15 11:02:23 WST: %HSRP-5-STATECHANGE: Vlan762 Grp 1 state Active -> Speak
Dec 15 11:02:24 WST: %HSRP-5-STATECHANGE: Vlan762 Grp 1 state Speak -> Standby
Dec 15 11:02:25 WST: %HSRP-5-STATECHANGE: Vlan102 Grp 1 state Standby -> Active
Dec 15 11:02:25 WST: %HSRP-5-STATECHANGE: Vlan102 Grp 1 state Active -> Speak
Dec 15 11:02:26 WST: %HSRP-5-STATECHANGE: Vlan102 Grp 1 state Speak -> Standby
Dec 15 11:02:29 WST: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for REP Cam Flush . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= D9BAA8 D9C1F4 1738F20 A6E77C A6524C
Dec 15 11:02:33 WST: %COMMON_FIB-3-HW_API: HW API failure for IPv4 CEF [0x009A8070]: Platform IPv4 Fib malloc failed (fatal) (1 subsequent failure).
Dec 15 11:02:33 WST: %COMMON_FIB-4-DISABLING: IPv4 CEF is being disabled due to a fatal error.
Dec 15 11:02:40 WST: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for REP Cam Flush . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= D9BAA8 D9C1F4 1738F20 A6E77C A6524C
Dec 15 11:02:46 WST: %SYS-2-MALLOCFAIL: Memory allocation of 260 bytes failed from 0x1B8010, alignment 0
Pool: I/O  Free: 2312  Cause: Memory fragmentation
Alternate Pool: None  Free: 0  Cause: No Alternate pool
-Process= "IGMPSN L2MCM", ipl= 0, pid= 198
-Traceback= D9BAA8 D9C1F4 17398D4 173BDF4 173BF68 1B8014 1B8464 1B8BAC 1B7184 7AAF1C 7AD220 7D0474 7D8274 7CB0F8 7AEF18 7AFBCC
Dec 15 11:02:50 WST: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for REP notify Mul. No memory available -Proc                                 ^
% Invalid input detected at '^' marker.ess= "Chunk Manager", ipl= 4, pid= 1
-Traceback= D9BAA8 D9C1F4 1738F20 A6E77C A6524C
Dec 15 11:03:01 WST: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for REP Cam Flush . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= D9BAA8 D9C1F4 1738F20 A6E77C A6524C

This output from below was after rebooting to get it back.

sh proc cpu | excl 0.00
CPU utilization for five seconds: 100%/0%; one minute: 99%; five minutes: 70%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   3        7409     42578        174  1.25%  1.22%  0.92%   0 OSPF-120 Hello  
  40         334       221       1511  0.15%  0.03%  0.02%   0 Compute load avg
  85        1369      4839        282  0.47%  0.27%  0.15%   0 hpm main process
  88         920       592       1554  0.31%  0.12%  0.11%   0 hpm counter proc
119        4551      9994        455  0.62%  0.71%  0.52%   0 Hulc LED Process
175        7950     34997        227  0.78%  0.99%  0.83%   0 IP Input        
184         516      1555        331  0.15%  0.06%  0.03%   0 Spanning Tree   
198      238231       549     433936 57.60% 48.72% 31.35%   0 IGMPSN L2MCM    
213       66173       794      83341 28.32% 18.53%  9.41%   0 REP BPA/EPA Proc
272        6970     24061        289  0.31%  1.48%  0.96%   0 HSRP Common     
276        2480       867       2860  4.09%  1.81%  0.50%   0 IGMP Input

Current System image file is "flash:me340x-metroipaccessk9-mz.122-50.SE1/me340x-metroipaccessk9-mz.122-50.SE1.bin"

Any ideas why downing the interface would cause these two proccess to go off the rails?

Thanks

Mark

[paolo bevilacqua · Accepted solution]

Any ideas why downing the interface would cause these two proccess to go off the rails?

Because of some bug. You should contact the TAC for assistance.

[Michael Simon · Accepted solution]

A complicating factor is that you ran out of memory.

From the memory allocation errors recorded in the logs you caputred your IO pool memory was down to less than 3000 bytes, not nearly enough to run.

Dec 15 11:02:16 WST: %SYS-2-MALLOCFAIL: Memory allocation of 260 bytes failed from 0x1B8010, alignment 0
Pool: I/O  Free: 2312  Cause: Memory fragmentation
Alternate Pool: None  Free: 0  Cause: No Alternate pool
-Process= "IGMPSN L2MCM", ipl= 0, pid= 198

Note the process trying to allocate memory and failing was one of the ones that had high CPU.

Also your processor memory pool was heavily depleted and seriously fragmented causing similar problems:

Dec 15 11:01:46 WST: %SYS-2-MALLOCFAIL: Memory allocation of 4120 bytes failed from 0x7ADECC, alignment 0
Pool: Processor  Free: 658564  Cause: Memory fragmentation
Alternate Pool: None  Free: 0  Cause: No Alternate pool
-Process= "REP Topology change", ipl= 0, pid= 157

The processor pool had no free block of memory largenough to allocate s 4K block.  This is a severe impairment.

When you exhaust either memory pool you will likely crash.

You should consider opening a TAC case to have assistnace validating if there is a memory leak on this system.

There are several IOS bugs involving memory leaks resolved in newer 12.2SE images.

........Mike

[mark_tegg]

Hi Mike

The logs gathered are only there after pulling the fibre.

Memory and CPU are fine until the cable is pulled.

I'll get hold of TAC on this one anyhow.

Thanks for the assistance,

Mark