Re: %C4K_SWITCHINGENGINEMAN-3-PACKETMEMORYTESTPARTIALFAILURE

kingdiamond_ · ‎08-18-2020

Hi guys,

Anyone familiar with this issue and how you were able to resolved it? Thanks in advanced for the help.

*Mar 12 18:53:09.499: %C4K_SYSMAN-2-POWERONSELFTESTFAIL: Supervisor module in slot 1 failed Power-On-Self-Test(POST). Line cards are **NOT** initialized. Please use 'show diagnostic result module 1 test <tid> detail' command for details.

...

Aug 8 23:57:56.891 UTC: %C4K_SWITCHINGENGINEMAN-3-PACKETMEMORYTESTPARTIALFAILURE: Packet Memory buffer test detected errors with 25% of the packet buffers. Switch operation will continue, with potentially reduced performance. Use 'show diagnostic result module all detail' command to see test results.

marce1000 · ‎08-19-2020

- What is the result of the command : show diagnostic result module 1 test <tid> detail

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

kingdiamond_ · ‎08-19-2020

Hi marce1000,

Here's the output:

switch01#show diagnostic result module 1 test 1 detail

Test results: (. = Pass, F = Fail, U = Untested)

___________________________________________________________________________

1) supervisor-bootup ---------------> F

Error code ------------------> 4 (DIAG_PARTIAL_FAILURE)
Total run count -------------> 1
Last test execution time ----> Mar 12 2019 18:53:24
First test failure time -----> Mar 12 2019 18:53:24
Last test failure time ------> Mar 12 2019 18:53:24
Last test pass time ---------> n/a
Total failure count ---------> 1
Consecutive failure count ---> 1

Power-On-Self-Test Results for ACTIVE Supervisor

Power-on-self-test for Module 1: WS-X4013+
Port/Test Status: (. = Pass, F = Fail, U = Untested)
Reset Reason: PowerUp

Cpu Subsystem Tests ...
seeprom: . temperature_sensor: .

Port Traffic: L2 Serdes Loopback ...
0: F 1: F 2: F 3: F 4: F 5: F 6: F 7: F 8: F 9: F 10: F 11: F
12: F 13: F 14: F 15: F 16: F 17: F 18: F 19: F 20: F 21: F 22: F 23: .
24: F 25: F 26: F 27: F 28: F 29: F 30: F 31: .

Port Traffic: L2 Asic Loopback ...
0: F 1: F 2: F 3: F 4: F 5: F 6: F 7: F 8: F 9: F 10: F 11: F
12: F 13: F 14: F 15: F 16: F 17: F 18: F 19: F 20: F 21: F 22: F 23: F
24: F 25: F 26: F 27: F 28: F 29: . 30: . 31: F

Port Traffic: L3 Asic Loopback ...
0: . 1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: .
12: . 13: . 14: . 15: . 16: . 17: . 18: F 19: F 20: F 21: F 22: F 23: F
24: F 25: F 26: F 27: F 28: F 29: F 30: F 31: F

Switch Subsystem Memory ...
1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: . 12: .
13: . 14: . 15: . 16: . 17: . 18: . 19: . 20: . 21: . 22: . 23: . 24: .
25: . 26: . 27: . 28: . 29: . 30: . 31: . 32: . 33: . 34: . 35: . 36: .
37: . 38: . 39: . 40: . 41: . 42: . 43: . 44: . 45: . 46: . 47: . 48: .
49: . 50: . 51: . 52: . 53: . 54: . 55: .

Module 1 Failed

___________________________________________________________________________

switch01#show diagnostic result module 1 test 2 detail

Test results: (. = Pass, F = Fail, U = Untested)

___________________________________________________________________________

2) packet-memory-bootup ------------> F

Error code ------------------> 4 (DIAG_PARTIAL_FAILURE)
Total run count -------------> 1
Last test execution time ----> Mar 12 2019 18:53:24
First test failure time -----> Mar 12 2019 18:53:24
Last test failure time ------> Mar 12 2019 18:53:24
Last test pass time ---------> n/a
Total failure count ---------> 1
Consecutive failure count ---> 1
packet buffers on free list: 48410 bad: 16384 used for ongoing tests: 742

Number of errors found: 2568
SRAM: blocks
U21 {0-3}
Cells with hard errors (failed two or more tests): 753
Cells with soft errors (failed one test, includes hard): 754
Suspect bad cells (uses a block that tested bad): 15630
total buffers: 65536
bad buffers: 16384 (25.0%)
good buffers: 49152 (75.0%)
Bootup test results:2
Displaying first 2048 errors
# Address Observed Expected Data Bits
SramAddr Data Bits Row,Column,Block
131104 0x01001040 0x0000000000084000 0x0000000000000000 {14,19}
0x10200 U21:{1,4} R:256,C:4,B:0
131111 0x010013C0 0x0000000000080000 0x0000000000000000 {19}
0x13A00 U21:{1} R:312,C:4,B:0
131168 0x01003040 0x0000000000020000 0x0000000000000000 {17}
0x10600 U21:{3} R:260,C:4,B:0
131172 0x01003238 0x0800000000000000 0x0000000000000000 {123}
0x12600 U21:{0} R:292,C:4,B:0
131175 0x010033B8 0x0800000000000000 0x0000000000000000 {123}
0x13E00 U21:{0} R:316,C:4,B:0
131177 0x010034B8 0x0800000000000000 0x0000000000000000 {123}
0x14E00 U21:{0} R:332,C:4,B:0
131232 0x01005040 0x00000040000A8000 0x0000000000000000 {15,17,19,38}
0x10201 U21:{1,3,6,8} R:256,C:4,B:1
131233 0x010050C0 0x0000000000080000 0x0000000000000000 {19}
0x10A01 U21:{1} R:264,C:4,B:1
131234 0x01005140 0x0000000000080000 0x0000000000000000 {19}
0x11201 U21:{1} R:272,C:4,B:1
131237 0x010052C0 0x0000000000080000 0x0000000000000000 {19}
0x12A01 U21:{1} R:296,C:4,B:1
131238 0x01005340 0x0000000000080000 0x0000000000000000 {19}
0x13201 U21:{1} R:304,C:4,B:1
131243 0x010055C0 0x0000000000080000 0x0000000000000000 {19}
0x15A01 U21:{1} R:344,C:4,B:1
131296 0x01007040 0x0000004000008000 0x0000000000000000 {15,38}
0x10601 U21:{6,8} R:260,C:4,B:1
131297 0x010070B8 0x0800000000000000 0x0000000000000000 {123}

switch01#show diagnostic result module 1 test 3 detail

Test results: (. = Pass, F = Fail, U = Untested)

___________________________________________________________________________

3) packet-memory-ongoing -----------> U

Error code ------------------> 0 (DIAG_SUCCESS)
Total run count -------------> 0
Last test execution time ----> n/a
First test failure time -----> n/a
Last test failure time ------> n/a
Last test pass time ---------> n/a
Total failure count ---------> 0
Consecutive failure count ---> 0
packet buffers on free list: 48410 bad: 16384 used for ongoing tests: 742

Packet memory errors: 0 5107
Current alert level: green
Per 5 seconds in the last minute:
0 0 0 0 0 0 0 0 0 0
0 0
Per minute in the last hour:
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Per hour in the last day:
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0
Per day in the last 30 days:
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Direct memory test failures per minute in the last hour:
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Potential false positives: 0 0
Ignored because of rx errors: 0 0
Ignored because of cdm fifo overrun: 0 0
Ignored because of oir: 0 0
Ignored because isl frames received: 0 0
Ignored during boot: 0 0
Ignored after writing hw stats: 0 0
Ignored on high gigaport: 0
Ongoing diag action mode: Normal
Last 1000 Memory Test Failures:
Log start time: 15271603
0 0 addr 131168[8] 0xFFFFFFFFFFFFFFFF 0xFFFFFFFFFFD77FFF
1 15 addr 131169[7] 0xFFFFFFFFFFFFFFFF 0xF7FFFFFFFFFFFFFF

marce1000 · ‎08-19-2020

- Which switch-model /software-version is this ?

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

kingdiamond_ · ‎08-19-2020

Hi @marce1000

SW version: Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-IPBASEK9-M), Version 12.2(50)SG1, RELEASE SOFTWARE (fc2)

Device Model: cisco WS-C4506 (MPC8245) processor (revision 2) with 262144K bytes of memory

Thanks!

marce1000 · ‎08-20-2020

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCse80413

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

kingdiamond_ · ‎08-22-2020

Hi marce1000,

Based on the bug search article you've shared, they are suggesting to use the workaround below:

Workaround: Reset the module with faulty staus using command hw-module module reset and the module will come up without any failures.

My question is if running this command hw-module module x reset will have any impact on production switch device? Thank you so much! :)

marce1000 · ‎08-23-2020

- Not , if you have two supervisors.

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

kingdiamond_ · ‎08-23-2020

Hi @marce1000 ,

I tried running below command, based on the url link that you gave. But I didn't find any Symbol error. Also, it's our supervisor engine module having the issue and not line cards. We can still issue the command "hw-module module x reset" for a supervisor engine issue? We only have one sup engine as well. Thanks a lot!

ditri-switch#show platform software interface stub internal | inc Symbol
ditri-switch#

marce1000 · ‎08-24-2020

- Hardware reset (command) will indeed not be advisable when there is only one supervisor. You may have a look at the Known fixed releases and upgrade , if applicable. Or discuss the issue further with Cisco TAC.

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '