Problem Detail:
For one of the customer, we found one of our closet switches with several cards all amber. Logs shows several errors resulting in the modules all resetting. They all passed diag but many were inoperable. Subsequently, a hw-mod mod reset was performed on one module and the connectivity restored to end devices but it still produced an error message in the logs during the reset. The switch is rebooted.
The main logs noticed are extracted here. The complete logs are attached.
<snip>
Feb 12 03:32:19.584 CST: Frames of RPC c6k_sp_environmental process (pid 350) on 6 (proc|slot) after blocking rpc call failed: 40A1C290 413AB06C 409D2DFC 409D2DE8^A
Feb 12 03:32:19.584 CST: RPC-2-RETRY Recovered from RPC send failure for request c6k_sp_environmental:env_get_sensor_value_sp. Resending request.
-Traceback= 41CEDAE0 41FF9F44 41FFA830 41FFBB30 42002480 4137E16C 4137E158
.
.
Feb 12 03:32:38.336 CST: PM-SP-2-NOMEM Not enough memory available for card internal data
.
Feb 12 03:33:13.500 CST: SYS-SP-2-MALLOCFAIL Memory allocation of 234624 bytes failed from 0x4126CC74, alignment 0 Pool: Processor Free: 1637660 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool -Process= "RPC pm-cp", ipl= 0, pid= 355 -Traceback= 409E82A8 409F4EE8 4126CC7C 4125AF78 413060EC 41307944 413AAD98 413AB0A4 409D2DFC 409D2DE8
.
.
Feb 12 03:29:37.100 CST: SW_VLAN-SP-3-VLAN_PM_NOTIFICATION_FAILURE VLAN Manager synchronization failure with Port Manager over port mode change
-Traceback= 4094AD7C 412AAA7C 412AAB3C 408FE1D4 412B2494 4125AEB4 412FC76C 41300828 409D2DFC 409D2DE8
Feb 12 03:29:37.100 CST: SW_VLAN-SP-3-VLAN_PM_NOTIFICATION_FAILURE VLAN Manager synchronization failure with Port Manager over port link change (1)
-Traceback= 4094A83C 412A3780 412A3890 408FE1D4 412B2494 4125AEB4 412FC76C 41300828 409D2DFC 409D2DE8
.
.
Feb 12 03:29:53.202 CST: SYS-SP-2-CHUNKEXPANDFAIL Could not expand chunk pool for messages. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1
-Traceback= 409E7028 409D2DFC 409D2DE8
...
Feb 12 03:29:40.542 CST: SP: icc_req_debug_send_message: icc_send_request failed
Feb 12 03:29:40.542 CST: SP: icc_req_debug_get_pid_info: failed to get the process stack on the peer after blocking rpc call failed
...
Upon analysis, I identified the below bugs. However, both bugs are duplicates.
CSCtd60912 Crash after 'pm failed to send port admin change notification'
CSCtx47138 SW_VLAN-3-VLAN_PM_NOTIFICATION_FAILURE:VLAN Manager synchronization fail
After reload the device is working normal as expected.
Can someone provide some assistance in this case?
Thanks, Anand
