Solved: Re: unresolved fault detected on Cisco ASR1001-X

BorislavPenchev0962 · ‎09-06-2021

Hi

we got constantly the following errors on ASR1001 / Version 16.09.06 last night,

then router rebooted and now seems to be ok, but we are wondering what happened ?

 %CPPHA-3-FAULT: R0/0: cpp_ha_top_level_server: CPP:0.0 desc:DPE1_CSR32_DPE_ERR_MISC_LEAF_INT__INT_DRAM_MBE det:DRVR(interrupt) class:MBE sev:FATAL id:7075 cppstate:STOPPED res:UNKNOWN flags:0x7 cdmflags:0x1
 %CPPOSLIB-3-ERROR_NOTIFY: R0/0: cpp_ha_top_level_server: cpp_ha encountered an error -Traceback= 1#e0a954143630838337b21cc147cd64f3 errmsg:7FE4B50A5000+17D4 cpp_common_os:7FE4B9C4A000+DB8C cpp_common_os:7FE4B9C4A000+1B9EE cpp_drv_cmn:7FE4B772D000+45987 :400000+27309 :400000+26DCC :400000+26824 :400000+1522D :400000+1411C cpp_common_os:7FE4B9C4A000+11DF0 cpp_common_os:7FE4B9C4A000+124E6 evlib:7FE4B3269000+8942 evlib:7FE4B3269000+944C cpp_common_os:7FE4B9C4A000+140C2 :400000+F6A7 c:7FE4A9E68000+208E5 :400000+9F79
%CPPHA-3-FAULTCRASH: R0/0: cpp_ha_top_level_server: CPP 0.0 unresolved fault detected, initiating crash dump.
 %CPPHA-3-FAULTCRASH: R0/0: cpp_ha_top_level_server: CPP 0.0 unresolved fault detected, initiating crash dump.
 %IOSXE-1-PLATFORM: R0/0: kernel: QFP0.0: Fatal Fault: HW reported: QFP interrupt %IOSXE-3-PLATFORM: R0/0: cpp_cdm: CPP crashed, generating core file.
%CPPDRV-3-LOCKDOWN: R0/0: cpp_ha_top_level_server: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
 %CPPOSLIB-3-ERROR_NOTIFY: R0/0: fman_fp_image: fman_fp encountered an error -Traceback= 1#cc7b7547c627e12f66e908540c2a55d1 errmsg:7F619E21A000+17D4 cpp_common_os:7F61C7EA4000+DB8C cpp_client_ha:7F61C2AD8000+439F cpp_common_os:7F61C7EA4000+11DF0 cpp_common_os:7F61C7EA4000+124E6 evlib:7F617F233000+8942 evlib:7F617F233000+944C :561C833EE000+385DBF :561C833EE000+60E132 :561C833EE000+934862 :561C833EE000+932FAF :561C833EE000+932E49 :561C833EE000+932CE2 c:7F6177085000+208E5 :561C833EE000+1BA019
 %CPPDRV-3-LOCKDOWN: R0/0: cpp_cp_svr: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
 %CPPDRV-3-LOCKDOWN: R0/0: fman_fp_image: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
 %CPPOSLIB-3-ERROR_NOTIFY: R0/0: cpp_cp_svr: cpp_cp encountered an error -Traceback= 1#10dcd90466307a0dff0ce3916fa1e8be errmsg:7F1401D7C000+17D4 cpp_common_os:7F1403FC3000+DB8C cpp_common_os:7F1403FC3000+1B9EE cpp_rrm_svr_lib:7F1417741000+8553 cpp_rrm_svr_lib:7F1417741000+64C2 cpp_rrm_svr_lib:7F1417741000+487E cpp_rrm_svr_lib:7F1417741000+37F8 cpp_common_os:7F1403FC3000+11DF0 cpp_common_os:7F1403FC3000+124E6 evlib:7F1401B6D000+8942 evlib:7F1401B6D000+944C cpp_common_os:7F1403FC3000+140C2 :400000+7376 c:7F13ED156000+208E5 :
 %CPPHA-3-CDMDONE: R0/0: cpp_ha_top_level_server: CPP 0 microcode crashdump creation completed.
 %PMAN-3-PROCHOLDDOWN: R0/0: pman: The process cpp_cdm_svr has been helddown (rc 69)
 %PMAN-3-PROCHOLDDOWN: R0/0: pman: The process cpp_ha_top_level_server has been helddown (rc 69)
 %PMAN-3-PROCHOLDDOWN: R0/0: pman: The process fman_fp_image has been helddown (rc 134)

Regards

Boris

balaji.bandi · ‎09-06-2021

Looks for me some bug, but not able to get the right bug information, if this kit in smartnet contract, worth open a TAC case.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

View solution in original post

Leo Laohoo · ‎09-06-2021

@BorislavPenchev0962 wrote:

 %CPPDRV-3-LOCKDOWN: R0/0: cpp_cp_svr: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
 %CPPDRV-3-LOCKDOWN: R0/0: fman_fp_image: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
 %CPPHA-3-CDMDONE: R0/0: cpp_ha_top_level_server: CPP 0 microcode crashdump creation completed. %PMAN-3-PROCHOLDDOWN: R0/0: pman: The process cpp_cdm_svr has been helddown (rc 69) 
 %PMAN-3-PROCHOLDDOWN: R0/0: pman: The process cpp_ha_top_level_server has been helddown (rc 69) 
 %PMAN-3-PROCHOLDDOWN: R0/0: pman: The process fman_fp_image has been helddown (rc 134)

@BorislavPenchev0962,

Is this the same router as the other thread (ISR with a very high percentage of the DRAM)?

If this is a different router, kindly post the complete output to the following commands:

dir
dir bootflash:/core
sh log on up

BorislavPenchev0962 · ‎09-07-2021

@Leo Laohooits another router, here the output of this ASR:

dir bootflash:
Directory of bootflash:/

11 drwx 16384 Dec 31 2014 15:49:01 +01:00 lost+found
289729 drwx 4096 May 26 2021 08:29:20 +02:00 .prst_sync
12 -rw- 762178097 Mar 22 2021 07:48:50 +01:00 asr1001x-universalk9.16.09.06.SPA.bin
160961 drwx 4096 Sep 2 2021 02:38:42 +02:00 .installer
482881 drwx 4096 Sep 2 2021 02:36:44 +02:00 core
144865 drwx 4096 Dec 31 2014 16:24:13 +01:00 .rollback_timer
13 -rw- 0 Dec 31 2014 16:24:21 +01:00 tracelogs.226
177057 drwx 24576 Sep 7 2021 13:19:43 +02:00 tracelogs
48289 -rw- 6613972 May 26 2021 08:10:59 +02:00 asr1000-rommon.169_4r_SPA.pkg
338017 drwx 4096 Mar 22 2021 10:14:23 +01:00 virtual-instance
370209 drwx 4096 Mar 22 2021 10:20:32 +01:00 .dbpersist
144867 -rw- 30 Sep 2 2021 02:39:53 +02:00 throughput_monitor_params
209249 drwx 4096 Mar 22 2021 10:15:10 +01:00 onep

6741659648 bytes total (5441617920 bytes free)

#dir bootflash:/core
Directory of bootflash:/core/

498977 drwx 4096 Dec 31 2014 16:24:13 +01:00 modules
482882 -rw- 1 Sep 7 2021 13:10:19 +02:00 .callhome
482884 -rw- 174258856 Sep 2 2021 02:36:44 +02:00 Berlin-43-route-02_RP_0_fman_fp_image_25382_20210902-023554-CEST.core.gz
482883 -rw- 2556277 Sep 2 2021 02:36:02 +02:00 Berlin-43-route-02_RP_0_cpp-mcplo-ucode_20210902-023550-CEST.core.gz

6741659648 bytes total (5441617920 bytes free)

#sh log onboard uptime
Slot Reset reason Power On
---------------------------------------------------------
0 reset local software 01/01/10 01:26:43
0 reset local software 01/01/10 01:26:45
0 reset local software 01/01/10 01:26:47
0 reset local software 01/01/10 03:36:08
0 reset local software 01/01/10 03:36:09
0 reset local software 01/01/10 03:36:12
0 reset local software 12/29/14 05:53:26
0 reset local software 12/29/14 05:53:27
0 reset local software 12/29/14 05:53:30
0 reset local software 12/29/14 12:28:37
0 reset local software 12/29/14 12:28:38
0 reset local software 12/29/14 12:28:41
0 reset local software 12/31/14 12:15:22
0 reset local software 12/31/14 12:15:23
0 reset local software 12/31/14 12:15:26
0 reset local software 12/31/14 12:50:42
0 reset local software 12/31/14 12:50:43
0 reset local software 12/31/14 12:50:46
0 reset local software 12/31/14 15:24:04
0 reset local software 12/31/14 15:24:05
0 reset local software 12/31/14 15:24:08
0 reset local software 12/31/14 15:41:41
0 reset local software 12/31/14 15:41:42
0 reset local software 12/31/14 15:41:45
0 reset power on 12/31/14 16:24:06
0 reset power on 12/31/14 16:24:07
0 reset power on 12/31/14 16:24:10
0 reset power on 05/11/15 06:27:45
0 reset power on 05/11/15 06:27:46
0 reset power on 05/11/15 06:27:49
0 reset local software 05/11/15 06:34:35
0 reset local software 05/11/15 06:34:36
0 reset local software 05/11/15 06:34:39
0 reset power on 05/12/15 23:15:01
0 reset power on 05/12/15 23:15:02
0 reset power on 05/12/15 23:15:05
0 reset power on 05/16/15 05:38:03
0 reset power on 05/16/15 05:38:04
0 reset power on 05/16/15 05:38:07
0 reset power on 05/18/15 03:00:13
0 reset power on 05/18/15 03:00:14
0 reset power on 05/18/15 03:00:17
0 reset power on 05/21/15 01:03:26
0 reset power on 05/21/15 01:03:27
0 reset power on 05/21/15 01:03:30
0 reset power on 05/27/15 01:18:09
0 reset power on 05/27/15 01:18:10
0 reset power on 05/27/15 01:18:13
0 reset power on 06/18/15 01:21:50
0 reset power on 06/18/15 01:21:51
0 reset power on 06/18/15 01:21:54
0 reset power on 06/19/15 01:06:11
0 reset power on 06/19/15 01:06:12
0 reset power on 06/19/15 01:06:15
0 reset local software 06/19/15 01:20:25
0 reset local software 06/19/15 01:20:26
0 reset local software 06/19/15 01:20:29
0 reset local software 06/19/15 01:40:46
0 reset local software 06/19/15 01:40:47
0 reset local software 06/19/15 01:40:50
0 reset local software 06/19/15 02:02:09
0 reset local software 06/19/15 02:02:10
0 reset local software 06/19/15 02:02:13
0 reset power on 06/21/15 23:49:20
0 reset power on 06/21/15 23:49:21
0 reset power on 06/21/15 23:49:24
0 reset power on 06/22/15 05:42:52
0 reset power on 06/22/15 05:42:53
0 reset power on 06/22/15 05:42:56
0 reset power on 06/23/15 05:38:10
0 reset power on 06/23/15 05:38:11
0 reset power on 06/23/15 05:38:14
0 reset local software 07/08/15 23:56:58
0 reset local software 07/08/15 23:56:59
0 reset local software 07/08/15 23:57:02
0 reset local software 12/21/16 14:37:17
0 reset local software 12/21/16 14:37:18
0 reset local software 12/21/16 14:37:21
0 reset local software 10/08/20 11:54:41
0 reset local software 10/08/20 11:54:42
0 reset local software 10/08/20 11:54:45
0 reset power on 10/21/20 11:06:00
0 reset power on 10/21/20 11:06:01
0 reset power on 10/21/20 11:06:04
0 reset local software 03/22/21 08:06:45
0 reset local software 03/22/21 08:06:46
0 reset local software 03/22/21 08:06:49
0 reset local software 03/22/21 08:22:32
0 reset local software 03/22/21 08:22:33
0 reset local software 03/22/21 08:22:36
0 reset local software 03/22/21 09:27:41
0 reset local software 03/22/21 09:27:42
0 reset local software 03/22/21 09:27:45
0 upgrade flash reset 03/22/21 10:13:54
0 reset local software 03/22/21 10:23:42
0 upgrade flash reset 05/26/21 08:28:18
0 reset local software 09/02/21 02:38:46

Leo Laohoo · ‎09-07-2021

The traceback does not have time and date stamp.

Is there a reason why this is omitted?

BorislavPenchev0962 · ‎09-07-2021

@Leo Laohool have erased this for better visibility, all the lines are from few days ago and within 5 min. ;

Leo Laohoo · ‎09-07-2021

@BorislavPenchev0962 wrote:

482884 -rw- 174258856 Sep 2 2021 02:36:44 +02:00 Berlin-43-route-02_RP_0_fman_fp_image_25382_20210902-023554-CEST.core.gz
482883 -rw- 2556277 Sep 2 2021 02:36:02 +02:00 Berlin-43-route-02_RP_0_cpp-mcplo-ucode_20210902-023550-CEST.core.gz

Kindly attach both files to the thread. We want to have a look inside.

BorislavPenchev0962 · ‎09-08-2021

second file is bigger, l can send it per moveit if you share e-mail ?

Leo Laohoo · ‎09-08-2021

Attach it to the thread.

BorislavPenchev0962 · ‎09-13-2021

upload limit is 70 MB , file size is 160 MB

Leo Laohoo · ‎09-14-2021

Contact TAC and see if they can confirm this is CSCvx57833 or CSCvf11949.

paul driver · ‎09-07-2021

Hello

Check the power supply!

Show platform
Show environment

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

BorislavPenchev0962 · ‎09-07-2021

Hi,

here the output of these commands:

#sh platform
Chassis type: ASR1001-X

Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ASR1001-X ok 5d10h
0/0 BUILT-IN-2T+6X1GE ok 5d10h
R0 ASR1001-X ok, active 5d10h
F0 ASR1001-X ok, active 5d10h
P0 ASR1001-X-PWR-AC ok 5d10h
P1 ASR1001-X-PWR-AC ok 5d10h
P2 ASR1001-X-FANTRAY ok 5d10h

Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 14041015 16.9(4r)
R0 14041015 16.9(4r)
F0 14041015 16.9(4r)

Berlin#sh environment

Number of Critical alarms: 0
Number of Major alarms: 0
Number of Minor alarms: 0

Slot Sensor Current State Reading Threshold(Minor,Major,Critical,Shutdown)
---------- -------------- --------------- ------------ ---------------------------------------
P0 PEM Iout Normal 5 A na
P0 PEM Vout Normal 12 V DC na
P0 PEM Vin Normal 244 V AC na
P0 Temp: PEM In Normal 33 Celsius (80 ,90 ,95 ,100)(Celsius)
P0 Temp: PEM Out Normal 31 Celsius (80 ,90 ,95 ,100)(Celsius)
P0 Temp: PEM Int Normal 39 Celsius (80 ,90 ,95 ,100)(Celsius)
P1 PEM Iout Normal 5 A na
P1 PEM Vout Normal 12 V DC na
P1 PEM Vin Normal 243 V AC na
P1 Temp: PEM In Normal 31 Celsius (80 ,90 ,95 ,100)(Celsius)
P1 Temp: PEM Out Normal 30 Celsius (80 ,90 ,95 ,100)(Celsius)
P1 Temp: PEM Int Normal 33 Celsius (80 ,90 ,95 ,100)(Celsius)
R0 Temp: sTCAM Normal 29 Celsius (65 ,72 ,80 ,100)(Celsius)
R0 Temp: Inlet Normal 24 Celsius (55 ,65 ,75 ,80 )(Celsius)
R0 Temp: Outlet Normal 35 Celsius (75 ,80 ,85 ,100)(Celsius)
R0 Temp: QFP Die Normal 37 Celsius (100,110,120,125)(Celsius)
R0 Temp: Center Normal 37 Celsius (75 ,80 ,85 ,100)(Celsius)
R0 Temp: Oct Die Normal 39 Celsius (100,110,120,125)(Celsius)
R0 Temp: CPU Inlt Normal 29 Celsius (75 ,85 ,95 ,100)(Celsius)
R0 Temp: CPU VRM Normal 27 Celsius (75 ,85 ,95 ,100)(Celsius)
R0 Temp: CPU Die Normal 33 Celsius (100,110,120,125)(Celsius)
R0 Temp: FC FANS Fan Speed 60% 24 Celsius (25 ,35 ,0 )(Celsius)
P0 Temp: FC FAN0 Fan Speed 60% 24 Celsius (25 ,35 ,0 )(Celsius)
P1 Temp: FC FAN1 Fan Speed 60% 24 Celsius (25 ,35 ,0 )(Celsius)

Regards

Boris

BorislavPenchev0962 · ‎09-13-2021

Hi,

was your comment about the Power Supply related to #sh log onboard uptime output ?

l have edit it as initially was from another device - my mistake.

thanks and regards

Boris

balaji.bandi · ‎09-06-2021

Looks for me some bug, but not able to get the right bug information, if this kit in smartnet contract, worth open a TAC case.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help