Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
5708
Views
5
Helpful
18
Replies

ASR 1001-X QFP/ESP crash - CPPHA-3-FAULT: CPP:0.0

paul amaral
Level 4
Level 4

‎02-14-2023 09:08 PM

Hi, I have a Cisco ASR 1001-X that has been solid for years. Suddenly its crashing. I has crashed 3 times in the last 5 hours. From what I can tell its an error with the ESP, although I'm not sure if it's a remote vulnerability or bug. If it's a bug It's one I never hit before while running 16.12.5 XE for years. I have searched I cannot seem to find what is causing this issue, below I have all relevant information including the error. If anyone has any ideas it would be tremendously helpful. 

TIA, P

 

 

System returned to ROM by Reload reason not captured at 00:05:00 est Fri Feb 26 2021
System restarted at 21:07:36 est Tue Feb 14 2023
System image file is "bootflash:asr1001x-universalk9_noli.16.12.05.SPA.bin"
Last reload reason: Reload reason not captured

Directory of bootflash:/core/

696257  drwx             4096  Jan 22 2019 01:53:43 -05:00  modules
680066  -rw-                1  Feb 14 2023 21:53:05 -05:00  .callhome
680067  -rw-        216677893  Feb 14 2023 20:26:52 -05:00  EDGE-FRV-MA-ASR1001X-EDGE_RP_0-system-report_20230214-202648-est.tar.gz
680068  -rw-            20115  Feb 14 2023 20:27:09 -05:00  EDGE-FRV-MA-ASR1001X-EDGE_RP_0-system-report_20230214-202648-est-info.txt
680069  -rw-        131926947  Feb 14 2023 21:03:05 -05:00  EDGE-FRV-MA-ASR1001X-EDGE_RP_0-system-report_20230214-210302-est.tar.gz
680070  -rw-            16610  Feb 14 2023 21:03:17 -05:00  EDGE-FRV-MA-ASR1001X-EDGE_RP_0-system-report_20230214-210302-est-info.txt

 

-- Logs begin at Tue 2023-02-14 20:28:46 est, end at Tue 2023-02-14 21:03:19 est. --
Feb 14 21:02:02 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_ha[24905]: %CPPHA-3-FAULT: CPP:0.0 desc:JTB_CSR32_JTB_ERR_JTB_LEAF_INT__INT_SPI4_PLL_LOSTLOCK det:DRVR(interrupt) class:OTHER sev:FATAL id:6628 cppstate:STOPPED res:UNKNOWN flags:0x7 cdmflags:0x0
Feb 14 21:02:02 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_ha[24905]: %CPPOSLIB-3-ERROR_NOTIFY: cpp_ha encountered an error -Traceback= 1#32e24f4e1e5bb3c64e4c8160ec8a858f   errmsg:7FA514D74000+A80 cpp_common_os:7FA5199AB000+DB8C cpp_common_os:7FA5199AB000+1BA6E cpp_drv_cmn:7FA517453000+47C37 :400000+28849 :400000+2830C :400000+27D64 :400000+15ABD :400000+1499C cpp_common_os:7FA5199AB000+11DF0 cpp_common_os:7FA5199AB000+124E6 evlib:7FA512E5C000+8F37 evlib:7FA512E5C000+997C cpp_common_os:7FA5199AB000+14142 :400000+FD7F c:7FA508D27000+209B2 :400000+A629
Feb 14 21:02:02 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_ha[24905]: %CPPHA-3-FAULTCRASH: CPP 0.0 unresolved fault detected, initiating crash dump.
Feb 14 21:02:02 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_ha[24905]: %CPPHA-3-FAULTCRASH: CPP 0.0 unresolved fault detected, initiating crash dump.
Feb 14 21:02:02 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_cdm[25456]: CPP crashed, collecting state.
Feb 14 21:02:02 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_driver[25086]: %CPPDRV-6-INTR: Luke(0) Interrupt : 23-Feb-14 21:02:02.715253 UTC-0500:FIRST:HALT:JTB_CSR32_JTB_ERR_JTB_LEAF_INT__INT_SPI4_PLL_LOSTLOCK
Feb 14 21:02:03 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_cp[25318]: %CPPDRV-3-LOCKDOWN: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
Feb 14 21:02:03 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_ha[24905]: %CPPDRV-3-LOCKDOWN: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
Feb 14 21:02:03 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 fman_fp_image[24173]: %CPPDRV-3-LOCKDOWN: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
Feb 14 21:02:03 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 audispd[498]: type=ANOM_ABEND msg=audit(1676426523.365:100): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=24173 comm="fman_fp_image" exe="/tmp/sw/mount/asr1001x-espbase.16.12.05.SPA.pkg/usr/binos/bin/fman_fp_image" sig=6
Feb 14 21:02:05 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_cdm[25456]: CPP crashed, generating core file.
Feb 14 21:02:35 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_ha[24905]: %CPPHA-3-CDMDONE: CPP 0 microcode crashdump creation completed.
Feb 14 21:02:35 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_cdm[25456]: Shutting down CPP MDM while client(s) still connected
Feb 14 21:02:35 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_ha[24905]: Shutting down CPP MDM while client(s) still connected
Feb 14 21:02:35 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_ha[24905]: Shutting down CPP CDM while client(s) still connected
Feb 14 21:02:35 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: QFP0.0: Fatal Fault: HW reported: QFP interrupt
Feb 14 21:02:35 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: CPP 0 pid 25456: Unregistered all subdevices for access error
Feb 14 21:02:35 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: CPP 0 pid 24905: Unregistered all subdevices for access error
Feb 14 21:02:35 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 root[13748]: %PMAN-3-PROCHOLDDOWN: The process cpp_ha_top_level_server has been helddown (rc 69)
Feb 14 21:02:35 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 root[13749]: %PMAN-3-PROCHOLDDOWN: The process cpp_cdm_svr has been helddown (rc 69)
Feb 14 21:02:36 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 xinetd[13835]: execve /usr/bin/rsync
Feb 14 21:02:36 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_cp[25318]: Shutting down CPP MDM while client(s) still connected
Feb 14 21:02:36 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_stats[24561]: %CPPOSLIB-3-ERROR_NOTIFY: cpp_stats encountered an error -Traceback= 1#0086d91d8299ea2d9c5e950517ddceba   errmsg:7F67393F1000+A80 cpp_common_os:7F673B715000+DB8C cpp_common_os:7F673B715000+1BA6E cpp_cdm:7F674A29F000+248D cpp_cdm:7F674A29F000+1EFD cpp_cdm:7F674A29F000+1C99 cpp_cdm:7F674A29F000+1B0C
Feb 14 21:02:36 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: CPP 0 pid 25086: Unregistered all subdevices for access error
Feb 14 21:02:36 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 cpp_sp[24724]: Shutting down CPP MDM while client(s) still connected
Feb 14 21:02:36 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: CPP 0 pid 24724: Unregistered all subdevices for access error
Feb 14 21:02:36 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: CPP 0 pid 24561: Unregistered all subdevices for access error
Feb 14 21:02:36 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: CPP 0 pid 25318: Unregistered all subdevices for access error
Feb 14 21:02:46 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 fman_rp[18031]: %FMANRP-3-PEER_IPC_STUCK: IPC to fman-log-bay0-peer0 is stuck for more than 30 seconds
Feb 14 21:02:49 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: CPP 0 pid 24173: Unregistered all subdevices for access error
Feb 14 21:02:49 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 root[14237]: %PMAN-3-PROCHOLDDOWN: The process fman_fp_image has been helddown (rc 134)
Feb 14 21:02:50 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 pvp[14270]: %PMAN-5-EXITACTION: Process manager is exiting: process exit with reload fru code
Feb 14 21:02:50 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: WARNING: SPA 1 is not EBFC, ignore
Feb 14 21:02:50 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 btman_rotate_immediate[14747]: %SERVICES-3-INVALID_CHASFS: Thread 0x7f5d13320380 has no global chasfs context
Feb 14 21:02:50 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 btman_rotate_immediate[14747]: %SERVICES-2-NORESOLVE_ACTIVE: Error resolving active FRU: BINOS_FRU_RP
Feb 14 21:02:50 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 btman_rotate_immediate[14877]: %SERVICES-3-INVALID_CHASFS: Thread 0x7f476600d380 has no global chasfs context
Feb 14 21:02:50 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 btman_rotate_immediate[14877]: %SERVICES-2-NORESOLVE_ACTIVE: Error resolving active FRU: BINOS_FRU_RP
Feb 14 21:02:51 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 xinetd[14950]: execve /usr/bin/rsync
Feb 14 21:02:51 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 xinetd[14954]: execve /usr/bin/rsync
Feb 14 21:02:52 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 xinetd[14959]: execve /usr/bin/rsync
Feb 14 21:02:58 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: cpp_pdma_proc_thp_chan_info: len: 2856
Feb 14 21:02:58 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: cpp_pdma_proc_thp_chan_info: len: 2783
Feb 14 21:02:58 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: cpp_pdma_proc_thp_chan_info: len: 2839
Feb 14 21:02:58 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: cpp_pdma_proc_thp_chan_info: len: 2835
Feb 14 21:03:16 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 fman_rp[18031]: %FMANRP-3-PEER_IPC_RESUME: IPC to fman-log-bay0-peer0 has returned to normal after previous stuck
Feb 14 21:03:17 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 pvp[16246]: %PMAN-3-PROCESS_NOTIFICATION: System report core/EDGE-FRV-MA-ASR1001X-EDGE_RP_0-system-report_20230214-210302-est.tar.gz (size: 128835 KB) generated and System report info at core/EDGE-FRV-MA-ASR1001X-EDGE_RP_0-system-report_20230214-210302-est-info.txt
Feb 14 21:03:18 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 kernel: LSMPI: Deregister dual stack diverter
Feb 14 21:03:19 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 pvp[17464]: %PMAN-5-EXITACTION: Process manager is exiting: reload fru action requested
Feb 14 21:03:19 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 btman_rotate_immediate[17544]: %SERVICES-3-INVALID_CHASFS: Thread 0x7f0e6a0ba380 has no global chasfs context
Feb 14 21:03:19 EDGE-FRV-MA-ASR1001X-EDGE_RP_0 btman_rotate_immediate[17544]: %SERVICES-2-NORESOLVE_ACTIVE: Error resolving active FRU: BINOS_FRU_RP

 

1 person had this problem
Labels:
0 Helpful
18 Replies 18

paul amaral
Level 4
Level 4
In response to marce1000

‎02-15-2023 12:33 PM

Its happening with the router isolated and not connected to the network, it keep crashing every few hours. I also dont have access to the CLI analyzer but I will look into it. 

Thanks, Paul 

0 Helpful

Georg Pauwen
VIP
VIP

‎02-14-2023 11:58 PM

Hello,

check if you might be hitting the bug below. The workaround is to disable 'ip virtual-reassembly' on your interfaces (which apparently is enabled by default after 16.9):

 

IOS-XE cpp ucode crash with fragmented packets
CSCvy30209  
Symptom: IOS-XE router might crash when we receive two(or more) of same(duplicated) fragments on different interfaces.
%CPPDRV-3-LOCKDOWN: R0/0: cpp_cp: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).@ %CPPDRV-3-LOCKDOWN: R0/0: fman_fp_image: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).@ %CPPDRV-3-LOCKDOWN: R0/0: cpp_ha: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).@ %IOSXE-1-PLATFORM: R0/0: kernel: QFP0.0: Fatal Fault: HW reported: QFP interrupt@
These syslog messages might be seen before the crash
%FRAG-3-REASSEMBLY_ERR: Reassembly/VFR encountered an error: the first fragment's copy of packet state does not exist %FRAG-3-REASSEMBLY_DBG: Reassembly/VFR encountered an error: IPv4 vFR complete with wrong refcnt, actual refcnt 4, expected refcnt 1 %FRAG-3-REASSEMBLY_ERR: Reassembly/VFR encountered an error: the first fragment's copy of packet state does not exist
Conditions: Virtual fragmentation reassembly (VFR) has been configured. This is a VFR day-1 issue and very corner case. We may hit this issue only when same fragments are received on different interfaces at the same time. And these interfaces must have VFR configured via "ip virtual-reassembly" explicitly or enabled automatically by some features (such as NAT, Cisco IOS XE Firewall, IPSec) on them. Workaround: Remove Virtual fragmentation reassembly (VFR) configuration from the interfaces. Starting from 16.9.x and later releases, "Virtual fragmentation reassembly (VFR) is automatically enabled by some features (such as NAT, Cisco IOS XE Firewall, IPSec) ...". It is not required to manually configure the "ip virtual-reassembly" for these features after 16.9.x. Please refer to ?IP Addressing: Fragmentation and Reassembly Configuration Guide? for additional information. https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_frre/configuration/xe-16/frre-xe-16-book/virt-frag-reassembly.html
Further Problem Description: Kernel crashes might be seen after the CCP ucode crash
0 Helpful

paul amaral
Level 4
Level 4
In response to Georg Pauwen

‎02-15-2023 12:26 PM

I don't have this turn on, also i removed the device from the network and even with zero packets it is crashing. Looking more and more like a hardware issue. 

Customers Also Viewed These Support Documents