Cisco ASR1004 - ASR1000-ESP20

V_V · ‎01-12-2012

Hi,

We have an ASR1004 with ASR1000-ESP20 which keeps on rebooting:

Router#

*Jan 13 01:09:02.020: %ASR1000_OIR-6-ONLINECARD: Card (fp) online in slot F0

*Jan 13 01:09:16.649: %CPPHA-7-START: F0: cpp_ha: CPP 0 preparing image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

*Jan 13 01:09:17.001: %CPPHA-7-START: F0: cpp_ha: CPP 0 startup init image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

*Jan 13 01:09:20.764: %CPPOSLIB-3-ERROR_NOTIFY: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6117]: cpp_driver-0 encountered an error -Traceback= 1#939a1e943669c32003ea84676fda1830 errmsg:F4DB000+2160 cpp_common_os:FB1E000+BB20 cpp_common_os:FB1E000+B930 cpp_drv_cmn:FC73000+69030 cpp_drv_cmn:FC73000+DC740 cpp_drv_cmn:FC73000+D51E8 cpp_drv_cmn:FC73000+D8978 cpp_drv_cmn:FC73000+B926C cpp_drv_cmn:FC73000+B9624 :10000000+6E50 :10000000+9188 cpp_drv_cmn:FC73000+643EC binos:F77C000+9BCC cpp_drv_cmn:FC73000

*Jan 13 01:09:20.767: %CPPDRV-3-FATAL_MEM_CLEAR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6117]: CPP0: (0x6e) Failed to clear memory 'cpp/driver' detected the 'warning' condition 'CPP driver error in devobj layer': Connection timed out -Traceback= 1#939a1e943669c32003ea84676fda1830 errmsg:F4DB000+2160 cpp_drv_cmn:FC73000+61D60 cpp_drv_cmn:FC73000+B937C cpp_drv_cmn:FC73000+B9624 :10000000+6E50 :10000000+9188 cpp_drv_cmn:FC73000+643EC binos:F77C000+9BCC cpp_drv_cmn:FC73000+64298 :10000000+88D4 :100

*Jan 13 01:09:20.768: %CPPHA-3-FAULT: F0: cpp_ha: CPP:0 desc:DPE2_CPE_CPE_DPE_INT_SET_0_LEAF_INT_INT_PHY_ERROR det:DRVR(interrupt) class:OTHER sev:FATAL id:950 cppstate:STOPPED res:UNKNOWN flags:0x3 cdmflags:0x5

*Jan 13 01:09:20.768: %CPPHA-3-FAULTCRASH: F0: cpp_ha: CPP 0 unresolved fault detected, initiating crash dump.

*Jan 13 01:09:20.778: %CPPHA-3-FAULT: F0: cpp_ha: CPP:0 desc:DPE2_CPE_CPE_DPE_DUI_LEAF_INT_INT_DUI_CHN0_DRAM_MBE det:DRVR(interrupt) class:MBE sev:FATAL id:979 cppstate:STOPPED res:SUCCESS flags:0x3 cdmflags:0x5

*Jan 13 01:09:20.778: %CPPHA-3-FAULTCRASH: F0: cpp_ha: CPP 0 unresolved fault detected, initiating crash dump.

*Jan 13 01:09:20.791: %CPPDRV-3-FATAL_INIT_ID: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6117]: CPP0: (0x1) Failed to initialize init device drivers(s) - Operation not permitted -Traceback= 1#939a1e943669c32003ea84676fda1830 errmsg:F4DB000+2160 cpp_drv_cmn:FC73000+62008 :10000000+6E74 :10000000+9188 cpp_drv_cmn:FC73000+643EC binos:F77C000+9BCC cpp_drv_cmn:FC73000+64298 :10000000+88D4 :10000000+85CC cpp_client_ha:FC11000+2F08 cpp_common_os:FB1E000+10A54 cpp_common_os:FB1E000+1108C evlib:F835000+D8A

*Jan 13 01:09:20.799: %CPPHA-3-INITFAIL: F0: cpp_ha: CPP 0 initialization failed - startup init (0x1)

*Jan 13 01:09:20.799: %CPPHA-3-INITFAIL: F0: cpp_ha: CPP 0 initialization failed - start CPP (0x1)

*Jan 13 01:09:20.803: %CPPOSLIB-3-ERROR_NOTIFY: F0: cpp_ha: cpp_ha encountered an error -Traceback= 1#4d2666c969d0680445c0a2b0d833ea64 errmsg:F853000+2160 cpp_common_os:FF5D000+BB20 cpp_common_os:FF5D000+B930 cpp_common_os:FF5D000+19010 cpp_drv_cmn:FB6F000+73040 cpp_dmap:FEFC000+17D68 cpp_dmap:FEFC000+1B688 :10000000+1AC80 :10000000+1789C :10000000+DAC0 cpp_tm:FFDC000+23E8 cpp_tm:FFDC000+284C :10000000+DE24 cpp_tm:FFDC000+23E8 :10000000+C080 cpp_tm:FFDC000+23E8 :10000000+DF8C cpp_tm:FFDC000+23E8 cpp_tm:FFD

*Jan 13 01:09:20.805: %CPPHA-3-FAULT: F0: cpp_ha: CPP:0 desc:Platform Collection Error: Message STARTUP_INIT to client cpp_driver0 failed with error [Operation not permitted] det:HA class:CLIENT_SW sev:FATAL id:2 cppstate:RUNNING res:UNKNOWN flags:0x0 cdmflags:0x0

*Jan 13 01:09:20.824: %IOSXE-3-PLATFORM: F0: cpp_cdm: CPP crashed, core file /tmp/corelink/ESP_0_cpp-mcplo-ucode_011312010920.core.gz

*Jan 13 01:09:34.325: %IOSXE-6-PLATFORM: F0: cpp_cdm: Shutting down CPP MDM while client(s) still connected

*Jan 13 01:09:34.334: %CPPHA-3-CDMDONE: F0: cpp_ha: CPP 0 microcode crashdump creation completed.

*Jan 13 01:09:34.335: %IOSXE-6-PLATFORM: F0: cpp_ha: Shutting down CPP MDM while client(s) still connected

*Jan 13 01:09:34.336: %IOSXE-6-PLATFORM: F0: cpp_ha: Shutting down CPP CDM while client(s) still connected

*Jan 13 01:09:34.420: %PMAN-3-PROCHOLDDOWN: F0: pman.sh: The process cpp_ha_top_level_server has been helddown (rc 69)

*Jan 13 01:09:34.424: %PMAN-3-PROCHOLDDOWN: F0: pman.sh: The process cpp_cdm_svr has been helddown (rc 69)

*Jan 13 01:09:39.464: %ASR1000_OIR-6-OFFLINECARD: Card (fp) offline in slot F0

F0 ASR1000-ESP20 init, active 00:17:48

Slot: F0, ASR1000-ESP20

Running state : unknown

Internal state : offline

Internal operational state : unknown

Physical insert detect time : 00:01:28 (00:18:02 ago)

Software declared up time : 00:18:45 (00:00:45 ago)

Hardware ready signal time : 00:00:00 (never ago)

Packet ready signal time : 00:00:00 (never ago)

CPLD version : 08041102

Firmware version : 15.0(1r)S

Anyone seen this before ?

Leo Laohoo · ‎01-12-2012

ASR is rebooting because it's crashing. Not sure if the cause is an IOS bug or a hardware fault.

Can you find the crashdump or crashinfo file? If you can, please post them?

If you can't, upgrade to a different IOS and try again.

V_V · ‎01-22-2012

Hi Leolaohoo,

Thanks for your response.

Its been determined that it is a hardware fault

Regards,

VV

pganesha · ‎07-18-2016

F0 is the ESP module. As stated, it keeps restarting due to multibit parity error.

Please reseat the ESP. If log errors stop, then this is just a transient event.

If errors continue to reoccur, a hardware replacement is required.