ASR1001: Crash/reload, 3 times in a row and then stable

[noc@ipeva.fr]

Hi all,

Last January 21st, one of my ASR1001 crashed/reloaded with the following error:

Jan 21 22:29:43 10.10.10.248 11451: ASR1.EQNX-PA3: Jan 21 22:29:42.492: %IOSXE_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
Jan 21 22:29:57 10.10.10.248 11452: ASR1.EQNX-PA3: Jan 21 22:29:57.664: %CPPHA-3-FAULT: SIP0: cpp_ha: CPP:0.0 desc:CPP Client process failed: cpp_cp det:HA class:CLIENT_SW sev:FATAL id:1 cppstate:RUNNING res:UNKNOWN flags:0x0 cdmflags:0x0
Jan 21 22:29:57 10.10.10.248 11453: ASR1.EQNX-PA3: Jan 21 22:29:57.668: %IOSXE-6-PLATFORM: SIP0: cpp_ha: Shutting down CPP MDM while client(s) still connected
Jan 21 22:29:57 10.10.10.248 11454: ASR1.EQNX-PA3: Jan 21 22:29:57.678: %CPPOSLIB-3-ERROR_NOTIFY: SIP0: fman_fp_image: fman-fp encountered an error -Traceback= 1#7dbd6c29adccce9f8bf0691214205f54 errmsg:7F57C7A5B000+121D cpp_common_os:7F57AF00D000+E005 cpp_common_os:7F57AF00D000+DF04 cpp_common_os:7F57AF00D000+1A02E cpp_cent_lib:7F57A386C000+4948 cpp_common_os:7F57AF00D000+122F6 cpp_common_os:7F57AF00D000+11D5B evlib:7F57B33B0000+B8E7 evlib:7F57B33B0000+E1B0 :400000+45D5CF c:7F579FD07000+1E514 :400000+178089
Jan 21 22:29:57 10.10.10.248 11455: ASR1.EQNX-PA3: Jan 21 22:29:57.738: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The process cpp_ha_top_level_server has been helddown (rc 69)
Jan 21 22:29:58 10.10.10.248 11456: ASR1.EQNX-PA3: Jan 21 22:29:57.798: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The process cpp_cp_svr has been helddown (rc 134)
Jan 21 22:29:58 10.10.10.248 11457: ASR1.EQNX-PA3: Jan 21 22:29:57.910: %CPPDRV-3-LOCKDOWN: SIP0: fman_fp_image: CPP10(0) CPP Driver LOCKDOWN due to fatal error.

Then, it was stable until this evening (exactly 60 days after, minus 1h30). This time, it crashed/reloaded 3 times in 75 minutes:

Mar 21 20:48:29 10.10.10.248 1246: ASR1.EQNX-PA3: Mar 21 20:48:28.004: %IOSXE_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
Mar 21 20:48:42 10.10.10.248 1247: ASR1.EQNX-PA3: Mar 21 20:48:42.699: %CPPHA-3-FAULT: SIP0: cpp_ha: CPP:0.0 desc:CPP Client process failed: cpp_cp det:HA class:CLIENT_SW sev:FATAL id:1 cppstate:RUNNING res:UNKNOWN flags:0x0 cdmflags:0x0
Mar 21 20:48:42 10.10.10.248 1248: ASR1.EQNX-PA3: Mar 21 20:48:42.702: %IOSXE-6-PLATFORM: SIP0: cpp_ha: Shutting down CPP MDM while client(s) still connected
Mar 21 20:48:42 10.10.10.248 1249: ASR1.EQNX-PA3: Mar 21 20:48:42.714: %CPPOSLIB-3-ERROR_NOTIFY: SIP0: fman_fp_ima
ge: fman-fp encountered an error -Traceback= 1#7dbd6c29adccce9f8bf0691214205f54 errmsg:7FEB42A6D000+121D cpp_co
mmon_os:7FEB2A01F000+E005 cpp_common_os:7FEB2A01F000+DF04 cpp_common_os:7FEB2A01F000+1A02E cpp_cent_lib:7FEB1E87E0
00+4948 cpp_common_os:7FEB2A01F000+122F6 cpp_common_os:7FEB2A01F000+11D5B evlib:7FEB2E3C2000+B8E7 evlib:7FEB2E3C20
00+E1B0 :400000+45D5CF c:7FEB1AD19000+1E514 :400000+178089
Mar 21 20:48:42 10.10.10.248 1250: ASR1.EQNX-PA3: Mar 21 20:48:42.768: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The p
rocess cpp_ha_top_level_server has been helddown (rc 69)
Mar 21 20:48:43 10.10.10.248 1251: ASR1.EQNX-PA3: Mar 21 20:48:42.782: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The p
rocess cpp_cp_svr has been helddown (rc 134)
Mar 21 20:48:43 10.10.10.248 1252: ASR1.EQNX-PA3: Mar 21 20:48:43.090: %CPPDRV-3-LOCKDOWN: SIP0: fman_fp_image: C
PP10(0) CPP Driver LOCKDOWN due to fatal error.

Mar 21 21:49:17 10.10.10.248 135: ASR1.EQNX-PA3: Mar 21 21:49:16.464: %IOSXE_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
Mar 21 21:49:31 10.10.10.248 136: ASR1.EQNX-PA3: Mar 21 21:49:31.170: %CPPHA-3-FAULT: SIP0: cpp_ha: CPP:0.0 desc:CPP Client process failed: cpp_cp det:HA class:CLIENT_SW sev:FATAL id:1 cppstate:RUNNING res:UNKNOWN flags:0x0 cdmflags:0x0
Mar 21 21:49:31 10.10.10.248 137: ASR1.EQNX-PA3: Mar 21 21:49:31.172: %IOSXE-6-PLATFORM: SIP0: cpp_ha: Shutting down CPP MDM while client(s) still connected
Mar 21 21:49:31 10.10.10.248 138: ASR1.EQNX-PA3: Mar 21 21:49:31.189: %CPPOSLIB-3-ERROR_NOTIFY: SIP0: fman_fp_image: fman-fp encountered an error -Traceback= 1#7dbd6c29adccce9f8bf0691214205f54 errmsg:7F491EBD6000+121D cpp_common_os:7F4906188000+E005 cpp_common_os:7F4906188000+DF04 cpp_common_os:7F4906188000+1A02E cpp_cent_lib:7F48FA9E7000+4948 cpp_common_os:7F4906188000+122F6 cpp_common_os:7F4906188000+11D5B evlib:7F490A52B000+B8E7 evlib:7F490A52B000+E1B0 :400000+45D5CF c:7F48F6E82000+1E514 :400000+178089
Mar 21 21:49:31 10.10.10.248 139: ASR1.EQNX-PA3: Mar 21 21:49:31.214: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The process cpp_cp_svr has been helddown (rc 134)
Mar 21 21:49:31 10.10.10.248 140: ASR1.EQNX-PA3: Mar 21 21:49:31.251: %PMAN-0-PROCFAILCRIT: SIP0: pvp.sh: A critical process cpp_cp_svr has failed (rc 134)
Mar 21 21:49:32 10.10.10.248 141: ASR1.EQNX-PA3: Mar 21 21:49:31.285: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The process cpp_ha_top_level_server has been helddown (rc 69)
Mar 21 21:49:32 10.10.10.248 142: ASR1.EQNX-PA3: Mar 21 21:49:31.579: %CPPDRV-3-LOCKDOWN: SIP0: fman_fp_image: CPP10(0) CPP Driver LOCKDOWN due to fatal error.

Mar 21 22:13:27 10.10.10.248 135: ASR1.EQNX-PA3: Mar 21 22:13:26.197: %IOSXE_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
Mar 21 22:13:40 10.10.10.248 136: ASR1.EQNX-PA3: Mar 21 22:13:40.949: %CPPHA-3-FAULT: SIP0: cpp_ha: CPP:0.0 desc:CPP Client process failed: cpp_cp det:HA class:CLIENT_SW sev:FATAL id:1 cppstate:RUNNING res:UNKNOWN flags:0x0 cdmflags:0x0
Mar 21 22:13:40 10.10.10.248 137: ASR1.EQNX-PA3: Mar 21 22:13:40.951: %IOSXE-6-PLATFORM: SIP0: cpp_ha: Shutting down CPP MDM while client(s) still connected
Mar 21 22:13:41 10.10.10.248 138: ASR1.EQNX-PA3: Mar 21 22:13:40.963: %CPPOSLIB-3-ERROR_NOTIFY: SIP0: fman_fp_image: fman-fp encountered an error -Traceback= 1#7dbd6c29adccce9f8bf0691214205f54 errmsg:7FA7218A3000+121D cpp_common_os:7FA708E55000+E005 cpp_common_os:7FA708E55000+DF04 cpp_common_os:7FA708E55000+1A02E cpp_cent_lib:7FA6FD6B4000+4948 cpp_common_os:7FA708E55000+122F6 cpp_common_os:7FA708E55000+11D5B evlib:7FA70D1F8000+B8E7 evlib:7FA70D1F8000+E1B0 :400000+45D5CF c:7FA6F9B4F000+1E514 :400000+178089
Mar 21 22:13:41 10.10.10.248 139: ASR1.EQNX-PA3: Mar 21 22:13:41.015: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The process cpp_ha_top_level_server has been helddown (rc 69)
Mar 21 22:13:42 10.10.10.248 140: ASR1.EQNX-PA3: Mar 21 22:13:41.048: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The process cpp_cp_svr has been helddown (rc 134)
Mar 21 22:13:42 10.10.10.248 141: ASR1.EQNX-PA3: Mar 21 22:13:41.134: %CPPDRV-3-LOCKDOWN: SIP0: fman_fp_image: CPP10(0) CPP Driver LOCKDOWN due to fatal error.

As I hardly doubt that some piece of hardware could die progressively, I wonder if that crash could come from some kind of abnormal traffic (regular or not) coming from a customer or from the outside.

Thank you for any input you may provide.

David

[Leo Laohoo]

What firmware is the router on?

[noc@ipeva.fr]

Leo,

3.11.0S

i know it’s very old but you know, it has been stable for years…

David

[Leo Laohoo · Accepted solution]

---

noc@ipeva.fr wrote:
3.11.0S

---

(Using) A firmware that ends with a "0" is a dangerous gamble.  

Mar 21 21:49:32 10.10.10.248 142: ASR1.EQNX-PA3: Mar 21 21:49:31.579: %CPPDRV-3-LOCKDOWN: SIP0: fman_fp_image: CPP10(0) CPP Driver LOCKDOWN due to fatal error.

CSCte19641

Mar 21 22:13:41 10.10.10.248 139: ASR1.EQNX-PA3: Mar 21 22:13:41.015: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The process cpp_ha_top_level_server has been helddown (rc 69)

CSCun09640, CSCwb22555

Mar 21 22:13:42 10.10.10.248 140: ASR1.EQNX-PA3: Mar 21 22:13:41.048: %PMAN-3-PROCHOLDDOWN: SIP0: pman.sh: The process cpp_cp_svr has been helddown (rc 134)

CSCur69193

At the very least, has anyone seen this before:  Multiple Vulnerabilities in Cisco IOS XE Software Web UI Feature

Take the security bulletin seriously because it is (still!) currently and aggressively exploited in the wild.

[noc@ipeva.fr]

Leo,

Ok I must admit I was afraid of a such answer.

It has been working fine for years, but after all, that's the definition of a bug: it works and then not.

I will plan an upgrade to 3.16.5S.

Thank you

David

[Leo Laohoo]

Cisco ISR & ASR 1k Routers: IOS-XE/Firmware Upgrade (Install Mode)