ASR 1002-X keep rebooting itself

armand.ndayikeza1 · ‎09-08-2017

Hello;

Am facing an issue where my Cisco ASR 1002-X keeps rebooting itself at random time. When i run the show version, i can see the reason for reload is: critical process fault, fman_fp_image, fp_0_0, rc=139

On my syslog server, i keep getting this error: %IOSXE-3-PLATFORM: SIP0: cpp_cp: QFP:0.0 Thread:171 TS:00000041045846946120 %IPSEC-3-HMAC_ERROR: IPSec SA receives HMAC error,

I dont know if that could be the reason of my router reload or if it's an IOS bug, am running asr1002x-universalk9.03.16.04b.S.155-3.S4b-ext.SPA.bin.

Your help will be highly appreciated.

Leo Laohoo · ‎09-08-2017

@armand.ndayikeza1 wrote:

reason for reload is: critical process fault, fman_fp_image, fp_0_0, rc=139

If this is the case, then post the complete output to the command "dir" and "sh crashinfo".

armand.ndayikeza1 · ‎09-08-2017

here is the output of dir
My_1002x#dir
Directory of bootflash:/

11 drwx 16384 May 14 2015 08:02:29 +02:00 lost+found
804801 drwx 4096 Sep 7 2017 21:11:12 +02:00 .prst_sync
12 -rw- 412555136 May 14 2015 08:04:38 +02:00 asr1002x-universalk9.03.11.00.S.154-1.S-std.SPA.bin
482881 drwx 4096 May 14 2015 08:06:34 +02:00 .installer
740417 drwx 4096 Sep 7 2017 21:08:02 +02:00 core
676033 drwx 4096 May 14 2015 08:33:03 +02:00 .rollback_timer
13 -rw- 0 May 14 2015 08:33:43 +02:00 tracelogs.758
177057 drwx 20480 Sep 8 2017 06:06:59 +02:00 tracelogs
14 -rw- 463824772 Jan 24 2017 04:53:14 +02:00 asr1002x-universalk9.03.16.04b.S.155-3.S4b-ext.SPA.bin
225345 drwx 4096 Jan 24 2017 05:25:30 +02:00 virtual-instance
15 -rw- 1427 Jan 24 2017 05:25:49 +02:00 octeon_pci_console_dev0_20170124_032547.log
16 -rw- 30 Sep 7 2017 21:11:25 +02:00 throughput_monitor_params
17 -rw- 1427 Mar 27 2017 08:12:34 +02:00 octeon_pci_console_dev0_20170327_061232.log
18 -rw- 1531 Mar 27 2017 08:30:43 +02:00 octeon_pci_console_dev0_20170327_063041.log
19 -rw- 1155 Mar 27 2017 09:43:09 +02:00 ASR1.lic
--More-- 20 -rw- 1427 Mar 27 2017 09:47:15 +02:00 octeon_pci_console_dev0_20170327_074713.log
21 -rw- 1427 Mar 29 2017 08:38:45 +02:00 octeon_pci_console_dev0_20170329_063843.log
22 -rw- 1427 Mar 30 2017 08:03:05 +02:00 octeon_pci_console_dev0_20170330_060303.log
23 -rw- 1427 Apr 2 2017 19:14:27 +02:00 octeon_pci_console_dev0_20170402_171425.log
24 -rw- 1140 May 28 2017 17:06:58 +02:00 ipsec.lic
25 -rw- 1427 May 28 2017 17:23:04 +02:00 octeon_pci_console_dev0_20170528_152302.log
26 -rw- 1427 Sep 5 2017 07:08:11 +02:00 octeon_pci_console_dev0_20170905_050809.log
27 -rw- 1427 Sep 5 2017 18:11:58 +02:00 octeon_pci_console_dev0_20170905_161156.log
28 -rw- 1427 Sep 7 2017 21:11:15 +02:00 octeon_pci_console_dev0_20170907_191113.log

6741659648 bytes total (5437456384 bytes free)

For your information, the command show crashinfo is not supported

Leo Laohoo · ‎09-08-2017

There something in the coredump folder. Can you post the complete output to the command "dir bootflash:/core"?

armand.ndayikeza1 · ‎09-08-2017

My_1002x#dir bootflash:/core
Directory of bootflash:/core/

740420 -rw- 3222271 Dec 22 2016 08:22:30 +02:00 Router_RP_0_cpp_sp_svr_19697_1482387741.core.gz
740427 -rw- 1 Sep 8 2017 15:41:31 +02:00 .callhome
740428 -rw- 10568630 Sep 5 2017 07:05:01 +02:00 My_1002x_RP_0_fman_fp_image_32231_1504587883.core.gz
740419 -rw- 2408035 Dec 22 2016 08:22:31 +02:00 Router_RP_0_cpp_ha_top_leve_19575_1482387741.core.gz
740422 -rw- 9834508 Dec 22 2016 08:22:34 +02:00 Router_RP_0_cpp_cp_svr_18964_1482387742.core.gz
740423 -rw- 4956579 Dec 22 2016 08:22:39 +02:00 Router_RP_0_fman_fp_image_20430_1482387751.core.gz
740418 -rw- 2702658 Dec 22 2016 08:22:25 +02:00 Router_RP_0_cpp_driver_19365_1482387741.core.gz
740421 -rw- 5445433 Dec 22 2016 08:22:31 +02:00 Router_RP_0_cpp_stats_svr_19842_1482387741.core.gz
740424 -rw- 438833 Dec 22 2016 08:22:35 +02:00 Router_RP_0_cmand_11575_1482387755.core.gz
740425 -rw- 4978728 Dec 22 2016 08:26:53 +02:00 Router_RP_0_fman_fp_image_20333_1482388004.core.gz
740426 -rw- 4977101 Dec 22 2016 08:33:32 +02:00 Router_RP_0_fman_fp_image_20551_1482388404.core.gz
740429 -rw- 9047675 Sep 5 2017 18:08:49 +02:00 My_1002x_RP_0_fman_fp_image_31833_1504627712.core.gz
740430 -rw- 9408456 Sep 7 2017 21:08:02 +02:00 My_1002x_RP_0_fman_fp_image_31906_1504811265.core.gz

6741659648 bytes total (5437456384 bytes free)

Leo Laohoo · ‎09-08-2017

Copy those files to an external location and send off to TAC.

armand.ndayikeza1 · ‎09-14-2017

Here is the logs i get before the router restarts itself.

2017-09-14 13:36:09 local3.alert #.#.#.# 265: *Sep 14 11:37:38.240: %IOSXE-1-PLATFORM: R0/0: kernel: QFP0.0: Fatal Fault: SW reported: Userspace fault

2017-09-14 12:54:42 local3.emergency #.#.#.# 296: *Sep 14 10:56:11.216: %PMAN-0-PROCFAILCRIT: SIP0: pvp.sh: A critical process fman_fp_image has failed (rc 139)

2017-09-14 11:43:18 local3.emergency #.#.#.# 2738: *Sep 14 09:44:47.530: %PMAN-0-PROCFAILCRIT:

SIP0: pvp.sh: A critical process fman_fp_image has failed (rc 139)

2017-09-14 11:43:18 local3.error #.#.#.# 2737: *Sep 14 09:44:47.436: %PMAN-3-PROCHOLDDOWN:

SIP0: pman.sh: The process cpp_ha_top_level_server has been helddown (rc 69)

2017-09-14 11:43:18 local3.error #.#.#.# 2736: *Sep 14 09:44:47.421: %PMAN-3-PROCHOLDDOWN:

SIP0: pman.sh: The process fman_fp_image has been helddown (rc 139)

2017-09-14 11:43:18 local3.alert #.#.#.# 2735: *Sep 14 09:44:47.323: %IOSXE-1-PLATFORM: R0/0: kernel:

QFP0.0: Fatal Fault: SW reported: Userspace fault

2017-09-14 11:43:18 local3.error #.#.#.# 2734: *Sep 14 09:44:47.295: %CPPDRV-3-LOCKDOWN_INITIATED:

SIP0: cpp_ha: QFP0.0 CPP Driver LOCKDOWN being triggered due to fatal error.

2017-09-14 11:43:18 local3.error #.#.#.# 2733: *Sep 14 09:44:47.294: %CPPOSLIB-3-ERROR_NOTIFY:

SIP0: cpp_ha: cpp_ha encountered an error -Traceback= 1#ff47c47aa400ae1e261b0532453d046f errmsg:

7FB57B7B4000+121D cpp_common_os:7FB57F2C4000+DA3C cpp_common_os:7FB57F2C4000+1B5AE cpp_drv_cmn:

7FB57EAEF000+285B7 :400000+24438 :400000+23C1A :400000+14617 :400000+C0DE :400000+129AD :400000+F9E6

:400000+13B62 cpp_common_os:7FB57F2C4000+1257F evlib:7FB57A72E000+B937 evlib:7FB57A72E000+E200

cpp_common_os:7FB57F2C4000+13E42 :400000+DA85 c:7FB572FA1000+1E514 :400000+83E

2017-09-14 11:43:18 local3.error #.#.#.# 2732: *Sep 14 09:44:47.294: %CPPHA-3-FAULT: SIP0: cpp_ha:

CPP:0.0 desc:CPP Client process failed: FMAN-FP det:HA class:CLIENT_SW sev:FATAL id:1 cppstate:RUNNING

res:UNKNOWN flags:0x0 cdmflags:0x0

2017-09-14 11:43:10 local3.error #.#.#.# 2731: *Sep 14 09:44:38.944: %SSH-3-RSA_SIGN_FAIL:

Signature creation failed, status 21

2017-09-14 11:43:10 local3.error #.#.#.# 2730: *Sep 14 09:44:38.944: %SSH-3-RSA_SIGN_FAIL:

Signature connection failed, status 3

2017-09-14 11:43:09 local3.error #.#.#.# 2729: *Sep 14 09:44:38.209: %SSH-3-RSA_SIGN_FAIL:

Signature creation failed, status 21

2017-09-14 11:43:09 local3.error #.#.#.# 2728: *Sep 14 09:44:38.209: %SSH-3-RSA_SIGN_FAIL: Signature

connection failed, status 3

2017-09-14 11:43:09 local3.error #.#.#.# 2727: *Sep 14 09:44:37.473: %CRYPTO-3-IKE_PAK_IN_Q_TIME_LIMIT

_EXCEED: Pak spent too much time in the IKE input queues

2017-09-14 11:43:09 local3.error #.#.#.# 2726: *Sep 14 09:44:37.470: %ACE-3-TRANSTO: IOSXE-ESP(14):

IKEA trans 0x23E7; opcode 0x23; param 0x0; too many retries

2017-09-14 11:43:06 local3.error #.#.#.# 2725: *Sep 14 09:44:35.712: %BGP-3-NOTIFICATION:

sent to neighbor 154.70.156.109 4/0 (hold time expired) 0 bytes

2017-09-14 11:43:04 local3.error #.#.#.# 2724: *Sep 14 09:44:33.664: %BGP-3-NOTIFICATION:

sent to neighbor 154.70.198.2 4/0 (hold time expired) 0 bytes

2017-09-14 11:43:01 local3.warning #.#.#.# 2723: *Sep 14 09:44:29.179: %IOSXE-4-PLATFORM:

R0/0: kernel: copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:43:00 local3.warning #.#.#.# 2722: *Sep 14 09:44:28.178: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:43:00 local3.warning #.#.#.# 2721: *Sep 14 09:44:28.178: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:43:00 local3.warning #.#.#.# 2720: *Sep 14 09:44:28.178: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:43:00 local3.warning #.#.#.# 2719: *Sep 14 09:44:28.178: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:42:59 local3.error #.#.#.# 2718: *Sep 14 09:44:28.178: %ACE-3-TRANSERR: IOSXE-ESP(14):

IKEA trans 0x23E4; opcode 0x23; param 0x0; error 0xA; retry cnt 3

2017-09-14 11:42:59 local3.error #.#.#.# 2717: *Sep 14 09:44:28.178: %ACE-3-TRANSERR: IOSXE-ESP(14):

IKEA trans 0x23E3; opcode 0x23; param 0x0; error 0xA; retry cnt 3

2017-09-14 11:42:59 local3.warning #.#.#.# 2716: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0: kernel:

copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:42:59 local3.warning #.#.#.# 2715: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:42:59 local3.warning #.#.#.# 2714: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:42:59 local3.warning #.#.#.# 2713: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:42:59 local3.warning #.#.#.# 2712: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0: kernel:

copy_user_buffer: No copy done, status: 3 (0x3)

2017-09-14 11:42:56 local3.error #.#.#.# 2711: *Sep 14 09:44:25.173: %ACE-3-TRANSERR: IOSXE-ESP(14):

IKEA trans 0x23E2; opcode 0x23; param 0x0; error 0xA; retry cnt 2

Any idea how to solve this.

Leo Laohoo · ‎09-14-2017

Traceback could mean either a hardware issue or a software bug. Upgrade the IOS and see if there are any improvements.

Georg Pauwen · ‎09-08-2017

Hello,

there are numerous bugs, but they are all configuration specific. Can you post your configuration ?

armand.ndayikeza1 · ‎09-08-2017

I have a huge configuration of around 1500 lines,,,, which kind of configuration can be causing those bugs...

mnmirza2011 · ‎05-20-2019

Configuration does not cause any bugs.