cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3485
Views
0
Helpful
10
Replies

ASR 1002-X keep rebooting itself

Hello;

Am facing an issue where my Cisco ASR 1002-X keeps rebooting itself at random time. When i run the show version, i can see the reason for reload is: critical process fault, fman_fp_image, fp_0_0, rc=139

 

On my syslog server, i keep getting this error: %IOSXE-3-PLATFORM: SIP0: cpp_cp: QFP:0.0 Thread:171 TS:00000041045846946120 %IPSEC-3-HMAC_ERROR: IPSec SA receives HMAC error, 

 

I dont know if that could be the reason of my router reload or if it's an IOS bug, am running asr1002x-universalk9.03.16.04b.S.155-3.S4b-ext.SPA.bin.

 

Your help will be highly appreciated.

10 Replies 10

Leo Laohoo
Hall of Fame
Hall of Fame

@armand.ndayikeza1 wrote:

reason for reload is: critical process fault, fman_fp_image, fp_0_0, rc=139


If this is the case, then post the complete output to the command "dir" and "sh crashinfo".

here is the output of dir
My_1002x#dir
Directory of bootflash:/

11 drwx 16384 May 14 2015 08:02:29 +02:00 lost+found
804801 drwx 4096 Sep 7 2017 21:11:12 +02:00 .prst_sync
12 -rw- 412555136 May 14 2015 08:04:38 +02:00 asr1002x-universalk9.03.11.00.S.154-1.S-std.SPA.bin
482881 drwx 4096 May 14 2015 08:06:34 +02:00 .installer
740417 drwx 4096 Sep 7 2017 21:08:02 +02:00 core
676033 drwx 4096 May 14 2015 08:33:03 +02:00 .rollback_timer
13 -rw- 0 May 14 2015 08:33:43 +02:00 tracelogs.758
177057 drwx 20480 Sep 8 2017 06:06:59 +02:00 tracelogs
14 -rw- 463824772 Jan 24 2017 04:53:14 +02:00 asr1002x-universalk9.03.16.04b.S.155-3.S4b-ext.SPA.bin
225345 drwx 4096 Jan 24 2017 05:25:30 +02:00 virtual-instance
15 -rw- 1427 Jan 24 2017 05:25:49 +02:00 octeon_pci_console_dev0_20170124_032547.log
16 -rw- 30 Sep 7 2017 21:11:25 +02:00 throughput_monitor_params
17 -rw- 1427 Mar 27 2017 08:12:34 +02:00 octeon_pci_console_dev0_20170327_061232.log
18 -rw- 1531 Mar 27 2017 08:30:43 +02:00 octeon_pci_console_dev0_20170327_063041.log
19 -rw- 1155 Mar 27 2017 09:43:09 +02:00 ASR1.lic
--More--   20 -rw- 1427 Mar 27 2017 09:47:15 +02:00 octeon_pci_console_dev0_20170327_074713.log
21 -rw- 1427 Mar 29 2017 08:38:45 +02:00 octeon_pci_console_dev0_20170329_063843.log
22 -rw- 1427 Mar 30 2017 08:03:05 +02:00 octeon_pci_console_dev0_20170330_060303.log
23 -rw- 1427 Apr 2 2017 19:14:27 +02:00 octeon_pci_console_dev0_20170402_171425.log
24 -rw- 1140 May 28 2017 17:06:58 +02:00 ipsec.lic
25 -rw- 1427 May 28 2017 17:23:04 +02:00 octeon_pci_console_dev0_20170528_152302.log
26 -rw- 1427 Sep 5 2017 07:08:11 +02:00 octeon_pci_console_dev0_20170905_050809.log
27 -rw- 1427 Sep 5 2017 18:11:58 +02:00 octeon_pci_console_dev0_20170905_161156.log
28 -rw- 1427 Sep 7 2017 21:11:15 +02:00 octeon_pci_console_dev0_20170907_191113.log

6741659648 bytes total (5437456384 bytes free)

For your information, the command show crashinfo is not supported

There something in the coredump folder. Can you post the complete output to the command "dir bootflash:/core"?

My_1002x#dir bootflash:/core
Directory of bootflash:/core/

740420 -rw- 3222271 Dec 22 2016 08:22:30 +02:00 Router_RP_0_cpp_sp_svr_19697_1482387741.core.gz
740427 -rw- 1 Sep 8 2017 15:41:31 +02:00 .callhome
740428 -rw- 10568630 Sep 5 2017 07:05:01 +02:00 My_1002x_RP_0_fman_fp_image_32231_1504587883.core.gz
740419 -rw- 2408035 Dec 22 2016 08:22:31 +02:00 Router_RP_0_cpp_ha_top_leve_19575_1482387741.core.gz
740422 -rw- 9834508 Dec 22 2016 08:22:34 +02:00 Router_RP_0_cpp_cp_svr_18964_1482387742.core.gz
740423 -rw- 4956579 Dec 22 2016 08:22:39 +02:00 Router_RP_0_fman_fp_image_20430_1482387751.core.gz
740418 -rw- 2702658 Dec 22 2016 08:22:25 +02:00 Router_RP_0_cpp_driver_19365_1482387741.core.gz
740421 -rw- 5445433 Dec 22 2016 08:22:31 +02:00 Router_RP_0_cpp_stats_svr_19842_1482387741.core.gz
740424 -rw- 438833 Dec 22 2016 08:22:35 +02:00 Router_RP_0_cmand_11575_1482387755.core.gz
740425 -rw- 4978728 Dec 22 2016 08:26:53 +02:00 Router_RP_0_fman_fp_image_20333_1482388004.core.gz
740426 -rw- 4977101 Dec 22 2016 08:33:32 +02:00 Router_RP_0_fman_fp_image_20551_1482388404.core.gz
740429 -rw- 9047675 Sep 5 2017 18:08:49 +02:00 My_1002x_RP_0_fman_fp_image_31833_1504627712.core.gz
740430 -rw- 9408456 Sep 7 2017 21:08:02 +02:00 My_1002x_RP_0_fman_fp_image_31906_1504811265.core.gz

6741659648 bytes total (5437456384 bytes free)

Copy those files to an external location and send off to TAC.

Here is the logs i get before the router restarts itself.

 

 

2017-09-14 13:36:09 local3.alert #.#.#.#  265: *Sep 14 11:37:38.240: %IOSXE-1-PLATFORM: R0/0: kernel: QFP0.0: Fatal Fault: SW reported: Userspace fault

 

 

 

2017-09-14 12:54:42 local3.emergency #.#.#.#  296: *Sep 14 10:56:11.216: %PMAN-0-PROCFAILCRIT: SIP0: pvp.sh:  A critical process fman_fp_image has failed (rc 139)

 

2017-09-14 11:43:18 local3.emergency #.#.#.#  2738: *Sep 14 09:44:47.530: %PMAN-0-PROCFAILCRIT:

 SIP0: pvp.sh:  A critical process fman_fp_image has failed (rc 139)

 

2017-09-14 11:43:18 local3.error #.#.#.#  2737: *Sep 14 09:44:47.436: %PMAN-3-PROCHOLDDOWN:

SIP0: pman.sh:  The process cpp_ha_top_level_server has been helddown (rc 69)

 

2017-09-14 11:43:18 local3.error #.#.#.#  2736: *Sep 14 09:44:47.421: %PMAN-3-PROCHOLDDOWN:

 SIP0: pman.sh:  The process fman_fp_image has been helddown (rc 139)

 

2017-09-14 11:43:18 local3.alert #.#.#.#  2735: *Sep 14 09:44:47.323: %IOSXE-1-PLATFORM: R0/0: kernel:

QFP0.0: Fatal Fault: SW reported: Userspace fault

 

2017-09-14 11:43:18 local3.error #.#.#.#  2734: *Sep 14 09:44:47.295: %CPPDRV-3-LOCKDOWN_INITIATED:

SIP0: cpp_ha:  QFP0.0 CPP Driver LOCKDOWN being triggered due to fatal error.

2017-09-14 11:43:18 local3.error #.#.#.#  2733: *Sep 14 09:44:47.294: %CPPOSLIB-3-ERROR_NOTIFY:

 SIP0: cpp_ha:  cpp_ha encountered an error -Traceback= 1#ff47c47aa400ae1e261b0532453d046f   errmsg:

7FB57B7B4000+121D cpp_common_os:7FB57F2C4000+DA3C cpp_common_os:7FB57F2C4000+1B5AE cpp_drv_cmn:

7FB57EAEF000+285B7 :400000+24438 :400000+23C1A :400000+14617 :400000+C0DE :400000+129AD :400000+F9E6

 :400000+13B62 cpp_common_os:7FB57F2C4000+1257F evlib:7FB57A72E000+B937 evlib:7FB57A72E000+E200

cpp_common_os:7FB57F2C4000+13E42 :400000+DA85 c:7FB572FA1000+1E514 :400000+83E

 

2017-09-14 11:43:18 local3.error #.#.#.#  2732: *Sep 14 09:44:47.294: %CPPHA-3-FAULT: SIP0: cpp_ha:

 CPP:0.0 desc:CPP Client process failed: FMAN-FP det:HA class:CLIENT_SW sev:FATAL id:1 cppstate:RUNNING

 res:UNKNOWN flags:0x0 cdmflags:0x0

 

2017-09-14 11:43:10 local3.error #.#.#.#  2731: *Sep 14 09:44:38.944: %SSH-3-RSA_SIGN_FAIL:

Signature creation failed, status 21

 

2017-09-14 11:43:10 local3.error #.#.#.#  2730: *Sep 14 09:44:38.944: %SSH-3-RSA_SIGN_FAIL:

Signature connection failed, status 3

 

2017-09-14 11:43:09 local3.error #.#.#.#  2729: *Sep 14 09:44:38.209: %SSH-3-RSA_SIGN_FAIL:

Signature creation failed, status 21

2017-09-14 11:43:09 local3.error #.#.#.#  2728: *Sep 14 09:44:38.209: %SSH-3-RSA_SIGN_FAIL: Signature

 connection failed, status 3

2017-09-14 11:43:09 local3.error #.#.#.#  2727: *Sep 14 09:44:37.473: %CRYPTO-3-IKE_PAK_IN_Q_TIME_LIMIT

_EXCEED: Pak spent too much time in the IKE input queues

 

2017-09-14 11:43:09 local3.error #.#.#.#  2726: *Sep 14 09:44:37.470: %ACE-3-TRANSTO: IOSXE-ESP(14):

 IKEA trans 0x23E7; opcode 0x23; param 0x0; too many retries

 

2017-09-14 11:43:06 local3.error #.#.#.#  2725: *Sep 14 09:44:35.712: %BGP-3-NOTIFICATION:

sent to neighbor 154.70.156.109 4/0 (hold time expired) 0 bytes

2017-09-14 11:43:04 local3.error #.#.#.#  2724: *Sep 14 09:44:33.664: %BGP-3-NOTIFICATION:

sent to neighbor 154.70.198.2 4/0 (hold time expired) 0 bytes

 

2017-09-14 11:43:01 local3.warning #.#.#.#  2723: *Sep 14 09:44:29.179: %IOSXE-4-PLATFORM:

R0/0: kernel: copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:43:00 local3.warning #.#.#.#  2722: *Sep 14 09:44:28.178: %IOSXE-4-PLATFORM: R0/0:

 kernel: copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:43:00 local3.warning #.#.#.#  2721: *Sep 14 09:44:28.178: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:43:00 local3.warning #.#.#.#  2720: *Sep 14 09:44:28.178: %IOSXE-4-PLATFORM: R0/0:

 kernel: copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:43:00 local3.warning #.#.#.#  2719: *Sep 14 09:44:28.178: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:42:59 local3.error #.#.#.#  2718: *Sep 14 09:44:28.178: %ACE-3-TRANSERR: IOSXE-ESP(14):

IKEA trans 0x23E4; opcode 0x23; param 0x0; error 0xA; retry cnt 3

 

2017-09-14 11:42:59 local3.error #.#.#.#  2717: *Sep 14 09:44:28.178: %ACE-3-TRANSERR: IOSXE-ESP(14):

 IKEA trans 0x23E3; opcode 0x23; param 0x0; error 0xA; retry cnt 3

2017-09-14 11:42:59 local3.warning #.#.#.#  2716: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0: kernel:

 copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:42:59 local3.warning #.#.#.#  2715: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:42:59 local3.warning #.#.#.#  2714: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0:

 kernel: copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:42:59 local3.warning #.#.#.#  2713: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0:

kernel: copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:42:59 local3.warning #.#.#.#  2712: *Sep 14 09:44:27.176: %IOSXE-4-PLATFORM: R0/0: kernel:

 copy_user_buffer: No copy done, status: 3 (0x3)

 

2017-09-14 11:42:56 local3.error #.#.#.#  2711: *Sep 14 09:44:25.173: %ACE-3-TRANSERR: IOSXE-ESP(14):

IKEA trans 0x23E2; opcode 0x23; param 0x0; error 0xA; retry cnt 2

 

 

 

 Any idea how to solve this.

Traceback could mean either a hardware issue or a software bug. Upgrade the IOS and see if there are any improvements.

Hello,

there are numerous bugs, but they are all configuration specific. Can you post your configuration ?

I have a huge configuration of around 1500 lines,,,, which kind of configuration can be causing those bugs...

Configuration does not cause any bugs.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco