Re: unresolved fault detected on Cisco ASR1002-X

dominiaz · ‎01-21-2024

I have a problem with new firmwares on ASR 1002-X.

Affected versions:

Cupertino-17.9.4a
Bengaluru-17.6.5a

Not affected versions:

Bengaluru-17.6.5

2024-01-19T17:03:33.872847+01:00 10.254.254.6 308: Jan 19 17:03:33: %IOSXE_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
2024-01-19T17:03:33.876560+01:00 10.254.254.6 309: Jan 19 17:03:33: %CPPHA-3-FAULT: F0/0: cpp_ha_top_level_server: CPP:0.0 desc:CGI_CSR32_CGI_SETB_HIER_INT__INT_PA det:DRVR(interrupt) class:OTHER sev:FATAL id:85 cppstate:RUNNING res:UNKNOWN flags:0x7 cdmflags:0x8
2024-01-19T17:03:33.876560+01:00 10.254.254.6 310: Jan 19 17:03:33: %CPPOSLIB-3-ERROR_NOTIFY: F0/0: cpp_ha_top_level_server: cpp_ha encountered an error -Traceback= 1#11a2cd5cf38f19f1c5ee156d9c04025f  errmsg:7FAE9634C000+E8A cpp_common_os:7FAE9CAB7000+1E26C cpp_common_os:7FAE9CAB7000+140BE cpp_drv_cmn:7FAE9C4BA000+29ABF :557F7BCF5000+30D93 :557F7BCF5000+3096D :557F7BCF5000+305EA :557F7BCF5000+28D45 :557F7BCF5000+27D3C cpp_common_os:7FAE9CAB7000+22111 cpp_common_os:7FAE9CAB7000+227DE evlib:7FAE998D0000+8E16 evlib:7FAE998D0000+9B60 cpp_common_os:7FAE9CAB7000+241F2 :557F7BCF5
2024-01-19T17:03:33.876560+01:00 10.254.254.6 311: Jan 19 17:03:33: %CPPHA-3-FAULTCRASH: F0/0: cpp_ha_top_level_server: CPP 0.0 unresolved fault detected, initiating crash dump.
2024-01-19T17:03:33.876560+01:00 10.254.254.6 312: Jan 19 17:03:33: %CPPHA-3-FAULTCRASH: F0/0: cpp_ha_top_level_server: CPP 0.0 unresolved fault detected, initiating crash dump.
2024-01-19T17:03:33.876560+01:00 10.254.254.6 313: Jan 19 17:03:33: %IOSXE-3-PLATFORM: R0/0: cpp_cdm: CPP crashed, collecting state.
2024-01-19T17:03:33.876560+01:00 10.254.254.6 314: Jan 19 17:03:33: %CPPDRV-6-INTR: F0/0: cpp_driver: Yoda(0) Interrupt : 24-Jan-19 17:03:33.468965 UTC+0100:CGI_CSR32_CGI_SETB_HIER_INT__INT_PA
2024-01-19T17:03:34.875074+01:00 10.254.254.6 315: Jan 19 17:03:33: %CPPDRV-3-LOCKDOWN: F0/0: cpp_cp_svr: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
2024-01-19T17:03:34.875074+01:00 10.254.254.6 316: Jan 19 17:03:33: %CPPOSLIB-3-ERROR_NOTIFY: F0/0: cpp_cp_svr: cpp_cp encountered an error -Traceback= 1#2ca28244486577f71e6ddfd1d901cd90  errmsg:7FC7ECDA4000+E8A cpp_common_os:7FC7F1EE2000+1E26C cpp_common_os:7FC7F1EE2000+140BE cpp_icmp_svr:7FC7F420C000+19563 cpp_icmp_svr:7FC7F420C000+13945 cpp_icmp_svr:7FC7F420C000+1747D cpp_common_os:7FC7F1EE2000+22111 cpp_common_os:7FC7F1EE2000+227DE evlib:7FC7F0328000+8E16 evlib:7FC7F0328000+9B60 cpp_common_os:7FC7F1EE2000+241F2 :562443832000+304EF c:7FC7E6C5C000+26E60 :562443832000+274
2024-01-19T17:03:36.663715+01:00 10.254.254.6 317: Jan 19 17:03:35: %CPPDRV-3-LOCKDOWN: F0/0: cpp_ha_top_level_server: QFP0.0 CPP Driver LOCKDOWN encountered due to previous fatal error (HW: QFP interrupt).
2024-01-19T17:03:36.663715+01:00 10.254.254.6 318: Jan 19 17:03:35: %CPPOSLIB-3-ERROR_NOTIFY: F0/0: fman_fp_image: fman_fp encountered an error -Traceback= 1#5d4dc199a84a0633cc1bc613f00343bb  errmsg:7F2D698E5000+E8A cpp_common_os:7F2D8796A000+1E26C cpp_client_ha:7F2D876EC000+290F cpp_common_os:7F2D8796A000+22111 cpp_common_os:7F2D8796A000+227DE evlib:7F2D443AE000+8E16 evlib:7F2D443AE000+9B60 :55803417E000+8471AF :55803417E000+710BD4 :55803417E000+984A58 :55803417E000+983E41 :55803417E000+983CC8 :55803417E000+983C41 c:7F2D3D4DB000+26E60 :55803417E000+1DB72A

I am attaching core log file:

https://biuro.alfaline.pl/index.php/s/BGfrnPn4L8s6QLm/download/ASR-R2_RP_0-system-report_20240119-170443-CET.tar.gz

It is happening every 6 to 12 hours causing router crash and full reload.

Is it any known bug? What could be related to this?

P.S. 17.6.5a provides only fix for CSCwh87343 Cisco IOS XE Software Web UI Privilege Escalation Vulnerability". It is very strange that I don't see that bug on version 17.6.5.

balaji.bandi · ‎01-21-2024

It is happening every 6 to 12 hours causing router crash and full reload.
Is it any known bug? What could be related to this?

as of now not known since what causing this issue on your environment

Suggest to Open a TAC case as this is effecting your environment - If you are not able to downgrade to 17.6.5.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

dominiaz · ‎01-22-2024

It's no problem for me to use 17.6.5, but I am shocked that Cisco ASR 1000 is so unstable device.

Unfortunately I can't open TAC case because I don't have a support for 2024.

dominiaz · ‎01-23-2024

I have found a bug in many IOS-XE versions (eg. 17.6.5a and 17.9.4) caused router reload every couple of hours.

You cannot use too many static CGNATs:

ip nat pool NAT-45.148.43.1 45.148.43.1 45.148.43.1 prefix-length 24
ip nat pool NAT-45.148.43.2 45.148.43.2 45.148.43.2 prefix-length 24
...
ip nat pool NAT-45.148.43.254 45.148.43.254 45.148.43.254 prefix-length 24

ip nat inside source list NAT-172.16.0.0/27 pool NAT-45.148.43.1 overload pap
ip nat inside source list NAT-172.16.0.32/27 pool NAT-45.148.43.2 overload pap
...
ip nat inside source list NAT-172.19.7.96/27 pool NAT-45.148.43.254 overload pap

ip access-list standard NAT-172.16.0.0/27
 10 permit 172.16.0.0 0.0.0.31
ip access-list standard NAT-172.16.0.32/27
  10 permit 172.16.0.32 0.0.0.31
...
ip access-list standard NAT-172.19.7.96/27
 10 permit 172.19.7.96 0.0.0.31

Workaround:

ip nat pool CGNAT_POOL_1 45.148.43.1 45.148.43.254 prefix-length 24
ip nat inside source list ACL-CLIENTS-CGNAT-1 pool CGNAT_POOL_1 overload pap
ip access-list standard ACL-CLIENTS-CGNAT-1
  permit 172.16.0.0 0.15.255.255

Can someone send TAC case with that issue?