Re: Line Card(A9K-24X10GE-1G-TR) not booting. Showing Error "IPU secure boot failed"

hellopavil1 · ‎09-22-2019

Hi,

We are facing issue with ASR 9904 line card. After power on line card status led is amber only and showing below error

RP/0/RSP0/CPU0:mes-ngl-pbx-pbx-cdi-1#0/RSP0/ADMIN0:Sep 19 15:14:28.524 UTC: canbus_driver[3923]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC0 in slot 0/0, alarm code CBC_ALRM_IPU_SECURE_BOOT_FAILED

0/RSP0/ADMIN0:Sep 19 15:14:28.526 UTC: shelf_mgr[3958]: %INFRA-SHELF_MGR-3-HW_FAILURE_EVENT : HW failure event HW_EVENT_FAILURE, event_reason_str 'IPU secure boot failed' for card 0/0

0/RSP0/ADMIN0:Sep 19 15:14:28.526 UTC: shelf_mgr[3958]: %INFRA-SHELF_MGR-3-CARD_SW_FAILED : Card: 0/0 software state going to FAILED

0/RSP0/ADMIN0:Sep 19 15:14:28.526 UTC: shelf_mgr[3958]: %INFRA-SHELF_MGR-3-CARD_HW_FAILED : Card: 0/0 hardware state going to FAILED

Software Version : asr9k-xr-6.5.3 version=6.5.3 [Boot image]

0/0-IDPROM - 24X10G/1G Packet Transport Optimized LC
Info
Controller Family : 003d
Controller Type : 02cb
PID : A9K-24X10GE-1G-TR
Version Identifier : V01
UDI Description : 24X10G/1G Packet Transport Optimized LC
Top Assy. Part Number : 68-6156-02
Top Assy. Revision : F0
PCB Serial Number : FOC2328NSHS
PCA Number : 73-17904-02
CLEI Code : IP9IAZNCAA
Capabilities : 00
sysadmin-vm:0_RSP0# sh hw ow hw-module platform
Thu Sep 19 16:26:29.166 UTC
Location Card Type HW State SW State Config State
----------------------------------------------------------------------------
0/0 A9K-24X10GE-1G-TR PRESENT SW_INACTIVE NSHUT
0/RSP0 A9K-RSP880-TR OPERATIONAL OPERATIONAL NSHUT
0/RSP1 A9K-RSP880-TR OPERATIONAL OPERATIONAL NSHUT
0/FT0 ASR-9904-FAN OPERATIONAL N/A NSHUT
0/PT0 A9K-DC-PEM-V2 OPERATIONAL N/A NSHUT

smilstea · ‎09-22-2019

What version of code were you running before?

Were the FPDs up to date on the old code?

Did you do fpd auto-upgrade when upgrading to 6.5.3?

What FPD versions are currently loaded on the LC?

Was this card working in old code but not working now or is it newly inserted card?

Sam

hellopavil1 · ‎09-24-2019

We were upgrading from 6.4.2 to 6.5.3 and suddenly power failed. After that this issue started. When we power up, RSP0 was not booting. I choose boot from previous boot and it worked, after i upgraded to 6.5.3 and both RSP was fine. But line card was not booting. It was showing IPU_SECURE_BOOT_FAILED.

What version of code were you running before?

6.4.2 64-bit

Were the FPDs up to date on the old code?

yes

Did you do fpd auto-upgrade when upgrading to 6.5.3?

yes

What FPD versions are currently loaded on the LC?

Now its not showing, The problem is its not booting

Was this card working in old code but not working now or is it newly inserted card?

Its factory fit card. It was working before.

smilstea · ‎09-24-2019

In 6.4.2 what BIOS version do you have?

From admin mode run the below command.

show hw-module location <lc>

Thanks,

Sam

smilstea · ‎09-24-2019

Can you also try this?

Available workarounds:
1) Disable the fpd auto-upgrade

2) Bridge SMU of CSCvb12358 in V1 affected release.
6.0.x, 6.1.x, 6.2.1, 6.2.3

3) Day-one smu of CSCvp46026 in the V2 release
6.5.2, 6.5.3, 6.6.1 and 6.6.2

Since you have 6.4.2 we would need to upgrade to 6.5.3 with CSCvp46026 SMU at the same time, recommend disabling fpd auto-upgrade too.

Thanks,

Sam

hellopavil1 · ‎10-19-2019

Hi,

We cannot open the bug page. Its showing like below

*Insufficient Permissions to View Bug

This bug contains proprietary information and is not yet publicly available.

You may find useful information within the Cisco Community*

Second thing currently both RSP is in 6.5.3 version. Only Line card have problem.

smilstea · ‎10-19-2019

Can you try installing the SMU for CSCvp46026?

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvp46026

Sam

hellopavil1 · ‎10-19-2019

Hi,

Still its saying same this thing

*Insufficient Permissions to View Bug

This bug contains proprietary information and is not yet publicly available.

You may find useful information within the Cisco Community*

I believe only Cisco employees can see It. Not yet open for others. Please see the attached image.

Kindly provide the solution as snapshot or text. Thanks in advance

smilstea · ‎10-19-2019

I'll have to work on making it visible. For now here are the release notes.

Symptom:
A99-12x100GE,A99-12x100GE-CM, A9K-24X10GE-1G-SE, A9K-24X10GE-1G-TR, A9K-24X10GE-1G-CM, A9K-48X10GE-1G-SE, A9K-48X10GE-1G-TR, A9K-48X10GE-1G-CM shows up as FAILED in 64 bit OS in release 652/653/661/662.

Conditions:
When performing system upgrade from 32 bit OS release 602/611/612/613/614/621/622 to 32 Bit OS release 652/653/661/662 followed by migration to 64-Bit OS, below mentioned PIDs ends up in failed state

PID: A99-12x100GE,A99-12x100GE-CM, A9K-24X10GE-1G-SE, A9K-24X10GE-1G-TR, A9K-24X10GE-1G-CM, A9K-48X10GE-1G-SE, A9K-48X10GE-1G-TR, A9K-48X10GE-1G-CM.

Workaround:
A SMU will be available in target eXR release shortly.

Available workarounds:
1) Disable the fpd auto-upgrade

2) Bridge SMU of CSCvb12358 in V1 affected release.
6.0.x, 6.1.x, 6.2.1, 6.2.3

3) Day-one smu of CSCvp46026 in the V2 release
6.5.2, 6.5.3, 6.6.1 and 6.6.2

4) Still use auto FPD upgrade

Shutdown Skyhammer / Powerglide LCs
Configure auto fpd upgrade
Perform 6.1.4 to 6.5.3 upgrade
Disable auto fpd upgrade
Re-enable Skyhammer/Powerglide LCs
Perform manual FPD upgrades on Skyhammer/Powerglide LCs and reload Skyhammer/Powerglide LCs

Yongyu Jia · ‎04-07-2020

Hi,

Just FYI, both DDTS are visible now.

Thanks,