A9K MOD80 Cannot Power up

[Rps-Cheers]

HI Experts

I am meeting a strange issue of ASR9K MOD80. This MOD80 Card cannot power up, and always output log as below:

shelfmgr[385]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/1/CPU0 A9K-MOD80-SE state:MBI-RUNNING 
shelfmgr[385]: %PLATFORM-SHELFMGR-6-NODE_KERNEL_DUMP_EVENT : Node 0/1/CPU0 indicates it is doing a kernel dump. 
shelfmgr[385]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/1/CPU0 A9K-MOD80-SE state:IOS XR FAILURE 
shelfmgr[385]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/1/CPU0 A9K-MOD80-SE state:BRINGDOWN 
invmgr[249]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/1/CPU0, state: BRINGDOWN
canb-server[151]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/1/CPU0 , Power Cycle (0x05000000)
canb-server[151]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/1/CPU0 , Power Cycle (0x05000000) 
...

I have installed the Card to another A9K ( which version is same as the A9K) , and the MOD80 card can boot up and IOS XR RUN.

Did you have occur this issue ? Please help me to check it. many thks.

Attachment logging after MOD80 OIR collect..

[viweyam774]

Are you Looking for Waikiki Health Portal? If you want to Access Waikiki Health Portal Official webpage then here you can see the official Waikiki Health Portal link. You can access all…  https://e-healthportal.org/

[LaloRam]

Hi @Rps-Cheers,

this LC is having kernel crashes

shelfmgr[385]: %PLATFORM-SHELFMGR-6-NODE_KERNEL_DUMP_EVENT : Node 0/1/CPU0 indicates it is doing a kernel dump.

Open a TAC case to get these dumps decoded.

Eduardo

[Rps-Cheers]

Yes, i have noticed this crash event and check it. I found there are some crashinfo ... i will try to get it.

I feel most strange that is the LC Insert to another A9K, it will run well. But it cannot PWR UP if i insert it to this A9K chassis any slot.

[LaloRam]

this could be related to the differences between both ASR 9000s. Can you please share the show platform and show install active summary of both ASR 9000?

Eduardo

[Rps-Cheers]

• Same Chassis
• Same software version
• Same install active (including Active Packages counts)
• Same RSP (A9K-RSP440-TR)
• Same MOD Card and MPA card .. ( A9K-MOD80-SE/A9K-MOD80-TR and A9K-MPA-4X10GE/A9K-MPA-20X1GE ..)

[LaloRam]

hi @Rps-Cheers 

What about FPD version on both ASRs?

Eduardo,

[Rps-Cheers]

Normal one:

                                        HW                       Current SW Upg/
Location     Card Type                Version Type Subtype Inst   Version   Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/RSP0/CPU0  A9K-RSP440-TR              1.0   lc   cbc     0      16.116    No 
                                              lc   fpga1   0       0.10     No 
                                              lc   fpga3   0       4.09     No 
                                              lc   fpga2   0       1.06     Yes
                                              lc   rommon  0       0.73     No 
--------------------------------------------------------------------------------
0/0/CPU0     A9K-MOD80-TR               1.0   lc   cbc     0      20.118    No 
                                              lc   fpga2   0       1.04     No 
                                              lc   fpga4   0       1.05     No 
                                              lc   rommon  0       3.02     Yes
--------------------------------------------------------------------------------
0/2/CPU0     A9K-MOD80-SE               1.0   lc   cbc     0      20.118    No 
                                              lc   fpga2   0       1.04     No 
                                              lc   fpga4   0       1.05     No 
                                              lc   rommon  0       3.03     Yes
--------------------------------------------------------------------------------

<omit>

Abnormal one

                                        HW                       Current SW Upg/
Location     Card Type                Version Type Subtype Inst   Version   Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/RSP0/CPU0  A9K-RSP440-TR              1.0   lc   cbc     0      16.116    No 
                                              lc   fpga3   0       4.09     No 
                                              lc   fpga2   0       1.06     Yes
                                              lc   fpga1   0       0.10     No 
                                              lc   rommon  0       0.73     No 
--------------------------------------------------------------------------------
0/0/CPU0     A9K-MOD80-SE               1.0   lc   cbc     0      20.118    No 
                                              lc   fpga2   0       1.04     No 
                                              lc   fpga4   0       1.05     No 
                                              lc   rommon  0       3.03     Yes
--------------------------------------------------------------------------------
0/1/CPU0     A9K-MOD80-TR               1.0   lc   cbc     0      20.118    No 
                                              lc   fpga4   0       1.05     No 
                                              lc   fpga2   0       1.04     No 
                                              lc   rommon  0       3.02     Yes
--------------------------------------------------------------------------------
<omit>

[LaloRam]

FPDs seem to be ok in both setups, I recommend you to open a case with TAC to get these dumps decoded. It will be very useful to share this info with the TAC engineer.

Eduardo

[Rps-Cheers]

well, i will. Many thxs.

[DUNG · Accepted solution]

I had the same problem.LC tested ok on another ASR9000. But it can't PWR UP any slot on this A9K chassis .I try redundancy switchover, and then LC can boot up.

shelfmgr[427]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/5/CPU0 A9K-MOD80-SE state:MBI-RUNNING 
shelfmgr[427]: %PLATFORM-SHELFMGR-6-NODE_KERNEL_DUMP_EVENT : Node 0/6/CPU0 indicates it is doing a kernel dump.
shelfmgr[427]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/6/CPU0 A9K-MOD80-SE state:IOS XR FAILURE
shelfmgr[427]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/6/CPU0 A9K-MOD80-SE state:BRINGDOWN
invmgr[264]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/6/CPU0, state: BRINGDOWN
canb-server[158]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/6/CPU0 , Power Cycle (0x05000000)
canb-server[158]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/6/CPU0 , Power Cycle (0x05000000)

ASR9010 with RSP440-TR

[smilstea]

So to answer you and the original poster (OP) there are many reasons why the card may not transition to XR RUN from MBI RUNNING, in short the main reason is for lack of an XR heartbeat message. That can be due to the control ethernet (CE) or also known as EOBC dropping the packet, XR VM not sending the heartbeat, etc etc. There was a bug way back that was super difficult to detect that it could be CSCuv65231. It affects all version before 5.3.3. It is caused by a slow leak in the mqueue process. Your best bet as the bug says is to connect to the console of the LC as its booting to see what is going on, you can do that on the asr9k with  attachcon.

RP/0/RSP1/CPU0:ASR9006-B#run attachCon 0/0/cpu0
Wed Aug 29 14:02:41.699 UTC

attachCon: Starting console session to node 0/0/cpu0
attachCon: To quit console session type 'detach'
Current Baud 115200
Setting Baud to 9600

ksh: e: not found
#

In the above example the card is already up and running in IOS-XR and this takes me to the ksh of the linecard.

Below is of a linecard starting to boot up normally.

Selecting ROMMON Image... B
DDR in Interleaved mode
POST 1 : PASSED : code 0 : DDR2 Memory

System Bootstrap, Version 1.03(20100211:014208) [ASR9K ROMMON],
Copyright (c) 1994-2010 by Cisco Systems, Inc.

Compiled Wed 10-Feb-10 17:42 by

CPU Reset Reason = 0x0005
PPC 8641D (partnum 0x0003), Revision 0.2, (Core Version 2.20136)
M8641 CLKIN: 66 Mhz
Core Clock: 1333 Mhz
MPX Clock: 533 Mhz
LBC Clock: 33 Mhz

Daughter Board Present: yes
Daughter Board ID: 7
Memory Option: 1
Board Type: 40221
POST 3 : PASSED : code 0 : Slot ID/Board Type

Loading Field Programmable Devices:
FPGA 0-B PROGRAMMED : image: 0xfd800028 - 0xfe1dbb28, et: 480ms
FPGA 1-B PROGRAMMED : image: 0xfd800028 - 0xfe1dbb28, et: 930ms
FPGA 2-B PROGRAMMED : image: 0xc0300028 - 0xc039f270, et: 29ms

Main Board: 0x40221, rev 0x3
PLD: 1.2
Bridge0: 1.2
Bridge1: 1.2
CBC: 6.2
Daughter Board: 0x7, rev 0x2, bom 0x2
DB Main PLD: 1.3
DB PLD: 0.8
IO FPGA: 0.11

PCI-E1: Ready as Root Complex
PCI-E2: Ready as Root Complex

ASR9K (8641D PPC) platform with 4096 Mb of main memory

CARD_SLOT_NUMBER: 2
CPU_INSTANCE: 1
MBI Validation starts ...
tsec_init_hw: configuring TSEC (port 1) for: 1GB, Full Duplex

tsec_init_interface: hardware initialization completed
Interface link changed state to UP.
Interface link state up.

MBI validation sending request.
HIT CTRL-C to abort
.
mbi_val_process_packet: received repsonse
Remote image to boot : tftp:/disk0/asr9k-os-mbi-4.2.1/lc/mbiasr9k-lc.vm

IP_ADDRESS: 127.0.1.2
IP_SUBNET_MASK: 255.255.0.0
DEFAULT_GATEWAY: 127.0.1.1
TFTP_SERVER: 127.0.1.1
TFTP_FILE: /disk0/asr9k-os-mbi-4.2.1/lc/mbiasr9k-lc.vm

Performing tftpdnld
tsec_init_hw: configuring TSEC (port 1) for: 1GB, Full Duplex

Sam

[DUNG]

Hi @smilstea 

Our device is running 6.2.3.I want to know what it is LC HW problem or software bug.It doesn't hit bug CSCuv65231.Thanks!

RP/0/RSP1/CPU0:ASR9010#run attachCon 0/7/CPU0

Sat Aug 20 14:56:26.275 GMT

attachCon: Starting console session to node 0/7/CPU0
attachCon: To quit console session type 'detach'
attachCon: WARNING - Type only 'detach' in the shell. Do not combine other keys.
Current Baud 115200
Setting Baud to 9600
;C?System Bootstrap, Version 3.3(20150930:043930) [ASR9K ROMMON],

Copyright (c) 1994-2015 by Cisco Systems, Inc.

 

Compiled Wed 30-Sep-15 10:09 by vgothe

 

Starting Initialization of FMAN0

Loading ucode for FMAN0, size: 6992, ver: 101.08.00

Starting Initialization of FMAN1

Loading ucode for FMAN1, size: 6992, ver: 101.08.00

Initializing CPU serdes for rev2...

Serdes Protocol: 0xE

Internal Board type detected: 0x3

 

CPU Reset Reason = 0x0005

Initializing Zarlink(25M mode)...

Initializing MB ASICs...

Initializing Sacramento...

Initializing GIGE SW...

Initializing DB ASICs...

PPC P40X0 (partnum 0x8023), Revision 02.00, (Core Version 02.00)

 

Sys Clock: 100 Mhz

Core Clock: 1500 Mhz

Plat Clock: 800 Mhz

LBC Clock: 25 Mhz

MEM Clock: 1300 Mhz

 

 

Board Type: 0x2a0254

Memory Option: 0x3

CPU Board => Rev: 0x7

=========

CPU PLD: 01.05.00

CBC: 20.118

 

Secondary Board => Rev: 0x3

===============

DB PLD: 01.04 ()

 

PCI-E1: is enabled

PCI-E2: is enabled

PCI-E3: is enabled

 

Applying IDT PCIE SW Errata1 WA

ASR9K (P40X0 PPC) platform with 8192 Mb of main memory

 

 

CARD_SLOT_NUMBER: 9

CPU_INSTANCE: 1

MBI Validation starts ...

dtsec_init_hw: configuring DTSEC (port for: 1GB, Full Duplex

Interface link changed state to UP.

Interface link state up.

 

MBI validation sending request.

HIT CTRL-C to abort

 

mbi_val_process_packet: received repsonse (rack 0)

Remote image to boot : tftp:/disk0/asr9k-os-mbi-6.2.3.CSCvg74808-1.0.0/lc/mbiasr9k-lc.vm

 

IP_ADDRESS: 127.0.1.9

IP_SUBNET_MASK: 255.255.0.0

DEFAULT_GATEWAY: 127.0.1.5

TFTP_SERVER: 127.0.1.5

TFTP_FILE: /disk0/asr9k-os-mbi-6.2.3.CSCvg74808-1.0.0/lc/mbiasr9k-lc.vm

 

Performing tftpdnld

dtsec_init_hw: configuring DTSEC (port for: 1GB, Full Duplex

 

tftp_process_packet: received OACK.

tftp_process_packet: Change blksize to 1400.

Receiving /disk0/asr9k-os-mbi-6.2.3.CSCvg74808-1.0.0/lc/mbiasr9k-lc.vm from 127.0.1.5 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

tftp_process_packet: last packet, block=15556, size=1284.

File reception completed.

Image size=21778284, TFTP download size=21778284

....................................................................................

program load complete, entry point: 0x2037dc, size: 0x14c406c

Config = SMP, Running = SMP

Board type: 0x002a0254

Card Capability = 0x00000014

##################################################################################################################

BSP: Board type :XMEN-LC

tracelogger: starting tracing in background ring mode

tracelogger running with args: -startring -F 1 -F 2

 Restricted Rights Legend

 

Use, duplication, or disclosure by the Government is

subject to restrictions as set forth in subparagraph

(c) of the Commercial Computer Software - Restricted

Rights clause at FAR sec. 52.227-19 and subparagraph

(c) (1) (ii) of the Rights in Technical Data and Computer

Software clause at DFARS sec. 252.227-7013.

 

cisco Systems, Inc.

170 West Tasman Drive

San Jose, California 95134-1706

 

 

 

Cisco IOS XR Software for the Cisco XR ASR9K, Version 6.2.3

Copyright (c) 2019 by Cisco Systems, Inc.

# Aug 20 06:57:52.652: Install Setup: Preparing devices:

Aug 20 06:57:52.654: Install Setup: Complete

Aug 20 06:57:52.654: Install Setup: Starting package and meta-data sync

Aug 20 06:57:52.657: Install Setup: Syncing package 'asr9k-adv-video-supp-6.2.3':

Aug 20 06:57:52.814: Install Setup: Complete

Aug 20 06:57:52.815: Install Setup: Syncing package 'asr9k-base-6.2.3':

Aug 20 06:58:04.275: Install Setup: Complete

Aug 20 06:58:04.286: Install Setup: Syncing package 'asr9k-base-6.2.3.CSCvi90822-1.0.0':

Aug 20 06:58:04.606: Install Setup: Complete

Aug 20 06:58:04.606: Install Setup: Syncing package 'asr9k-base-6.2.3.CSCvj43752-1.0.0':

Aug 20 06:58:04.926: Install Setup: Complete

Aug 20 06:58:04.926: Install Setup: Syncing package 'asr9k-base-6.2.3.CSCvj55087-1.0.0':

Aug 20 06:58:04.984: Install Setup: Complete

Aug 20 06:58:04.984: Install Setup: Syncing package 'asr9k-base-6.2.3.CSCvn41916-1.0.0':

Aug 20 06:58:05.551: Install Setup: Complete

Aug 20 06:58:05.552: Install Setup: Syncing package 'asr9k-base-6.2.3.CSCvn51108-1.0.0':

Aug 20 06:58:08.688: Install Setup: Complete

Aug 20 06:58:08.692: Install Setup: Syncing package 'asr9k-bng-6.2.3':

Aug 20 06:58:08.777: Install Setup: Complete

Aug 20 06:58:08.778: Install Setup: Syncing package 'asr9k-ce-6.2.3':

Aug 20 06:58:08.929: Install Setup: Complete

Aug 20 06:58:08.930: Install Setup: Syncing package 'asr9k-cpp-6.2.3':

Aug 20 06:58:09.093: Install Setup: Complete

Aug 20 06:58:09.094: Install Setup: Syncing package 'asr9k-cpp-6.2.3.CSCvo06271-1.0.0':

Aug 20 06:58:09.152: Install Setup: Complete

Aug 20 06:58:09.153: Install Setup: Syncing package 'asr9k-diags-supp-6.2.3':

PID 40977: _dl_abort. libinst.dll:dllmgr: Dll not in proper format libinst.dll [/pkg/lib/libinst.dll] err_str [No such file or directory]

 

 

Process 49192 (instsetup) terminated SIGSEGV code=1 fltno=11 ip=4a440b68 ref=00000001 bdslot=1

Aug 20 06:58:12.568: Install Setup: Preparing devices:

Aug 20 06:58:12.571: Install Setup: Complete

Aug 20 06:58:12.571: Install Setup: Starting package and meta-data sync

Aug 20 06:58:13.269: Install Setup: Syncing package 'asr9k-diags-supp-6.2.3':

PID 94225: _dl_abort. libinst.dll:dllmgr: Dll not in proper format libinst.dll [/pkg/lib/libinst.dll] err_str [No such file or directory]

 

 

Process 90152 (instsetup) terminated SIGSEGV code=1 fltno=11 ip=4a440b68 ref=00000001

Aug 20 06:58:17.038: Install Setup: Preparing devices:

Aug 20 06:58:17.040: Install Setup: Complete

Aug 20 06:58:17.040: Install Setup: Starting package and meta-data sync

Aug 20 06:58:17.741: Install Setup: Syncing package 'asr9k-diags-supp-6.2.3':

PID 102417: _dl_abort. libinst.dll:dllmgr: Dll not in proper format libinst.dll [/pkg/lib/libinst.dll] err_str [No such file or directory]

 

 

Process 98344 (instsetup) terminated SIGSEGV code=1 fltno=11 ip=4a440b68 ref=00000001

Aug 20 06:58:21.477: Install Setup: Preparing devices:

Aug 20 06:58:21.480: Install Setup: Complete

Aug 20 06:58:21.480: Install Setup: Starting package and meta-data sync

KD: Copied LC7.220820-065546.crashinfo.by.init to /lcdisk0:/dumper

Aug 20 06:58:22.351: Install Setup: Syncing package 'asr9k-diags-supp-6.2.3':

KD: Copied LC7.220820-065546.pcds to /lcdisk0:/dumper

PID 110609: _dl_abort. libinst.dll:dllmgr: Dll not in proper format libinst.dll [/pkg/lib/libinst.dll] err_str [No such file or directory]

 

 

Process 106536 (instsetup) terminated SIGSEGV code=1 fltno=11 ip=4a440b68 ref=00000001

Aug 20 06:58:25.953: Install Setup: Preparing devices:

Aug 20 06:58:25.956: Install Setup: Complete

Aug 20 06:58:25.956: Install Setup: Starting package and meta-data sync

Aug 20 06:58:26.654: Install Setup: Syncing package 'asr9k-diags-supp-6.2.3':

PID 139281: _dl_abort. libinst.dll:dllmgr: Dll not in proper format libinst.dll [/pkg/lib/libinst.dll] err_str [No such file or directory]

 

 

Process 1352% Media type /nvram: doesn't exist in this node!

% Media type /configflash: doesn't exist in this node!

07 (instsetup) terminated SISuccessed to rename debug file, 2, src: /bootflash:/sysmgr.log.timeout.Z, target: /bootflash:/prev.sysmgr.log.timeout.Z

Aug 20 06:58:30.491 : SYSMGR_LITE: Saving init logs in /bootflash:/sysmgr.log.timeout.Z ...

GSEGV code=1 fltno=11 ip=4a440b68 ref=0000Aug 20 06:58:30.633 : SYSMGR_LITE: INIT: respawn 'instsetup' disabled, exit_code 11, INIT_MAX_SPAWN reached

% Media type /nvram: doesn't exist in this node!

% Media type /configflash: doesn't exist in this node!

0001

Successed to rename debug file, 2, src: /bootflash:/sysmgr.log.timeout.Z, target: /bootflash:/prev.sysmgr.log.timeout.Z

Aug 20 06:58:31.003 : SYSMGR_LITE: Saving init logs in /bootflash:/sysmgr.log.timeout.Z ...

Aug 20 06:58:31.143 : SYSMGR_LITE: INIT: respawn 'dumper' disabled, exit_code 6, INIT_MAX_SPAWN reached

reboot internal : cause code 671088647 cause INIT: respawn 'instsetup' disabled, exit_code 11, INIT_MAX_SPAWN reached

reboot internal : cause code 671088647 cause INIT: respawn 'dumper' disabled, exit_code 6, INIT_MAX_SPAWN reached

reboot_internal timeout 30 is graceful no

reboWoriting crashinfo

Active processes:

proc/boot/procnto-booke-smp-instr Thread ID 0 on cpu 0

 

Active processes:

proc/boot/procnto-booke-smp-instr Thread ID 1 on cpu 1

 

Active processes:

asr9k-os-6.2.3.CSCvg74808-1.0.0/0x30207/bin/init Thread ID 5 on cpu 2

 

Active processes:

proc/boot/procnto-booke-smp-instr Thread ID 10 on cpu 3

? ?;C?System Bootstrap, Version 3.3(20150930:043930) [ASR9K ROMMON],

Copyright (c) 1994-2015 by Cisco Systems, Inc.

 

Compiled Wed 30-Sep-15 10:09 by vgothe

[Rps-Cheers]

I guess it's a software issue. As your meaning, it will recovery after RSP switchover, just as RSP reload..

[smilstea]

Hi,

Your logs are indicating corruption in the package diags. That is why the package fails to sync to the LC and the LC reboots.
Corruption can happen for a number of reasons, you can verify with an 'install verify'. You can only do a repair if the corruption exists on the standby, we don't want to mess with the active running packages on the active rsp for obvious reasons so the system doesn't allow to fix corruption on the active rsp. Even if you try a repair it might not work and then the only solution is to turboboot the router.

Sam