ASR9k - Failed to boot the local MBI

Nuno Marques · ‎07-31-2013

Hi,

while upgrading from 4.0.1 to 4.2.3 on a several dozens 9ks network, im experiencing this issue ocasionally.

It seems that MBI file is not found , the router starts a tftp internal download for the image and the syncs all the old and new packages for about an hour,

and ends up booting the right image successfully.

This has issue was detected only on two of the devices, and all of them have exactly the same sw releases, hw, bootflash and disk sizes, etc.

Cisco Doc explanation:

%INSTALL-INSTHELPER-6-INCORRECT_MBI This node was booted with an incorrect MBI ([chars]), rebooting self with correct boot image ([chars]).

Explanation The dSC node was booted with an incorrect MBI. There are a few cases where this can occur:

1) User manually boots an MBI that is different from the MBI that RP should boot.

2) User manually sets BOOT ROMMON variable that is different from the MBI that RP should boot.

3) An install causing an MBI change without 'install commit' will have the BOOT ROMMON variable set to MBI from uncommitted software set if the dSC is subsequently rebooted.

Recommended Action: The node will reboot to pick up the new mbi. If the node ends in ROMMON (as the config-register was not set to auto-boot) then it is recommended to simply set the config-register to autoboot and reset to boot the correct MBI.

Its pretty straight forward, but in our case none of issues seems to map to our current enviorment ( dont know for sure if previous release was installed commited )

Anyone had experienced this problem?

Follows the relevante error output:

Cisco IOS XR Software for the Cisco XR ASR9K, Version 4.2.3

00:08:11.959 : Install (Node Preparation): Initializing VS Distributor...
00:08:34.504 : Install (Node Preparation): Booting with software activated by previous install operation
Insthelper encountered a fatal error condition, and is exiting: Operation (Failed to boot the local MBI.), Error value = (2), Error string = (No such file or directory)
00:09:22.876 : Install (Node Preparation): Initializing VS Distributor...
00:09:32.984 : Install (Node Preparation): Booting with committed software
Node 0/RSP0/CPU0: insthelper: This node was booted with an incorrect MBI (/bootflash/disk0/asr9k-os-mbi-4.2.3/0x100000/mbiasr9k-rp.vm), rebooting self with correct boot image (/bootflash/disk0/asr9k-os-mbi-4.0.1.CSCtk03031-1.0.0/mbiasr9k-rp.vm).Spawning instdir_show_ltrace -r failed. : No such file or directory
Spawning instdir_show_ltrace -Z -r failed. : No such file or directory
Spawning inst_rds_show_ltrace -r failed. : No such file or directory
[0x40d457503] Record Reboot History, reboot cause = 0x4000048, descr = Cause: dSC booted with incorrect MBI

.....<removed>......

Remote image to boot : tftp:/disk0/asr9k-os-mbi-4.2.3/0x100000/mbiasr9k-rp.vm

IP_ADDRESS: 127.0.1.4

IP_SUBNET_MASK: 255.255.0.0

DEFAULT_GATEWAY: 127.0.1.5

TFTP_SERVER: 127.0.1.5

TFTP_FILE: /disk0/asr9k-os-mbi-4.2.3/0x100000/mbiasr9k-rp.vm

Performing tftpdnld
tsec_init_hw: configuring FE (port 2) for: Auto Speed, Auto Duplex
tsec_init_interface: hardware initialization completed

Receiving /disk0/asr9k-os-mbi-4.2.3/0x100000/mbiasr9k-rp.vm from 127.0.1.5 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
tftp_process_packet: last packet, block=41694, size=444.
File reception completed.

Cisco IOS XR Software for the Cisco XR ASR9K, Version 4.2.3

00:15:34.798 : Install (Node Preparation): Initializing VS Distributor...
00:15:38.987 : Install (Node Preparation): Install device root from dSC is /disk0/
00:15:38.998 : Install (Node Preparation): Trying device disk0:
00:15:53.046 : Install (Node Preparation): Checking size of device disk0:
00:15:53.055 : Install (Node Preparation):    OK
00:15:53.056 : Install (Node Preparation): Cleaning packages on device disk0:
00:15:53.058 : Install (Node Preparation):    Please wait...
00:19:07.883 : Install (Node Preparation):    Complete
00:19:07.885 : Install (Node Preparation): Cleaning files on MBI device bootflash:
00:19:07.887 : Install (Node Preparation):    Please Wait...
00:19:08.657 : Install (Node Preparation):    Complete
00:19:08.660 : Install (Node Preparation): Checking free space on disk0:
00:19:08.662 : Install (Node Preparation):    OK
00:19:08.663 : Install (Node Preparation): Checking free space on MBI device bootflash:
00:19:08.665 : Install (Node Preparation):    OK
00:19:08.668 : Install (Node Preparation): Starting package and meta-data sync
00:19:08.685 : Install (Node Preparation): Syncing package/meta-data contents: /disk0/asr9k-p-4.2.3.CSCud54093-1.0.0
00:19:08.687 : Install (Node Preparation):    Please wait...
00:19:08.867 : Install (Node Preparation):    Completed syncing: /disk0/asr9k-p-4.2.3.CSCud54093-1.0.0
00:19:08.870 : Install (Node Preparation): Syncing package/meta-data contents: /disk0/asr9k-base-4.2.3.CSCud54093-1.0.0
00:19:08.872 : Install (Node Preparation):    Please wait...
00:19:14.616 : Install (Node Preparation):    Completed syncing: /disk0/asr9k-base-4.2.3.CSCud54093-1.0.0
00:19:14.619 : Install (Node Preparation): Syncing package/meta-data contents: /disk0/asr9k-os-mbi-4.2.3.CSCud54093-1.0.0
00:19:14.621 : Install (Node Preparation):    Please wait...
00:19:27.974 : Install (Node Preparation):    Completed syncing: /disk0/asr9k-os-mbi-4.2.3.CSCud54093-1.0.0
00:19:27.976 : Install (Node Preparation): Syncing package/meta-data contents: /disk0/asr9k-p-4.2.3.CSCub96985-1.0.0
00:19:27.978 : Install (Node Preparation):    Please wait...

Thanks in advance.

smilstea · ‎07-31-2013

Hi,

As the Cisco Doc says there are a few things this could be, so I cannot tell without checking the system before the upgrade to see what could have happened. Also as you mentioned you were not sure if the install was committed; briefly on this topic we do store multiple MBIs on bootflash:(Typically the active, committed, and an upgrade/inactive one)

However, fortunately for us when a node goes to boot (in this case the RSP you are viewing) we do have an MBI check to verify what code we need to load. This is good for several scenarios such as upgrades, reloading / installing a new card, and rack bringup in nV to name a few. When this check was made the issue was corrected.

Did you follow the 4.2.3 upgrade guide, loading mandatory SMUs, system checks, etc?

This is very important for several reasons, such as possible package issues by not having the mandatory SMUs loaded before the 'install activate' of the new code.

HTH,

Sam

Nuno Marques · ‎08-01-2013

Hi Samuel,

thanks for the reply, yes the procedure was follow by the book and it as been tested performed on several dozens of devices, later analisys also revealed that the install commit from the previous release had been done.

So aparently none of the reasons for the message description applies to this specific case.

Regards.

smilstea · ‎08-01-2013

Hi,

Unfortunately from this information we cannot provide you with more information on why this happened. It may be possible to get more information on why this happened via a TAC case as several long outputs will need to be analyzed, however it may not be possible to get more information on the cause of the issue as many of the logs and relevant data are lost when the router reloads due to the 'install activate' operation.

The good news as I mentioned previously is that the issue was resolved and the router is on 4.2.3

Thanks,

Sam