ASR9001 reboots itself frequently

[Mery]

Hello,

ASR9001 is rebooting itself continuously. The only thing that Ive notice so far is that the line card module stuck on MBI Running.

ASR9001-LC    MBI-RUNNING  

Also I see this log too 

wdsysmon[457]: %HA-HA_WD-4-DISK_WARN : A monitored device /disk0:/config/lr is above 80% utilization. Current utilization = 91. Please remove unwanted user files and configuration rollback points.

I rolled back the configurations but still the same issue. IT can stay up for a little bit and then starts the reloading

Anyone has a similar issue?

Thank you all

[smilstea]

I don't think the config size / rollback size is the issue but we can try clearing out the commitDB / rollback points to see if that helps.

'clear config commits oldest <x>'

Can you also paste the show install active summary and full show log here?

Thanks,

Sam

[Mery]

Hi and thank you for your reply. Actually, after rebooting and rebooting several times, now it is stuck in Rommon and not finding any image to fully reboot. I am trying to reboot it using an USB. What format should I use for that? I am using FAT32 and when I do:

boot disk0:asr9k-mini-px.pie-5.3.4

It is showing cant load the image 

[smilstea]

That is just the mini pie file, not the boot image.

On the RP issue the command 'more nvram:/classic-rommon-var' and see what the boot image is set to and try booting from that. Or from rommon you can try the 'set' command to see if its already set and just try 'boot'. Most likely you exceeded the number of reboot attempts and will have to manually reload the card from admin mode (hw-module loc 0/0/cpu0 reload) to allow the card to try reloading.

Sam

[Mery]

I cant enter anymore in terminal, just in Rommon and I dont see anything in set:

rommon B18 > set
PS1=rommon ! >
IOX_ADMIN_CONFIG_FILE=
ACTIVE_FCD=1
BSI=0
CLUSTER_NO_BOOT=
BOOT_DEV_SEQ_CONF=
BOOT_DEV_SEQ_OPER=
?=0

And it stucks here 

MBI validation sending request.
HIT CTRL-C to abort
.........
No MBI confirmation received from dSC

AUTOBOOT: Boot string = <not set>
AUTOBOOT: autobootstate=0, autobootcount=0, cmd=boot

Thats why Im trying to boot it from USB

Thank you for your support 

[Mery]

Maybe the files are corrupted and thats why it is not finding the boot image. Can you please tell where I can find a boot image for ASR9001 and can I boot it from USB?

[Mery]

Hi,

I have booted the ASR using tftp and it enters the terminal, but after a while it keeps rebooting and enter Rommon. I noticed that this didnt changed the nvram var

RP/0/RSP0/CPU0:ios#more nvram:classic-rommon-var

PS1 = rommon ! > , IOX_ADMIN_CONFIG_FILE = , ACTIVE_FCD = 1, IP_ADDRESS = 20.20.20.1, IP_SUBNET_MASK = 255.255.255.0, DEFAULT_GATEWAY = 20.20.20.200, TFTP_RETRY_COUNT = 4, TFTP_TIMEOUT = 6000, TURBOBOOT = , TFTP_MGMT_INTF = 0, BOOT_DEV_SEQ_CONF = , BOOT_DEV_SEQ_OPER = , TFTP_FILE = asr9k-mini-px.vm-5.3.4, TFTP_SERVER = 20.20.20.200, ? = 0, WATCHDOG_MODE = 1, BSI = 0, CLUSTER_NO_BOOT = , confreg = 0x100▒

How can I change this to boot from disk0: where the boot image is installed?

Thank you

[smilstea]

I think we are getting off track here, we need to understand why the 9001 keeps rebooting. You can certainty reimage the router using https://community.cisco.com/t5/service-providers-documents/asr9000-xr-understanding-turboboot-and-initial-system-bring-up/ta-p/3146024 but that likely is not going to resolve the rebooting problem.

The image you want is the mini-px.vm image for the turboboot. For booting the card you need the .mbi image, this gets created automatically by the system in a directory on disk0.

For some reason your RSP has no mbi image set to the BOOT variable so the RSP doesn't know how to boot, did you turboboot or just boot the .vm image without the turboboot variable? If so then you will need to turboboot to reimage the router.

Sam

[Mery]

Yes right I used TURBOBOOT

TURBOBOOT=on,disk0:,format

But still it is rebooting and that short time that I can login in ASR, it didnt change the boot var, it is still showing tftp as below

[Mery]

---

Also I see these logs when it fails to full boot:

: easybake3[187]: %INSTALL-TURBOBOOT-3-INVALID_DEVICE : The requested install device bootflash: is either corrupt or missing. Turboboot cannot be performed. The797 turboboot settings will be cleared, and the node will reset. Please repair or insert the device and retry the boot sequence.
: TURBOBOOT: Failed to obtain turboboot parameters: 'Turboboot' detected the 'fatal' condition 'No devices that qualifies for boot device are found in the system'
 TURBOBOOT: Resetting TURBOBOOT rommon variable to (TURBOBOOT=).
 TURBOBOOT: Setting config-register to NOT autoboot the router and NOT disable send break.
Failed reading/validating turboboot settings: 'Turboboot' detected the 'fatal' condition 'No devices that qualifies for boot device are found in the system'

 TURBOBOOT: *** RESETTING IN 30 SECONDS ***

Any idea?

[smilstea]

It's trying to install to bootflash and not disk0:

There is this note in the guide that I put there a few years ago, can you give this a try?

NOTE: a recent tac case showed that the command for turboboot failed on the ASR9001.

Supposedly this was made to work by omitting the colon after disk0:

Suggesting to try the disk0 (without colon) if the command with colon fails.

Sam

[Mery]

I tried that as well, but no luck still same issue:

rommon B1 > set
PS1=rommon ! >
IOX_ADMIN_CONFIG_FILE=
ACTIVE_FCD=1
WATCHDOG_MODE=1
IP_ADDRESS=20.20.20.1
IP_SUBNET_MASK=255.255.255.0
TFTP_RETRY_COUNUERVER=20.20.20.200
DEFAULT_GATEWAY=20.20.20.200
TFTP_SERVER=20.20.20.200
BOOT_DEV_SEQ_CONF=
BOOT_DEV_SEQ_OPER=
TFTP_TIMEOUT=6000
TFTP_VERBOSE=2
TFTP_RETRY_COUNT=2
TFTP_MGMT_INTF=0
CLUSTER_NO_BOOT=
BOOT=
?=0
TURBOBOOT=on,disk0,format
BSI=0

boot tftp://20.20.20.200/asr9k-mini-px.vm-5.3.4

easybake3[187]: %INSTALL-TURBOBOOT-3-INVALID_DEVICE : The requested install device bootflash: is either corrupt or missing. Turboboot cannot be performed. The turboboot settings will be cleared, and the node will reset. Please repair or insert the device and retry the boot sequence.
TURBOBOOT: Failed to obtain turboboot parameters: 'Turboboot' detected the 'fatal' condition 'No devices that qualifies for boot device are found in the system'
TURBOBOOT: Resetting TURBOBOOT rommon variable to (TURBOBOOT=).
TURBOBOOT: Setting config-register to NOT autoboot the router and NOT disable send break.
TURBOBOOT: Failed reading/validating turboboot settings: 'Turboboot' detected the 'fatal' condition 'No devices that qualifies for boot device are found in the system'

TURBOBOOT: *** RESETTING IN 30 SECONDS ***

Thanks

[Mery]

Attach is the whole output from the loading from tftp till the phase where it fails and enter Rom again

Its weird that it is still choosing bootflash to install the software. Could it be a wrong syntax of Turboboot?

Thank you

[smilstea]

That's what I thought at first, but it looks like your bootflash where the MBI lives is corrupt. We can try to format it and do turboboot again, if this fails we will have to RMA the 9001 due to flash corruption.

1) From ROMMON try to boot into hsbi:

rommon B1 > b hsbi:
Beginning HSBI boot:
Loading B image:
...............
program load complete, entry point: 0x1fbfe8, size: 0x3b5a98
Config = uP, Running = uP
##################
BSP: Board type : RO-RSP2
Use private TLB mappings
***********************************************

2) When it presents you with options list, choose next:

To format bootflash: ksh /etc/flash0_init

Hence the command would be:

ksh /etc/flash0_init

If you get an error like this we need to RMA.
rommon B3 > b hsbi:

Boot Status 16
Beginning HSBI boot:
Loading B image:
loadprog: bad file magic number: 0x0
Loading A image:
loadprog: bad file magic number: 0x0
Error loading HSBI image.

Thanks,

Sam

[Mery]

Yes it is showing the error below:

Beginning HSBI boot:
Loading B image:
loadprog: bad file magic number: 0x0
Loading A image:
loadprog: bad file magic number: 0x0
Error loading HSBI image.

Boot Status 15

So is there a way that I can load this image from tftp and copy it in bootflash:? Like copying the file from another ASR we have and load it in this faulty ASR?

Also when I run the commands below itshows usb not found

rommon B3 > dir disk0:
USB device not found
rommon B4 > dir bootflash:
USB device not found
rommon B5 > dir disk0a:
rwxrwxr-x 4096 .
rwxrwxr-x 4096 ..
r--r--r-- 244736 .bitmap
r--r--r-- 8192 .inodes
rw------- 0 .boot
rw------- 0 .altboot
rwxr-xr-x 4096 LOST.DIR
rwxr-xr-x 4096 LOST.DIR
r--r--r-- 0 .longfilenames
rwxr-xr-x 4096 usr
rwxr-xr-x 4096 var
rommon B6 > dir nvram:
Device not found nvram:
rommon B7 > dir disk1:
USB device not found

So does this mean it has hardware issues?

thank you