topic IOS-XRv9k stuck in boot loop in DevNet Sandbox

IOS-XRv9k stuck in boot loop

Hemant Sharma — Sun, 04 Aug 2024 02:32:44 GMT

In the CML sandbox, the ios-xr is stuck in boot loop. I have tried to increase the RAM to 64GB

https://devnetsandbox.cisco.com/DevNet/catalog/cml-sandbox_cml

Re: IOS-XRv9k stuck in boot loop

bigevilbeard — Sun, 04 Aug 2024 11:17:38 GMT

Hey @Hemant Sharma ive see a few threads on this, but not seen one resolved not one which says it finally worked for example I think this is what you are seeing https://community.cisco.com/t5/devnet-sandbox/ios-xrv-9000-fails-to-boot-in-devnet-cml/td-p/5141544

I am guessing the sandbox team at Cisco would need to try this and comment. It’s been a while since I looked at the images and their size, so it might be newer images are bigger or too big for the environment itself. As you noted you increased the RAM. I’ve seen others boot this in 2.5 CML with 64GB of RAM and 4 CPUs, after a while XRv node finally boots successfully after about 16 - 20 minutes. From what I am reading most of this appears to be on versions running too.

Re: IOS-XRv9k stuck in boot loop

Alexander Stevenson — Tue, 06 Aug 2024 12:16:33 GMT

Hi @Hemant Sharma,

I'm with Developer Support. I've arrived here after receiving your Feedback Form. We apologize for the inconvenience. Did waiting some time like the above post suggests help at all?

Re: IOS-XRv9k stuck in boot loop

Hemant Sharma — Tue, 06 Aug 2024 12:26:54 GMT

Hi @Alexander Stevenson

Neither the waiting time nor increasing the RAM to 64GB help.

Could you please take a look at it yourself?

Thanks!

Re: IOS-XRv9k stuck in boot loop

Alexander Stevenson — Tue, 06 Aug 2024 12:33:25 GMT

Thanks for checking @Hemant Sharma. I've pinged a sandbox admin with a link to this discussion so they can have a look. I don't have sufficient privileges to do it myself.

Re: IOS-XRv9k stuck in boot loop

msanghera6 — Tue, 06 Aug 2024 13:27:27 GMT

Something I've noticed is that CML give the option to select the image to run on the device, but only includes one image option. If it ends up being a "newer image" problem on CML, then it wouldn't be a bad idea for the DevNet Sandbox to include multiple images to pick from (2, maybe 3)

Re: IOS-XRv9k stuck in boot loop

devnet_geek — Tue, 06 Aug 2024 14:27:01 GMT

Hi @Hemant Sharma, XR node takes a longer time to boot up in CML which may takes to 20-30mins. There's also a problem we have noticed that if any key being pressed during its boot process on console, it gets stuck. In the upcoming release, we will be upgrading CML to new version and will offer only the nodes with no issues.

Re: IOS-XRv9k stuck in boot loop

bigevilbeard — Tue, 06 Aug 2024 14:37:18 GMT

@msanghera6 I don’t know the answer to this I am afraid. Typically sandbox runs a pretty new image most of the time, maybe not the very latest, but new enough. I’m the case of CML it thinks it’s the default images which the version that CML ships with.

Re: IOS-XRv9k stuck in boot loop

Hemant Sharma — Tue, 06 Aug 2024 14:52:29 GMT

Hi @devnet_geek

Then perhaps I must be doing something wrong because the XRv9k nodes in the base topology, which comes with the devnet CML sandbox did not work, even after leaving them be for 11 hours.

I have started it again and would not even open the console for an hour. Time starts now...!

Re: IOS-XRv9k stuck in boot loop

msanghera6 — Tue, 06 Aug 2024 15:33:28 GMT

One bit of information I would like to share

if you start the 9k node with the default resources, it doesn't boot (often times it will say "call trace" along with a long list of numbers:

but if you give the node a bunch of resources (I gave it 61035MiB of RAM and 14 CPUs) (and selecting the image definition)

Tue Aug 6 14:45:59 UTC 2024 (/proc/self/fd/9): Hardware profile: vpe
Tue Aug 6 14:45:59 UTC 2024 (/proc/self/fd/9): Host has 58.51GB RAM / 14 vCPUs
Tue Aug 6 14:45:59 UTC 2024 (/proc/self/fd/9): Management plane: 1024MB RAM
Tue Aug 6 14:45:59 UTC 2024 (/proc/self/fd/9): XR control plane: 19456MB RAM
Tue Aug 6 14:45:59 UTC 2024 (/proc/self/fd/9): XR packet memory: 256MB RAM
Tue Aug 6 14:45:59 UTC 2024 (/proc/self/fd/9): Centralized LC: 38912MB RAM
Tue Aug 6 14:45:59 UTC 2024 (/proc/self/fd/9): Data plane core assignment: 2-13
Tue Aug 6 14:45:59 UTC 2024 (/proc/self/fd/9): Control plane core assignment: 0-1

it will boot in CML after a few minutes

RP/0/RP0/CPU0:ios#show version
Tue Aug 6 14:56:39.286 UTC
Cisco IOS XR Software, Version 7.4.1
Copyright (c) 2013-2021 by Cisco Systems, Inc.

Build Information:
Built By : ingunawa
Built On : Wed Aug 4 04:18:28 PDT 2021
Built Host : iox-ucs-012
Workspace : /auto/srcarchive17/prod/7.4.1/xrv9k/ws
Version : 7.4.1
Location : /opt/cisco/XR/packages/
Label : 7.4.1-0

cisco IOS-XRv 9000 () processor
System uptime is 7 minutes

RP/0/RP0/CPU0:ios#

I can't say this specific combination of resources is the only mix of RAM + CPU thats allows it to boot up successfully, since this is the first combination I tried and it worked, and I haven't experimented beyond this

Re: IOS-XRv9k stuck in boot loop

Hemant Sharma — Tue, 06 Aug 2024 15:47:55 GMT

Yup, that worked. But can run only one XRv. 😞

It is what it is.

Re: IOS-XRv9k stuck in boot loop

devnet_geek — Tue, 06 Aug 2024 16:10:43 GMT

Yes, we have a limit on resources assigned to a sandbox reservation. For CML specifically, we offer several Cisco based product nodes that you can deploy and test. Just be in mind that sandbox service is free and if you want to test a production level topology, I would recommend to look for CML enterprise offering.

Re: IOS-XRv9k stuck in boot loop

Hemant Sharma — Tue, 06 Aug 2024 16:13:26 GMT

But if this is a limitation of the CML platform itself, then one needs more than 3 times the vCPUs it would require to run 1 node, even on the enterprise platform, unless the enterprise edition is optimized for resource consumption.

Thanks!

Re: IOS-XRv9k stuck in boot loop

msanghera6 — Thu, 08 Aug 2024 02:35:17 GMT

Last update to share

You need to give the node 7 CPUs minimum (RAM can remain at the default)