topic Re: CML SandBox Node spin up issue in DevNet Sandbox

CML SandBox Node spin up issue

abhishekkumarrakeshkumarmishra — Fri, 24 Oct 2025 12:53:04 GMT

I cannot spin up more than 11 nodes. I get the following error.

Warning: High System Load (memory) on cml-controller: launch sequencing will be paused until system load stabilizes

And, for some reason the CSR1000v memory requirement is always stuck 99.98%. For IOS nodes, it power up and then stabilizes to around 60%.

I have re-started CML but the error is still the same. I clearly remembered running approximately 15 to 20 last time for a different LAB last time. (I guess before this recent maintenance)

Re: CML SandBox Node spin up issue

bigevilbeard — Fri, 24 Oct 2025 14:17:50 GMT

@abhishekkumarrakeshkumarmishra is this on the devnet sandbox?

The warning "Warning: High System Load (memory) on cml-controller: launch sequencing will be paused until system load stabilizes" is CML's way to prevent the host system from running out of resources (specifically memory or CPU) when starting new nodes.

Have you been able to run bigger labs before here on the sandbox?

Thanks!

Re: CML SandBox Node spin up issue

abhishekkumarrakeshkumarmishra — Sat, 25 Oct 2025 07:28:22 GMT

Yes I have been able to run bigger labs before as well.

Typically I have used at around 15 to 20 nodes as well.

Re: CML SandBox Node spin up issue

bigevilbeard — Sat, 25 Oct 2025 15:49:29 GMT

I saw the limit for reservations was increased back to the 120 from 50. It might be specific CML reservations can fluctuate based on what other users are doing on the same physical hardware.

When you ran the other larger labs was you using the same devices? Reason for asking is. The CSR1000v is a very intensive node image in CML, requiring a minimum of $\mathbf{3.0\text{ GB}}$ of RAM to even boot, and often more to run smoothly. If you swapped these out for ISOv devices would you get a bigger topology?

Edit: I just saw the max reservation was lowered to 75.