CML Memory Utilisation Limit

[blakemweston]

Hi, we have deployed CML 2.8 on baremetal and are hoping there is a way to adjust the system memory utilisation limit of 80%? Our server has 256GB of RAM and an 80% limit on large servers is quite a waste. We were hoping to be able to run at least a couple more Cat9k / N9K / ASR9K nodes for the "virtual twin" of our core but alas we hit the 80% limit with everything running well. I'm hoping there is switch via the console to increase the limit

[mpeerenboom]

Hello Blakemweston,

This is a well-known problem of CML. Unfortunately, you don't give much information about the hardware specifications except for the amount of RAM. CML demands a lot from the hardware. I already have some experience there and have previously dedicated an article here in which I compared CML with GNS3. In my opinion, CML uses much more resources than GNS3. The big advantage of GNS3 is that you can set a trotthle per device where you assign a limit to the maximum memory that can be used. As far as I know, this is not possible in CML. I run heavy labs in GNS3 with the images I download from the CML website. Less heavy labs are workable in CML. But often you have to wait quite a long time before all images are started, . Even though my (newest) hardware is quite hefty Intel I9 14th generation 32 cores 64 Gb ram

[blakemweston]

Hi mpeerenboom,

Thanks for the response but it isnt really appropriate or related to the core question. The hardware currently in place was selected to ensure good performance for running a significant number of virtual nodes. The server has:

2x Intel Xeon Silver 4416+ 

256GB RDIMM

2x NVME 480GB (RAID 1) as Boot

2x SAS 800GB (RAID 1) for additional lab storage

The base resource limits are a bit inconsistent, as the 80% memory limit leaves ~40GB unused in our deployment yet the CPU limit is 95%.

[mpeerenboom]

Hello Blakemweston,

Unfortunately, I don't have an answer to your question. I have a hp proliant server running with the specifications below. (see photos) Also a raid setup. However, I don't notice much difference in terms of performance between this monster and my minipc Intel I9 32 cores 64Gb Ram and 1Tb NVME, both with GNS3 and with CML 2.7.1 So the specifications don't always say everything. What I do know is that it is possible to throttle the limits in GNS3 and its really a good feature. As far as I know its not possible in CML and that is a disadvantage compared to GNS3.

[Ramblin Tech]

You might be able to game the system a bit by careful ordering of node start-up, as, IIRC, the 80% memory limit pertains to starting a new node. That is, I believe you can start a new node if RAM usage is at 79%, even though the new node will take you over 80%. That being the case, you could conceivably start an XRv9K node (nominally 20G RAM without kernel same-page merging) with utilization at 79% and push into high-80% territory. Not really a fix, but might get one more high-RAM node to start.

[mpeerenboom]

Thank you for your advice. However, I think it's about time that the Cisco CML developers start indicating more clearly which specifications the hardware really, I repeat, really have to meet in order for a lab with multiple routers and switches to run smoothly. The official recommendations are far from the truth. Anyone who has tried to boot multiple routers, switches, and hosts with these recommendations knows that this is not possible without massive delays in booting and working with the labs themselves. Playing with the images as you recommend above is a false solution in my opinion. It should be possible, just like in GNS3, to assign a certain percentage to an image in both the amount of RAM and the percentage of CPU used by CML processes.

[mpeerenboom]

Another solution is to lower prices of CML lab so that more people can be able to pay for the necessary hardware. In my opinion the prices are too high.

[blakemweston · Accepted solution]

Just an update to anyone who finds this thread, there is a process to be able to modify the total resource limits in CML. Not sure whether Cisco want the process publicly repeated. Our server is happily operating with the Memory Utilisation limit at 95% but that is also due to the large amount of memory we have. 

If you are able to reach out to Cisco for the process to modify the limits.

[mpeerenboom]

Update. I finally found my ideal lab. After VMware was acquired by Brocade I decided to look for another solution and that is Proxmox: How to install CML on Proxmox

I bought a barebone mini pc and expanded it with 64Gb DDR5 ram memory. I installed proxmox on it and installed CML 2.8.1 with the help of the manual. This is the best setup I have ever had and CML runs smoothly on it.