topic Re: New Updates to CML and NSO sandbox labs in DevNet Sandbox

New Updates to CML and NSO sandbox labs

jokearns1 — Fri, 11 Oct 2024 09:18:51 GMT

Hi Community,

Yesterday, we released updates to our popular CML and NSO sandboxes.

Cisco Modeling Labs (CML)
We want to scale the CML sandbox. Right now, if slots are full, users have to wait. This is annoying and we want to address it while keeping the core CML functionality.

CML updated to 2.7.2
Supported Routers --> CSR1000, IOL, IOSv
Switches --> CSTkv, IOL-L2, IOSvL2,
Linux, --> Ubuntu, Alpine, TREX
Other --> WLC, External Connector

We have removed the huge XRv node. It just ate too many resources to scale the sandbox adequately. If anyone wants to experiment with XRv programmatically, we have a dedicated XRd sandbox for that.

The sample topology has also been updated. See below:

These changes will allow us to scale up concurrent sessions and help address demand. We cant promise we will ever reach demand but this will definitely help.

Network Services Orchestrator (NSO)

The NSO sandbox also received some updates yesterday.

New Docker containers added for NSO
Updated CML topology
Updated access information and instructions

This is the first of a two part update, the second being pushed next week.

We would like your feedback on any updates we push. Please reach out to this thread with any questions or comments.

Thanks,

Sandbox Support

Re: New Updates to CML and NSO sandbox labs

mariem56 — Sun, 13 Oct 2024 19:34:55 GMT

Will you change the reservation time? I think the max is 2weeks? and maybe because of that CML is always full...

Re: New Updates to CML and NSO sandbox labs

jokearns1 — Mon, 14 Oct 2024 07:05:03 GMT

Hi @mariem56
Reservation time is normal 5 days. However, we may reduce this to get more labs available.
I have increased the max number of labs again so more are available now.
Thanks,

Joe

Re: New Updates to CML and NSO sandbox labs

wahezu — Mon, 14 Oct 2024 09:55:28 GMT

Hi,

NXOSv was also removed. Is there any other lab where we can test NXOSv?

Thanks,

Re: New Updates to CML and NSO sandbox labs

wahezu — Mon, 14 Oct 2024 10:16:39 GMT

Regarding my last comment, I can see other labs where we can test NXOSv but with a single node, I'd like to test a complete spine&leaf topology, which I was able to test on the CML sandbox until very recently when NXOSv node support was removed.

Thanks,

Re: New Updates to CML and NSO sandbox labs

Douglas Philyaw — Mon, 14 Oct 2024 14:00:59 GMT

We're having issues with the cat8000v images using more memory than prior to the update. We've tried increasing the available memory for each the nodes but this doesn't seem to help. We've also tried lowering it and the nodes won't boot up at all. Is there a limit on cat8000v nodes?

Re: New Updates to CML and NSO sandbox labs

mgottesfeld — Mon, 14 Oct 2024 19:41:32 GMT

Dear Sandbox Support:

First, I appreciate your efforts to scale the C.M.L. sandbox. Without the sandbox I never would have passed ENCOR last week after a significant hiatus from hands-on deployments!

That said, in case you are not already aware of this (which does seem unlikely), the A.S.A. is gone from the C.M.L. sandbox, and, as a result, several of the demo labs now refuse to load. I see nothing in the announcement, above, regarding removal of the A.S.A. Nor do I see any change to the stated purposes of the C.M.L. sandbox. I thus assume Cisco wishes to continue to make it available to folks like myself to learn.

Due to the ASA's ubiquity—a tribute to its success—it is quite impossible to model a realistic enterprise network without an A.S.A. pair.

I, too, miss the NXOS 9000. It was, in my recollection, CML's only layer-three switch with meaningful port density and vPC. That said—and I speak only for myself here—I can make do with any layer-three switch that supports dynamic routing protocols, FHRPs and vPC, stacking or V.S.S.

Last, it must be mentioned that, thus far, the update effectively ruins the C.M.L. sandbox as a pre-sales tool for smaller Cisco partners who lack the resources to run C.M.L. internally. Please see the attached YAML file, on which I was working when the update began and which now refuses to load. This type of lab (yes, I know, it is not all working, as I was in the middle of a reconfiguration), shows the former power of the C.M.L. sandbox as both a learning and sales tool. (No more than 20 of the nodes must be powered on to demonstrate any particular feature, e.g., D.M.V.P.N., M.P.L.S., eBGP E.C.M.P., etc.) My understanding is that Juniper continues to offer this kind of free sandbox. See Juniper vLabs https://jlabs.juniper.net/vlabs/.

I had planned to publish the attached lab for free as a learning and sales tool upon its completion, while I work toward C.C.I.E.

I thank you for your consideration of this matter,
by /s/ mgottesfeld, Cisco Certified Specialist—Enterprise Core; Cisco Certified Network Associate (2003, expired).

Re: New Updates to CML and NSO sandbox labs

bigevilbeard — Tue, 15 Oct 2024 09:29:46 GMT

@mgottesfeld some thoughts here. Whilst I understand that many people use the sandbox for different reasons, it’s real reason is to supplement learning for the Devnet learning labs, however as this was built and replaced virl that it should be more open for more general use. But this from what you are describing your use case not suitable and I am sure you have hit limits in your path here etc. Cisco does provide an environment for pre sales and demo, via Cisco dCloud. I would suggest for your use case this is more suitable for your needs and has more CML labs. Though not ideal if you are making content and pointing users to this as they might not all have the access required.

Re: New Updates to CML and NSO sandbox labs

mgottesfeld — Tue, 15 Oct 2024 18:21:33 GMT

First, thank you for pointing me to dCloud. When I am back in the field, which I am not now while I recertify, I will be sure to evaluate it. It is unclear to me, however, whether dCloud allows for customization of the demos, like adding or removing nodes and offices, changing WAN designs and parameters, saving and loading changes, etc.

Customers want to see their networks, their logos and that a Cisco partner has paid close attention to their specific needs. In the sales context, generalized demos are unpersuasive.

Second, I respectfully disagree with your statement that the C.M.L. sandbox's "real reason is to supplement learning for the Devnet learning labs." Cisco's statement of purpose that one reads when launching the C.M.L. sandbox makes no express mention of "Devnet":

Cisco Modeling Labs is a tool for building virtual network simulations (or labs) for you to test out new topologies, protocols and config changes; automate network tests via [continuous integration/continuous delivery] pipeline integration; and learn new things about the cool world of networking.

This sandbox provides access to a Cisco Modeling Labs system that can be used to explore the capabilities of the newest release of Cisco Modeling Labs Personal or Enterprise.

See screen capture, attached (emphasis added). Cisco may, of course, change this statement if it wishes. Unless and until it does, I and others may only assume that Cisco says what it means and means what it says.

The key elements of the C.M.L. file on which I was working were—

For those pursuing the enterprise-infrastructure (formerly R&S) track, it would have allowed them to study and test a realistic enterprise network in stages as they progressed through the syllabi; and
It would have been about as easy as possible for sales engineers to customize to suit particular leads.

The relatively level playing field Cisco has always offered through its certification program, e.g., no formal prerequisites for the C.C.I.E. and exams relatively light on marketing fluff, remains a beacon in a world that seems ever less concerned with a candidate's merit. The availability of the C.M.L. sandbox went a great way toward opening the door to the widest possible talent pool, to folks who otherwise would be priced out of a C.C.N.A.

Perhaps the primary limit that dissuaded me from pursuing a C.C.I.E. twenty years ago was the cost of effective lab time. If the changes announced in this thread had been made to the C.M.L. sandbox weeks earlier, I am unsure I would have passed ENCOR to become a C.C.I.E. candidate.

I continue to hope that Cisco is aware of and values the utility of the C.M.L. sandbox to "learn new things about the cool world of networking," especially given there are never enough certified engineers to meet every customer's needs,

by /s/ mgottesfeld, Cisco Certified Specialist—Enterprise Core; Cisco Certified Network Associate (2003, expired).

Re: New Updates to CML and NSO sandbox labs

Neeraj-Kochar — Tue, 15 Oct 2024 20:03:26 GMT

I have to admit, although I am very grateful for CML, I am incredibly disappointed with the latest release. the Nexus 9k was the only CML switch with any real advanced feature sets and the only Data Center centric switch in the lineup. As someone who works as a Cisco Data Center engineer this decision has significantly if not completely reduced the value of CML for me. I developed a complete mock up of our Data Center in CML and used it as a modeling and simulation lab prior to making any changes in production, which was incredibly helpful, and is now rendered useless. I truly hope this is not permanent.

Re: New Updates to CML and NSO sandbox labs

bigevilbeard — Wed, 16 Oct 2024 08:54:42 GMT

@Neeraj-Kochar you have the option to purchase CML?

Re: New Updates to CML and NSO sandbox labs

bigevilbeard — Wed, 16 Oct 2024 09:23:36 GMT

@mgottesfeld for sure checkout dcloud it’s a great resource, if you have access - you would need to see if this would meet your needs as far as building something custom. We can go back and forth on wording here of the 'why' CML is there, but using this as a POC, DEV infra, tool testing for company, i would never suggest this and lets face it, if someone is using a free resource which is meant for the community to build and generate money, thats not really fair right? For example in your case, you would not have been able to book CML and pass your exam. There is/was plenty of thread on the reservation of CML and now the team has had to make bigger changes to allow folks to use this now.

So, DevNet is small team inside Cisco which focuses on education and network automation, they are not part of learning, or Cisco corp so its not really a good fair to say 'Cisco' when talking about DevNet. Hence my comments on POC etc.. as there is a dedicated part of Cisco for small biz, POC etc.. so there is better methods for sale too here.

Yeah totally with you on the CCIE study, when i was doing this i had stuff from evil bay, was ripped off many times with broken, non working stuff, GNS3 was only just out and not really an option at the time, it was hardware. You will have been part of many talks why Cisco does not provide student images, test images.. this is as old as time rant, or better labs for learning for certs, free tier even which will get you so far, but scaling this is cost, hence why DevNet can only run a limited capacity and footprint.

So really my point here, DevNet and this small amazing team which provided all these free resources cannot be shouldered with holding up the big machine of Cisco and all its customer use cases.

Re: New Updates to CML and NSO sandbox labs

jokearns1 — Wed, 16 Oct 2024 09:25:16 GMT

Hi All

I lot of activity on this thread. Firstly, thanks for your feedback.

Its difficult to find a balance between CML sandbox functionality and available resources in our backend.

The CML sandbox was never meant to be a complete solution for everyone and catch all use cases. The primary aim of DevNet is to give developers real world access to Cisco APIs and sandbox is a tool in the armoury. We wanted to address scale issues as this was the major feedback from the community.
Also, the CML sandbox does not strictly follow any Cisco examination text. If we did, we would need more datacenters and lots more people. The CML sandbox has to be as generic as possible to catch as many use cases as we can while allowing scale.
We will consider feedback on available CML nodes and update this thread.
Once again, we appreciate the discussion around this.

Support

Re: New Updates to CML and NSO sandbox labs

mgottesfeld — Wed, 16 Oct 2024 16:07:06 GMT

Once again, I respectfully disagree.

After your post, Cisco partially validated both our points. The "primary"—but, noticeably, neither the "real" nor sole---"aim of DevNet is to give developers real world access to Cisco APIs." (emphasis added). See jokearns1 (1025 hrs. G.M.T. Oct. 16, 2024), below. Importantly, "The [C.M.L.] sandbox has to be as generic as possible to catch as many use cases as we can while allowing scale." Id.

Thus it seems that the truth of your statement, that the C.M.L. sandbox is "a free resource which is meant for the community to build and generate money," is wider in scope than you may have first considered.

As just one, recent, example, a 2020 "survey revealed that 86% of enterprises consider[ed] talent shortage to be a key barrier to achieving their desired outcomes." See Everest Group, 75% of Enterprises Bracing for Widening IT & Analytics Talent Shortage, Despite Recession (Aug. 19, 2020), available at https://www.everestgrp.com/2020-08-75-of-enterprises-bracing-for-widening-it-analytics-talent-shortage-despite-recession-press-release-.html (last accessed Oct. 16, 2024).

To focus on one particular area, relevant to the removal of the Adaptive Security Appliance from the C.M.L. sandbox, the "widening cybersecurity talent gap not only poses a risk to the global economy, it opens the door for increased cyber-attacks—especially as tech like AI makes it easier for bad actors to infiltrate systems." See Fore, Cisco is continuing to invest in the future of skills certifications. Here's how the company thinks it'll help fill the lack of cybersecurity experts, Fortune (Oct. 16, 2023), available at https://fortune.com/education/articles/cisco-u-cybersecurity-skills-certification-training-program-cisco-networking-academy/ (last accessed Oct. 16, 2024). "Thus, says Francine Katsoudas, executive vice president and chief of people, purpose and policy officer[sic] at Cisco, it is more important than ever to address the cybersecurity skills shortage.… 'Cisco's purpose is to Power an Inclusive Future for All, and building cybersecurity skills and tools is a crucial part of creating that future.'" Id. (emphasis added).

Please do not take for granted that Cisco-certified engineers enable, empower and secure the very solutions you come to DevNet to build.

I see nothing unfair about my or any other certification candidate's use of the platform. The separation between Cisco and DevNet that you assert is contradicted by some of Cisco's deliberate choices: We are on the Cisco.com forums, the administrator who started this thread requesting feedback is conspicuously labeled a "Cisco employee," not a "DevNet" or even a "Cisco DevNet" employee.

And, of course, regardless of how Cisco chooses to structure itself, it should aim to exceed, or at least for parity with, its competitors' offerings, e.g., Juniper vLabs.

Ironically, too, those who used GNS3 back in the day to run Cisco platforms, before Cisco offered VIRL images, were almost certainly breaking the law by their non-sanctioned uses of I.O.S. and A.S.A. images. Cisco could have cracked down hard on the practice. Its apparent choice to refrain seemed a deliberate, if tacit, acknowledgement of the learning community's needs and the benefits to the wider community of having some such platform available, at least until it got VIRL off the ground.

As for personal C.M.L. purchases, U.S.-based candidates are, for the most part, willing to pay $200 a year for C.M.L. That cost alone is minor compared to the costs of the actual exams and preparation materials. And CML's dollar-to-value ratio as a practice resource would be unsurpassable. Though sufficient for some candidates, especially those with some degree of corporate support, this approach has big showstoppers more broadly.

First, in many if not most parts of the world, $200 is quite a substantial sum.

Second, even in the U.S., most unaffiliated candidates lack the personal hardware to virtualize 20 concurrent nodes.

I thus reiterate that, if Cisco's goal is to tap the widest possible portion of the global talent pool, a learning resource like the C.M.L. sandbox is vital.

All this said, I hope for an outcome with the C.M.L. sandbox that acknowledges and empowers both our use cases. I will respond directly to jokearns1, below, with ideas, and I welcome your and others' commentary on my proposals.

Re: New Updates to CML and NSO sandbox labs

mgottesfeld — Wed, 16 Oct 2024 16:06:21 GMT

It is likely worth pointing out, in case the original commentor did not already know, that the purchasable version of C.M.L. still includes the Nexus 9k. Its removal from the sandbox does not indicate a removal from the base C.M.L. product. See VM Images for CML Labs (continuing to list the "NX OS 9000"), C.M.L. 2.7 User Guide, available at https://developer.cisco.com/docs/modeling-labs/vm-images-for-cml-labs/#non-cisco-vm-images (last accessed Oct. 16, 2024).

Re: New Updates to CML and NSO sandbox labs

bigevilbeard — Wed, 16 Oct 2024 17:01:50 GMT

@mgottesfeld I appreciate your detailed and thoughtful response. Let me share some additional context that might help frame this discussion better.

Speaking from my experience (12 years at Cisco, with 5 years on the DevNet team where I helped build many of these resources, inc DevNet cert and the DevNet Expert track), I can tell you that the sandbox challenges you've identified have been ongoing since its 2016/17 inception. Your points about accessibility and learning resources are correct, i 100% agree. The real issue we're grappling with isn't certification candidates or learners using the platform - that's exactly what it's meant for. As Joe noted, for using getting hands on exp with API's this was DevNet was formed.

The challenge comes from companies using it as a replacement for their own QA infrastructure, or people monetizing it through paid courses and services. This kind of usage consumes resources meant for learners and developers, undermining the original DevNet mission. While I strongly support certification candidates using the sandbox (and would never suggest otherwise), we need to acknowledge its technical and resource limitations. These constraints affect everything from scale to platform version control to security posture.

You make an excellent point about cost barriers - I faced similar challenges in my own journey. The solution isn't restricting legitimate learners, but rather finding ways to prevent commercial exploitation while expanding access. During my time at Cisco (and continuing now as a Cisco Champion and VIP), I've advocated for better solutions to support students and learners.

You've provided valuable insights here that highlight the continuing need to balance accessibility with sustainability. I share your vision for more comprehensive learning resources - we just need to ensure they're protected from misuse that could limit their availability to the people who need them most.

BR.

FYI The forums are managed by each Cisco org/team, although yes in the Cisco domain all Cisco staff show as 'employee' regardless of their role within the company.

Re: New Updates to CML and NSO sandbox labs

mgottesfeld — Wed, 16 Oct 2024 18:17:34 GMT

Hello jokearns1:

First, I thank you for seeking and reading feedback on this thread.

As a systems engineer and C.C.I.E. candidate, I appreciate both the value of the C.M.L. sandbox and the difficult decisions facing your team as a direct result of its success and popularity. I hope you feel the well-deserved admiration of this entire community every day for such a great resource.

I also hope that I have sufficiently pleaded the case, above, for learners' continued access to such a platform, and sufficiently highlighted its potential sales and pre-sales value, especially for smaller Cisco partners.

Throughout my career, Cisco has affirmatively taken steps to recognize, embrace and empower both meritorious candidates and smaller partners. I remember, two decades ago, 1) reading that the C.C.I.E. had no formal prerequisites and 2) my first 90%-off lab-discount order (available to Cisco partners). C.C.N.A. was my first certification, and I recall contrasting it to exams I later took from Citrix, Microsoft, RIM, etc., where, e.g., the right answer would be marked wrong and only the vendor's costly training warned candidates beforehand, exams where all the questions and answers had been leaked and exams devoid of substance but heavy on marketing jargon. As a result I, like many others, favored and continue to favor Cisco certifications heavily.

Cisco's broad, longstanding culture of empowerment and level playing fields now seemingly stands at a fork in the road with the vital goal of open developer access. I respectfully submit that this is a false choice and that Cisco can and should pursue both paths.

C.M.L. requires significant backend resources. For this reason foremost, its value to certification candidates and developers is high, and it also poses serious scalability challenges. I submit that Cisco is best suited to solve and manage the underlying infrastructure challenges, not other vendors, individual candidates and developers or a combination thereof. It is worth noting that many certification candidates have only their personal hardware on which to run a local instance of platforms like C.M.L. See, e.g., Bombal, Homeless security guard to CCIE: You can't train ambition: Katherine McNamara shares her story, Cisco Live, available at https://www.youtube.com/watch?v=hCRjDwuuQOA (last accessed Oct. 1, 2024).

That said, backend resources cost money. To the extent that Cisco can afford to provide open, free-as-in-beer access, I implore your team to continue to do so. I worry that the current demand on the sandbox is somewhat overstated. Even now, the system is sometimes below session capacity. The maximum wait I experienced to start a session before the changes announced in this thread was a little over an hour. I worry that the five-day session limit allows many sessions to sit idle for significant periods when they should have ended or been suspended, were suspending a session possible while also enforcing a five-day total cap (e.g., a user starts a session Monday morning, but then sits idle, causing automatic suspension of the session from Monday evening to Tuesday afternoon, at which point user interaction is required to resume the session, while still force-terminating it on Friday).

Of course, in the long term, Cisco is likely unable to meet the increasing demand for the C.M.L. sandbox for free. Its current popularity, however, is a solid viability indicator for reasonable, fixed-price market offerings to candidates and developers. Though incomplete and loaded with caveats, such offerings have long been on the roadmap. See, e.g., Schmieder, How to run Cisco Modeling Labs in the Cloud ("We have heard requests to have CML Software-as-a-Service (SaaS), and we’re working hard to make this a reality in the future.") (Sept. 1, 2023) (emphasis in original), available at https://blogs.cisco.com/learning/how-to-run-cisco-modeling-labs-in-the-cloud?gad_source=1&gclid=Cj0KCQjwyL24BhCtARIsALo0fSDjRxeTUA_kLwTww2qiOozGdRXN7UxfQlyAi4H4VcPneDzjqI4DSq8aAgUpEALw_wcB&gclsrc=aw.ds (last accessed Oct. 16, 2024).

As it stands, a single-vendor hosted offering remains conspicuously absent from the current C.M.L. product. See Cisco Modeling Labs—Personal, The Cisco Learning Network Store ("Cisco Modeling Labs—Personal is distributed as an OVA file for you to deploy as a virtual machine on supported VMware products or as an .ISO file for bare metal installation."), available at https://learningnetworkstore.cisco.com/cisco-modeling-labs-personal/cisco-modeling-labs-personal/CML-PERSONAL.html (last accessed Oct. 16, 2024).

If your team has not already offered to the relevant other Cisco team(s) to use the current C.M.L. sandbox as a blueprint for a fixed-price C.M.L. hosted solution, then I urge you to do so. Your team has already solved the hard challenges. Technically speaking, horizontally scaling the grid and checking subscription statuses as a requisite for entry seem minor additions to what you have built.

Last, I urge Cisco to continue to value the flatness of the playing fields it has thus far maintained. I hope that Cisco will consider the long-term value of tapping a broad, global talent pool for certified personnel when it considers profit and loss for a hosted C.M.L. offering. I also urge creative thinking as to its pricing models, e.g., account for the prevailing local wages in customers' areas, provide discounted access to those with competitors' certifications, those who have booked Cisco exams and those eligible to advance through higher-level Cisco exams and consider a scholarship program and discounts for partners, traditional high school, vocational school, college and community college students and those adversely impacted through no faults of their own by economic conditions, prejudice and injustice.

I sincerely thank you for your consideration,
by /s/ Martin MartyG Gottesfeld,
Rolling Stone-featured human-rights advocate,
Former HufffPost contributor,
Former Reporter, The Intercept and
Cisco Certified Specialist—Enterprise Core, Cisco Certified Network Associate (2003, expired).

Re: New Updates to CML and NSO sandbox labs

mgottesfeld — Wed, 16 Oct 2024 18:45:47 GMT

Thank you for the clarification as to candidates' legitimate uses of the sandbox. I totally misunderstood your first paragraph until now!

Indeed, we stand agreed on the major points. I was unaware of the natures of some of the commercial exploitations of the platform (though I do think folks who author courses should be able to point folks to the sandbox). And I know the platform could never possibly scale, for free, to meet what will ultimately be its huge global demand.

I say without overstatement that the DevNet team has built a lifechanging resource for learners. I thank you for your contributions to it,
by /s/ mgottesfeld.

Re: New Updates to CML and NSO sandbox labs

msanghera6 — Wed, 16 Oct 2024 22:40:38 GMT

Some of my thoughts on this new update

I like the new interface. It feels fluid to navigate around the topology. It does take some playing around to see where things have been moved to, but eventually you'll re-familiarize yourself with where everything is
I like how the node leave a "trail" behind it when you move it around. Very nice addition.
I like the ability to run a packet capture by right clicking a link. Its much quicker to do it way, rather than having to recall where the older packet capture option was in the older UI
The tabs for each device's console is another great improvement!
Also being able to have console windows side by side is very nice.
I am disappointed to see that ASA, XRv9k, NX-OS are no longer available. I understand the resource limitation, but hopefully the team finds a compromise, and allows these nodes to be re-added. I loved playing around with IOS-XR, and this was a great way to learn more about the OS and apply what I learn in the lab towards real world problems and scenarios at my workplace, where we have some routers that run XR. Same applies to ASA and NX-OS. Sad to see those go away 😞
As for suggestion, could you allow for us to select from a variety of Image Definitions?
Zooming in and out feels like it could be improved on. When I use my scroll wheel, the amount of zoom is a bit more than what I need, which results in me needing to use the + and - button to get the zoom just right. Perhaps the amount of zoom "per scroll" could be adjusted, or implement a zoom slider?
being able to reserve a lab for 4-5 days feels VERY LONG, as opposed to the older limit of 2 days.

Re: New Updates to CML and NSO sandbox labs

simranjitjohal1998 — Sun, 03 Nov 2024 16:53:49 GMT

Hello jokearns1,

are you planning to Nxos images in CML again ? these were very beneficial, i would say we prefer to wait for the slot rather than having half functionality.

Thanks,