CML SSH very unreliable

[radokristof]

I have created a network in Cisco DevNet - Cisco Modeling Enterprise lab.

The routers are connected to an unmanaged switch, the switch is connected to the external connector. External connector is set to bridged.

I have set a correct IP address for these interfaces in the routers (10.10.20.0/24, eg: 10.10.20.101/24), enabled SSH and everything, but the connection is really unreliable.

I have tried to put these interfaces into a VRF and add a route for that VRF - like in the sample config - that's even worse. Even the original Sample Network is really unreliable.

Sometimes SSH works, sometimes not (Connection timed out). Sometimes it drops when I have already SSHd into the router. Then I cannot access it again... Sometimes it appears again.

Alpine Linux hosts are working always correctly.

I don't know I'm doing wrong. Can someeone give me a hint?

[Alexander Stevenson]

This is the first time I've heard of this. It it persists, I'd open a sandbox ticket by clicking on the green 'Report a Sandbox Issue' button here --> https://community.cisco.com/t5/devnet-sandbox/bd-p/4426j-disc-dev-devnet-sandbox

[jdjalm23]

Came upon your post while searching for others having this very same issue.

It is indeed true that SSH is buggy and unreliable. Sometimes it will connect right away on the first attempt only to time out on subsequent attempts. Sometimes two Cisco devices of the same type and with the exact same config (except IP address) will have one of them connecting without issue and the other not. I haven't experimented much with the Linux nodes. At first I was racking my brains trying to figure out what I was missing on my Cisco SSH/routing config for this to be failing but after extensive testing it became obvious that the lab environment is at fault.

It'd be nice if Cisco would provide a bit more information on how one is supposed to go about setting up the SSH bridged access just so that we can be 100% sure that the issue isn't on the user end. It's not overly complicated, as you said, it just required mimicking the setup in the sample lab: external connector set to bridged mode connected to an unmanaged switch. But all of this isn't explicitly explained. One has to deduce this is how it's supposed to work, including what subnet, 10.10.20.0/24, we're supposed to use. Did you know that the VPN connection also pushes routes for 10.10.21.0/24 through 10.10.24.0/24? That's five /24s - are we okay to use any of them? How's the routing working on the back end? Next to no information is given out on this which is a shame because it's a pain to use the tiny console window in the CML GUI to interact with devices. This is my only gripe with CML sandbox, otherwise, it's awesome for labbing XE, NX-OS, XR and automation.

[radokristof]

Indeed there are problems with SSH and generally accessing devices through VPN in CML. I have contacted Cisco after creating this post. Mainly, if you can't access a device at all, and you connected a L2 switch to the cloud-connector, that usually means that somehow the MAC address of your device which is unable to connect changed. The L2 switch ARP table contains the old MAC address for that IP address, that's why you can't access it. These applies to all nodes obviously.

I think two solution is viable here:

1. Use a managed switch where you can issue the "clear arp" command
2. Restart the whole lab by wiping the nodes and everything.

After I have changed my switch to a managed one this problem gone. However sometimes I still got SSH connection problems. 

What you can check in this case:

• If you have imported the lab, check for SSH keys. The keys will not be exported, so you have to re-create them or manually insert them into the config script. Until the keys are not installed in the router, SSH will not work
• Just try to reconnect. Sometimes my routers disconnected for no reason. It would seem like a network/VPN issue, but 6 SSH sessions were running usually (different hosts in the same lab) and only one disconnected.
• I would only use the 10.10.20.0/24 subnet, I also saw that there are other subnets as well, but I don't know what its their purpose. The cloud-connector is connected to the 10.10.20.0/24 subnet I think.