Network configuration automation ssh hangs

[andreasfranz]

Hey community,

I'm having issues configuring a stack of four C9300L-48UXG (Version 17.8.1r[FC2]) switches with Ansible. My proof of concept worked fine, I'm able to setup everything we need to the first 48 Ports of the stack. But as soon I exceed a certain amount of ports to configure I'm facing timeouts.

So after lots of trials and testing I started trying the same with Netmiko, with almost the same result. The soon I want to configure all ports, the ssh connection stops responding.

So I tried the next pure lib paramiko and started investigation. I can reproduce, that simply try to enter config mode and exit about 200 times will fail.

So my question is, is there a setting limiting ssh config commands amount, or something timing related and getting triggered.

Thanks in advance.

[andreasfranz · Accepted solution]

The issue was neither related to Netmiko nor Cisco IOS. It turned out to be an SSH problem caused by asymmetric routing. Due to the access control lists in place, I could only reach the switches from a specific network. Today, I had the opportunity to test everything locally within the same VLAN, and it worked immediately.

The tricky thing about asymmetric routing with SSH is that the connection appears to be terminated only after a certain amount of time. I didn't notice this in the logs. When making a manual SSH connection, the problem didn’t occur.
Hopefully, this will help someone who encounters a similar issue.

If the SSH connection is unexpectedly terminated after about 60 seconds, asymmetric routing could be the cause.