topic Re: pyATS connection refused in Tools

pyATS connection refused

vsfomin — Mon, 20 Dec 2021 16:30:14 GMT

I'm trying to get pyATS to work, but it won't connect to the devices in any way.
There are several vIOSs deployed in the EVE-NG lab and connection refused occurs when pyATS try to connect to my devices.
The hostname matches the hostname in the testbed.yaml file. Also I try to create new environment, but have the same result.
If you try to connect manually via ssh client it works, python/nornir scripts also work. But pyATS show me "ssh: connect to host port 22: Connection refused" for all atempts for all devices.

Re: pyATS connection refused

fracjackmac — Mon, 20 Dec 2021 20:06:55 GMT

Hello @vsfomin,

Here's something you can try to get more information about why SSH is failing...

Go to your YAML testbed file, select a device, and modify it as follows:

Replace

protocol: ssh

with

command: "ssh -v <device-ip-addr> -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null"

Replace <device-ip-addr> with the IP address of the device that was modified in the YAML testbed file.

Save your YAML testbed file changes, then bring up a Python interactive shell session and enter the following:

from pyats.topology import loader
tb_file = <testbed-file-name>
tb = loader.load(tb_file)
dev = tb.devices["<device-name>"]
dev.connect()

Replace <testbed-file-name> with the name of your YAML testbed file.

Replace <device-name> with the name of the device that you modified.

The output from the dev.connect() command should provide you with verbose output from the SSH -v command and provide a clue regarding why the connection request is being refused.

Please post the output from your "dev.connect()" attempt here if you require additional assistance working through this issue.

fjm

@ittybittypacket

Re: pyATS connection refused

fracjackmac — Mon, 20 Dec 2021 20:04:44 GMT

Hi @vsfomin,

One additional thought occurred to me after replying -- are there access lists on the devices that are preventing you from logging in from the host that you're running Python on?

fjm

@ittybittypacket

Re: pyATS connection refused

vsfomin — Tue, 21 Dec 2021 06:41:12 GMT

This is output what I have. I get this result whether I change protocol: ssh to command: ssh -v... or not.

Traceback (most recent call last):
  File "/home/vsfomin/python-scripts/pyats/test.py", line 3, in <module>
    tb = loader.load(tb_file)
  File "/home/vsfomin/.local/lib/python3.6/site-packages/pyats/topology/loader/__init__.py", line 44, in load
    return TestbedFileLoader(**init_kwargs).load(loadable, **kwargs)
  File "src/pyats/utils/yaml/loader.py", line 116, in pyats.utils.yaml.loader.Loader.load
  File "src/pyats/topology/loader/base.py", line 170, in pyats.topology.loader.base.TestbedFileLoader.load_arbitrary
  File "src/pyats/utils/yaml/loader.py", line 247, in pyats.utils.yaml.loader.Loader.load_arbitrary
pyats.utils.yaml.exceptions.LoadError: Content of 'testbed1.yml' failed to load into a dict.
Got: testbed1.yml

My testbed file is:

devices:
  SW-PD-01:
    connections:
      cli:
        ip: 10.0.240.171
        command: "ssh -v 10.0.240.171 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null"
    credentials:
      default:
        password: secretpassword
        username: admin
      enable:
        password: '%ASK{}'
    os: iosxe
    type: iosxe
  SW-PD-02:
    connections:
      cli:
        ip: 10.0.240.172
        command: "ssh -v 10.0.240.172 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null"
    credentials:
      default:
        password: secretpassword
        username: admin
      enable:
        password: '%ASK{}'
    os: iosxe
    type: iosxe
  SW-PD-03:
    connections:
      cli:
        ip: 10.0.240.173
        command: "ssh -v 10.0.240.173 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null"
    credentials:
      default:
        password: secretpassword
        username: admin
      enable:
        password: cisco
    os: iosxe
    type: iosxe

my script test.py which I start to test if it works or not:

from pyats.topology import loader
tb_file = "testbed1.yml"
tb = loader.load(tb_file)
dev = tb.devices["SW-PD-01"]
dev.connect()

No any access-list on these devices.

Re: pyATS connection refused

fracjackmac — Tue, 21 Dec 2021 07:34:22 GMT

Hello @vsfomin,

Thanks for the feedback.

I suspect there is an issue with the YAML file, but the issue is not obvious because I copied and pasted the information you provided into a file, launched a Python shell, and imported the testbed file without an issue -- no errors similar to what you show above.

The shell did ask for "enable passwords" for two of the devices, but did NOT generate errors, even when I attempted to connect to one of the devices.

Here's the output showing the Python session:

(tvenv) [fjm@myServer .tmp]$ python
Python 3.8.0 (default, Apr 28 2020, 13:27:51) 
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> from pyats.topology import loader
>>> tb_file = "testbed1.yml"
Enter enable password for device SW-PD-01: 
Enter enable password for device SW-PD-02: 
>>> dev = tb.devices["SW-PD-01"]
>>> dev.connect()

2021-12-21 01:03:51,334: %UNICON-INFO: +++ SW-PD-01 logfile /tmp/SW-PD-01-cli-20211221T010351333.log +++

2021-12-21 01:03:51,335: %UNICON-INFO: +++ Unicon plugin iosxe +++
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to 10.0.240.171 [10.0.240.171] port 22.


2021-12-21 01:03:51,349: %UNICON-INFO: +++ connection to spawn: ssh -l admin -v 10.0.240.171 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null, id: 139941292532256 +++

2021-12-21 01:03:51,349: %UNICON-INFO: connection to SW-PD-01

This line in your output tells us that the YAML file wasn't loaded properly:

pyats.utils.yaml.exceptions.LoadError: Content of 'testbed1.yml' failed to load into a dict.

After loading the YAML file successfully, you should be able to "ask" Python to display the dictionary that is built from the data: tb.__dict__

Here's the dictionary that was built when I loaded your YAML file:

>>> tb.__dict__
{'name': 'testbed1', 'alias': 'testbed1', 'servers': AttrDict({}), 'custom': AttrDict({}), 'credentials': Credentials(NestedAttrDict({})), 'devices': TopologyDict({'SW-PD-01': <Device SW-PD-01 at 0x7f46a24794f0>, 'SW-PD-02': <Device SW-PD-02 at 0x7f46a2479400>, 'SW-PD-03': <Device SW-PD-03 at 0x7f46a2479700>}), 'tacacs': AttrDict({'login_prompt': 'login:', 'password_prompt': 'Password:', 'username': 'vsuri001'}), 'passwords': AttrDict({'tacacs': 'lab', 'enable': 'lab', 'line': 'lab', 'linux': 'lab'}), 'clean': AttrDict({}), 'testbed_file': 'testbed1.yml', 'ipv4_cache': <genie.conf.base.utils.IPv4InterfaceCache object at 0x7f46a2bce2e0>, 'ipv6_cache': <genie.conf.base.utils.IPv6InterfaceCache object at 0x7f46a2bb0790>, 'mac_cache': <genie.conf.base.utils.MACCache object at 0x7f46a2bce7c0>, 'raw_config': {'devices': {'SW-PD-01': {'connections': {'cli': {'ip': '10.0.240.171', 'command': 'ssh -v 10.0.240.171 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null'}}, 'credentials': {'default': {'password': 'secretpassword', 'username': 'admin'}, 'enable': {'password': '%ASK{}'}}, 'os': 'iosxe', 'type': 'iosxe'}, 'SW-PD-02': {'connections': {'cli': {'ip': '10.0.240.172', 'command': 'ssh -v 10.0.240.172 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null'}}, 'credentials': {'default': {'password': 'secretpassword', 'username': 'admin'}, 'enable': {'password': '%ASK{}'}}, 'os': 'iosxe', 'type': 'iosxe'}, 'SW-PD-03': {'connections': {'cli': {'ip': '10.0.240.173', 'command': 'ssh -v 10.0.240.173 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null'}}, 'credentials': {'default': {'password': 'secretpassword', 'username': 'admin'}, 'enable': {'password': 'cisco'}}, 'os': 'iosxe', 'type': 'iosxe'}}, 'testbed': {'testbed_file': 'testbed1.yml', 'name': 'testbed1'}, 'topology': {}}}
>>>

You won't be able to attach to any devices until you can successfully load the YAML file.

What version of pyATS are you using? Would you please provide the output from this command? pyats version check

I'm running an older version of pyATS -- 21.6, so perhaps there's an issue w/ the version that you're using.

There is a Webex Space where you can get help directly from the development team.

Here's the URL to join the Webex space and raise your question.

Re: pyATS connection refused

vsfomin — Tue, 21 Dec 2021 08:08:23 GMT

$ pyats version check
You are currently running pyATS version: 21.8.1
Python: 3.6.8 [64bit]

Package Version
---------------------------- -------
genie 21.8.2
genie.libs.clean 21.8
genie.libs.conf 21.8
genie.libs.filetransferutils 21.8
genie.libs.health 21.8.1
genie.libs.ops 21.8
genie.libs.parser 21.8
genie.libs.robot 21.8
genie.libs.sdk 21.8
genie.telemetry 21.8
genie.trafficgen 21.8.1
pyats 21.8.1
pyats.aereport 21.8.1
pyats.aetest 21.8.1
pyats.async 21.8.1
pyats.connections 21.8.1
pyats.contrib 21.8
pyats.datastructures 21.8.1
pyats.easypy 21.8.1
pyats.kleenex 21.8.1
pyats.log 21.8.1
pyats.reporter 21.8.1
pyats.results 21.8.1
pyats.robot 21.8.1
pyats.tcl 21.8.1
pyats.topology 21.8.1
pyats.utils 21.8.1
unicon 21.8
unicon.plugins 21.8

I've try to create new env and install pyats 21.6, copied testbed file and test.py script to new directory

2021-12-21 03:00:16,974: %UNICON-INFO: connection to SW-PD-01 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: configuration requests final Match pass debug1: re-parsing configuration debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: Connecting to 10.0.240.171 [10.0.240.171] port 22. debug1: connect to address 10.0.240.171 port 22: Connection refused ssh: connect to host 10.0.240.171 port 22: Connection refused Traceback (most recent call last): File "src/unicon/statemachine/statemachine.py", line 727, in unicon.statemachine.statemachine.StateMachine.go_to File "src/unicon/statemachine/statetransition.py", line 478, in unicon.statemachine.statetransition.AnyStateTransition.do_transitions File "src/unicon/eal/dialogs.py", line 459, in unicon.eal.dialogs.Dialog.process File "src/unicon/eal/dialog_processor.py", line 321, in unicon.eal.dialog_processor.SimpleDialogProcessor.process File "src/unicon/eal/dialog_processor.py", line 235, in unicon.eal.dialog_processor.SimpleDialogProcessor.expect_eval_statements File "/home/vsfomin/.local/lib/python3.6/site-packages/unicon/plugins/iosxe/cat9k/statements.py", line 29, in wrapper return func(spawn) File "/home/vsfomin/.local/lib/python3.6/site-packages/unicon/plugins/generic/statements.py", line 41, in connection_refused_handler raise Exception('Connection refused to device %s' % (str(spawn))) Exception: Connection refused to device spawn: ssh -l admin -v 10.0.240.171 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null, id: 140179752518880 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "src/unicon/bases/connection.py", line 757, in unicon.bases.connection.Connection.connect File "src/unicon/bases/routers/connection_provider.py", line 190, in unicon.bases.routers.connection_provider.BaseSingleRpConnectionProvider.connect File "src/unicon/bases/routers/connection_provider.py", line 217, in unicon.bases.routers.connection_provider.BaseSingleRpConnectionProvider.establish_connection File "src/unicon/statemachine/statemachine.py", line 730, in unicon.statemachine.statemachine.StateMachine.go_to unicon.core.errors.StateMachineError: Failed while bringing device to "any" state The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/vsfomin/python-scripts/pyats-21.6/test.py", line 5, in <module> dev.connect() File "src/pyats/connections/manager.py", line 453, in pyats.connections.manager.ConnectionManager.connect File "src/unicon/bases/connection.py", line 763, in unicon.bases.connection.Connection.connect unicon.core.errors.ConnectionError: failed to connect to SW-PD-01 Failed while bringing device to "any" state

Re: pyATS connection refused

fracjackmac — Tue, 21 Dec 2021 14:16:42 GMT

Hello @vsfomin,

I'm glad to see that the older environment (21.6) loaded the testbed file and now produces the same result that you originally saw.

I checked w/ the folks in the Webex pyATS Community space regarding the failure to properly load the YAML file into the 21.8.1 environment, and the suggestion came back that you may have kicked off the Python session from a directory that did not include the YAML file.

To avoid an issue like this, I always call out the full pathname to the testbed file.

Back to the login failure...

Time to figure out why the device is not allowing you in.

Please turn on debugging for AAA Authentication and SSH at the device to determine why the device is not accepting the login.

debug aaa authentication

debug ip ssh detail

terminal monitor

Here's what I received on a successful login to an IOS-based device with the two debug commands on plus terminal monitor:

Dec 21 08:52:32.368 EST: SSH1: starting SSH control process
Dec 21 08:52:32.368 EST: SSH1: sent protocol version id SSH-2.0-Cisco-1.25
Dec 21 08:52:32.368 EST: SSH1: protocol version id is - SSH-2.0-Sun_SSH_1.1.8
Dec 21 08:52:32.368 EST: SSH2 1: Server certificate trustpoint not found. Skipping hostkey algo = x509v3-ssh-rsa
Dec 21 08:52:32.368 EST: SSH2 1: kexinit sent: hostkey algo = ssh-rsa
Dec 21 08:52:32.368 EST: SSH2 1: kexinit sent: encryption algo = aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
Dec 21 08:52:32.368 EST: SSH2 1: kexinit sent: mac algo = hmac-sha1,hmac-sha1-96
Dec 21 08:52:32.368 EST: SSH2 1: SSH2_MSG_KEXINIT sent
Dec 21 08:52:32.376 EST: SSH2 1: SSH2_MSG_KEXINIT received
Dec 21 08:52:32.376 EST: SSH2 1: kex: client->server enc:aes256-ctr mac:hmac-sha1
Dec 21 08:52:32.376 EST: SSH2 1: kex: server->client enc:aes256-ctr mac:hmac-sha1
Dec 21 08:52:32.376 EST: SSH2 1: Using hostkey algo = ssh-rsa
Dec 21 08:52:32.376 EST: SSH2 1: Using kex_algo = diffie-hellman-group-exchange-sha1
Dec 21 08:52:32.576 EST: SSH2 1: SSH2_MSG_KEX_DH_GEX_REQUEST received
Dec 21 08:52:32.576 EST: SSH2 1: Range sent by client is - 1024 < 4096 < 8192
Dec 21 08:52:32.576 EST: SSH2 1: Modulus size established : 4096 bits
Dec 21 08:52:32.952 EST: SSH2 1: expecting SSH2_MSG_KEX_DH_GEX_INIT
Dec 21 08:52:32.952 EST: SSH2 1: SSH2_MSG_KEXDH_INIT received
Dec 21 08:52:33.412 EST: SSH2: kex_derive_keys complete
Dec 21 08:52:33.412 EST: SSH2 1: SSH2_MSG_NEWKEYS sent
Dec 21 08:52:33.412 EST: SSH2 1: waiting for SSH2_MSG_NEWKEYS
Dec 21 08:52:33.420 EST: SSH2 1: SSH2_MSG_NEWKEYS received
Dec 21 08:52:33.620 EST: SSH2 1: Authentications that can continue = publickey,keyboard-interactive,password
Dec 21 08:52:33.620 EST: SSH2 1: Using method = none
Dec 21 08:52:33.620 EST: SSH2 1: Authentications that can continue = publickey,keyboard-interactive,password
Dec 21 08:52:33.784 EST: SSH2 1: Using method = keyboard-interactive
Dec 21 08:52:33.784 EST: AAA/BIND(0000002C): Bind i/f
Dec 21 08:52:33.784 EST: AAA/AUTHEN/LOGIN (0000002C): Pick method list 'default'
Dec 21 08:52:45.717 EST: SSH2 1: authentication successful for vsuri001
Dec 21 08:52:45.717 EST: SSH2 1: channel open request
Dec 21 08:52:45.717 EST: SSH2 1: pty-req request
Dec 21 08:52:45.717 EST: SSH2 1: setting TTY - requested: height 45, width 142; set: height 45, width 142
Dec 21 08:52:45.717 EST: SSH2 1: shell request
Dec 21 08:52:45.717 EST: SSH2 1: shell message received
Dec 21 08:52:45.717 EST: SSH2 1: starting shell for vty

router01-lab-03#show debug
General OS:
AAA Authentication debugging is on

SSH:
ssh detail messages debugging is on

Hopefully something obvious will pop up to tell you why you're being rejected.

fjm

@ittybittypacket