PyATS Genie connect() function can't ssh to ASR9K with interactive login.

[vkoul]

Hey everyone I have an issue getting genie to connect to a production ASR9K. I setup my testbed.yaml using 'genie create testbed '. The rationale was to get a prompt whenever I attempt to login since we are using an authenticator. The yaml file ended up as follows:

devices:
 access-r.lim:
   connections:
     cli:
       ip: 10.20.0.10
       protocol: ssh
   credentials:
     default:
       password: '%ASK{}'
       username: vkoulermou
     enable:
       password: '%ASK{}'
   os: iosxr
   type: iosxr

I get the MOTD and then the 'Password:' (twice) returned in the console and finally an exception which ends up with 'Failed while bringing device to "any" state'. Even if i try to paste the password when prompted the PyCharm console returns a notification saying "Previous command is still running.....". Native bash console returns what I tried to paste back to the python console prompt

P.S. I can connect to the ASR9K via SSH.
Syslog returns:

RP/0/RSP0/CPU0:Apr 20 19:19:54.383 UTC: SSHD_[66063]: %SECURITY-SSHD-4-INFO_FAILURE : Failed authentication attempt by user 'vkoulermou' from '172.16.20.10' on 'vty1' <-Genie
RP/0/RSP0/CPU0:Apr 20 19:19:54.550 UTC: SSHD_[66063]: %SECURITY-SSHD-3-ERR_GENERAL : Failed to receive User authentication request <-Genie
RP/0/RSP0/CPU0:Apr 20 19:43:44.006 UTC: SSHD_[67113]: %SECURITY-SSHD-6-INFO_SUCCESS : Successfully authenticated user 'vkoulermou' from '172.16.20.10' on 'vty0'(cipher 'aes128-ctr', mac 'hmac-sha2-256') <- Me

[omz · Accepted solution]

Hi

Can you test with genie cli?

genie parse "show version" --testbed-file nexus_testbed.yaml -vvv

to see if you get more info.. 

validate your testbed file with - 

pyats validate testbed <test file name>

[vkoul]

Thanks for shinning some light here omz.

When I tried your suggestion I realized that the "interactive login" happens before Genie calls the connect() function

Output when I enter a password at the CLI prompt

vkoulermou$ genie parse "show version" --testbed-file testbed.yaml -vvv
loading yaml file /Users/vkoulermou/Automate-ACL/testbed.yaml
Deriving testbed name ...
Enter default password for device access-r.lim:       # Did not enter password hereEnter enable password for device access-r.lim:        # Did not enter password here
Using a default key as pyATS configuration under [secrets] string.key was not specified.
Using a default key as pyATS configuration under [secrets] string.key was not specified.
Using a default key as pyATS configuration under [secrets] string.key was not specified.
Using a default key as pyATS configuration under [secrets] string.key was not specified.
Credential credentials not found, trying default ...
Credential credentials not found, trying default ...
[2020-04-21 00:09:21,041] +++ access-r.lim logfile /tmp/access-r_lim-cli-20200421T000921040.log +++
[2020-04-21 00:09:21,041] +++ Unicon plugin iosxr +++

C
***********************************************************************
* MOTD here                                                           *
***********************************************************************
[2020-04-21 00:09:22,269] +++ connection to spawn: ssh -l vkoulermou 10.20.0.10, id: 4608966176 +++
[2020-04-21 00:09:22,270] connection to access-r.lim
***********************************************************************
* MOTD continutes here                                                *
***********************************************************************

You are connected to: 

****** access-r.lim ******


Password: Using a default key as pyATS configuration under [secrets] string.key was not specified.

Password: 111111     # Password entered here and returned my input at the console after exception
Traceback (most recent call last):
  File "src/unicon/statemachine/statemachine.py", line 719, in unicon.statemachine.statemachine.StateMachine.go_to
  File "src/unicon/statemachine/statetransition.py", line 486, in unicon.statemachine.statetransition.AnyStateTransition.do_transitions
  File "src/unicon/eal/dialogs.py", line 440, in unicon.eal.dialogs.Dialog.process
  File "src/unicon/eal/dialog_processor.py", line 301, in unicon.eal.dialog_processor.SimpleDialogProcessor.process
  File "src/unicon/eal/dialog_processor.py", line 238, in unicon.eal.dialog_processor.SimpleDialogProcessor.expect_eval_statements
  File "/Users/vkoulermou/Automate-ACL/lib/python3.8/site-packages/unicon/plugins/generic/statements.py", line 209, in password_handler
    credential = get_current_credential(context=context, session=session)
  File "/Users/vkoulermou/Automate-ACL/lib/python3.8/site-packages/unicon/plugins/utils.py", line 79, in get_current_credential
    raise CredentialsExhaustedError(
unicon.core.errors.CredentialsExhaustedError: The following credentials have been tried without success : ['default']

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "src/unicon/bases/connection.py", line 623, in unicon.bases.connection.Connection.connect
  File "src/unicon/bases/routers/connection_provider.py", line 141, in unicon.bases.routers.connection_provider.BaseSingleRpConnectionProvider.connect
  File "src/unicon/bases/routers/connection_provider.py", line 163, in unicon.bases.routers.connection_provider.BaseSingleRpConnectionProvider.establish_connection
  File "src/unicon/statemachine/statemachine.py", line 722, in unicon.statemachine.statemachine.StateMachine.go_to
unicon.core.errors.StateMachineError: Failed while bringing device to "any" state

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "src/pyats/cli/base.py", line 202, in pyats.cli.base.Command.main
  File "src/genie/cli/commands/parser.py", line 174, in genie.cli.commands.parser.ParserCommand.run
  File "src/genie/conf/base/device.py", line 597, in genie.conf.base.device.Device.connect
  File "src/pyats/connections/manager.py", line 334, in pyats.connections.manager.ConnectionManager.connect
  File "src/unicon/bases/connection.py", line 629, in unicon.bases.connection.Connection.connect
unicon.core.errors.ConnectionError: failed to connect to access-r.lim
Failed while bringing device to "any" state


failed to connect to access-r.lim
Failed while bringing device to "any" state

vkoulermou$ 111111
-bash: 111111: command not found
vkoulermou$

Genie works when I enter the password at  the "Enter default password for device access-r.lim:" prompt:

vkoulermou$ genie parse "show version" --testbed-file testbed.yaml -vvv
loading yaml file /Users/vkoulermou/Automate-ACL/testbed.yaml
Deriving testbed name ...
Enter default password for device access-r.lim: # Password entered here 
Enter enable password for device access-r.lim: # Password entered here 
Using a default key as pyATS configuration under [secrets] string.key was not specified.
Using a default key as pyATS configuration under [secrets] string.key was not specified.
Using a default key as pyATS configuration under [secrets] string.key was not specified.
Using a default key as pyATS configuration under [secrets] string.key was not specified.
Credential credentials not found, trying default ...
Credential credentials not found, trying default ...
[2020-04-21 00:07:05,342] +++ access-r.lim logfile /tmp/access-r_lim-cli-20200421T000705341.log +++
[2020-04-21 00:07:05,342] +++ Unicon plugin iosxr +++

C
***********************************************************************
* MOTD here                                                           *
*                                                                     *
* MOTD here                                                           *
***********************************************************************
[2020-04-21 00:07:06,618] +++ connection to spawn: ssh -l vkoulermou 10.20.0.10, id: 4549537312 +++
[2020-04-21 00:07:06,620] connection to access-r.lim
***********************************************************************
* MOTD continues here                                                 *
*                                                                     *
* MOTD continues here                                                 *
***********************************************************************

You are connected to: 

****** access-r.lim ******


Password: Using a default key as pyATS configuration under [secrets] string.key was not specified.



RP/0/RSP0/CPU0:access-r.lim#
[2020-04-21 00:07:08,066] +++ initializing handle +++
[2020-04-21 00:07:08,067] +++ access-r.lim: executing command 'terminal length 0' +++
terminal length 0
Mon Apr 20 21:07:08.088 UTC
RP/0/RSP0/CPU0:access-r.lim#
[2020-04-21 00:07:08,495] +++ access-r.lim: executing command 'terminal width 0' +++
terminal width 0
Mon Apr 20 21:07:08.520 UTC
RP/0/RSP0/CPU0:access-r.lim#
  0%|                                                                                                                                                         | 0/1 [00:00<?, ?it/s]TclHistory is enabled.
TclHistory is enabled.
TclHistory is enabled.
TclHistory is enabled.
TclHistory is enabled.
[2020-04-21 00:07:09,410] +++ access-r.lim: executing command 'show version' +++
show version
Mon Apr 20 21:07:09.435 UTC

Cisco IOS XR Software, Version 6.4.2
Copyright (c) 2013-2017 by Cisco Systems, Inc.

Build Information:
 Built By     : radharan
 Built On     : Fri Jul 27 02:55:47 PDT 2018
 Build Host   : iox-ucs-003
 Workspace    : /auto/srcarchive17/prod/6.4.2/asr9k-x64/ws
 Version      : 6.4.2
 Location     : /opt/cisco/XR/packages/

cisco ASR9K () processor 
System uptime is 46 weeks, 11 hours, 33 minutes

RP/0/RSP0/CPU0:access-r.lim#
{
  "device_family": "ASR9K",
  "operating_system": "IOSXR",
  "software_version": "6.4.2",
  "uptime": "46 weeks, 11 hours, 33 minutes"
}
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.03it/s]

vkoulermou$

So it seems I cannot debug connections via console, since my commands may be have a time difference between the time I load() the .yaml and the time I connect(), during which the authenticator will change.

>>>tb = load('testbed.yaml')Warning: Password input may be echoed.
Enter default password for device access-r.lim: >?   # Did not enter password here
Warning: Password input may be echoed.
Enter enable password for device access-r.lim: >?    # Did not enter password here
>>>access_r_lim = tb.devices['access-r.lim']
# I did not enter a password
# since authenticator will change
# until I execute connect() function
#
>>>access_r_lim.connect()
[2020-04-21 00:37:52,268] +++ access-r.lim logfile /tmp/access-r_lim-cli-20200421T003752262.log +++
[2020-04-21 00:37:52,268] +++ Unicon plugin iosxr +++
C
***********************************************************************
* MOTD here                                                           *
*                                                                     *
* MOTD here                                                           *
***********************************************************************
***********************************************************************
* MOTD here                                                           *
*                                                                     *
* MOTD here                                                           *
***********************************************************************
[2020-04-21 00:37:53,588] +++ connection to spawn: ssh -l vkoulermou 10.20.0.10, id: 4658969376 +++
[2020-04-21 00:37:53,589] connection to access-r.lim
***********************************************************************
* MOTD continues here                                                 *
*                                                                     *
* MOTD continues here                                                 *
***********************************************************************
You are connected to: 
****** access-r.lim ******
Password: 
Password: Traceback (most recent call last): # Password entered here but returned my input at the console after exception  File "src/unicon/statemachine/statemachine.py", line 719, in unicon.statemachine.statemachine.StateMachine.go_to
  File "src/unicon/statemachine/statetransition.py", line 486, in unicon.statemachine.statetransition.AnyStateTransition.do_transitions
  File "src/unicon/eal/dialogs.py", line 440, in unicon.eal.dialogs.Dialog.process
  File "src/unicon/eal/dialog_processor.py", line 301, in unicon.eal.dialog_processor.SimpleDialogProcessor.process
  File "src/unicon/eal/dialog_processor.py", line 238, in unicon.eal.dialog_processor.SimpleDialogProcessor.expect_eval_statements
  File "/Users/vkoulermou/Automate-ACL/lib/python3.8/site-packages/unicon/plugins/generic/statements.py", line 209, in password_handler
    credential = get_current_credential(context=context, session=session)
  File "/Users/vkoulermou/Automate-ACL/lib/python3.8/site-packages/unicon/plugins/utils.py", line 79, in get_current_credential
    raise CredentialsExhaustedError(
unicon.core.errors.CredentialsExhaustedError: The following credentials have been tried without success : ['default']
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "src/unicon/bases/connection.py", line 623, in unicon.bases.connection.Connection.connect
  File "src/unicon/bases/routers/connection_provider.py", line 141, in unicon.bases.routers.connection_provider.BaseSingleRpConnectionProvider.connect
  File "src/unicon/bases/routers/connection_provider.py", line 163, in unicon.bases.routers.connection_provider.BaseSingleRpConnectionProvider.establish_connection
  File "src/unicon/statemachine/statemachine.py", line 722, in unicon.statemachine.statemachine.StateMachine.go_to
unicon.core.errors.StateMachineError: Failed while bringing device to "any" state
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "src/genie/conf/base/device.py", line 597, in genie.conf.base.device.Device.connect
  File "src/pyats/connections/manager.py", line 334, in pyats.connections.manager.ConnectionManager.connect
  File "src/unicon/bases/connection.py", line 629, in unicon.bases.connection.Connection.connect
unicon.core.errors.ConnectionError: failed to connect to access-r.lim
Failed while bringing device to "any" state

>>> 222222   # Returned password

So my question now is there a way to make connect() function interactive during login?

[omz]

are you looking to get an interactive device prompt? I'm not sure if you can get that ..

there is pyats interactive shell  - 

bash$ pyats shell --testbed-file our-testbed-file.yaml

https://developer.cisco.com/docs/pyats/#!connection-to-devices/device-connections

[vkoul]

Thanks for the info omz.

Basically I am looking for a way to perform the interactive login when I call the connect() function and not the load() function.

The reason is that while coding I may have different devices in a testbed.yaml (or even multiple .yaml files) loaded to different variables. So everytime I want to connect to a device to execute a command to test/parse something I have to call one function then input the authenticator code and then another two functions and do it fast before the authenticator code expires. Which seems to be quite the hassle.

>>>tb = load('testbed.yaml')
Warning: Password input may be echoed.
Enter default password for device access-r.lim: 333333
Warning: Password input may be echoed.
Enter enable password for device access-r.lim: 333333
>>>access_r_lim = tb.devices['access-r.lim']
>>>access_r_lim.connect()

Thanks anyway!

[tperiasa]

wow, nice.  you taught a lot, saved lot of time.

[omz]

tested your testbed file with sandbox

cant see any issues with genie or pyats login .. 

test with adding alias and then target that device

edit:

test with adding alias and then target that device

[Olloff]

I just had the same issue and been struggling for 2 days now and finally got it right 😁.

So just a brief background what I am using:

• Latest Ubuntu 20.04LTS
• Latest Windows 10 version 2004
• Python 3.8.2
• WSL2

My goal is to learn and play with pyATS/Genie for network automation. I created a virtual environment and installed pyATS and Genie. 

I checked if any packages were outdated:
pyats version check --outdated

Then updated the versions:
pyats version update

After this I got the sample testbed file and added my devices accordingly. I first tested with telnet devices and it work immediately. When I tested the devices with SSH enabled, that is when it all started. I started doubting my testbed file and my commands and everything. It just did not work and changed my testbed file like 12 times. I got the exact same message as you did and WHAT WAS THE ISSUE??🤷‍♂️

• My pyATS were outdated --> SOLUTION: I upgraded my pyATS: pip install pyats[full] --upgrade and restarted and it got further by showing the progress bar but it failed again. 🤦‍♂️
• Then I checked another testbed file and changed it again and it worked. 🤪

  testbed:
      name: IOS_Testbed
      credentials:
          default:
              username: admin
              password: '%ASK{}'
          enable:
              password: '%ASK{}'
  
  devices:
      <Hostname>:
          os: ios
          type: ios
          connections:
              a:
                  protocol: telnet
                  ip: 10.0.0.1
                  port: 23
              vty:
                  protocol: ssh
                  ip: 10.0.0.1

  So what did I learn?

• Do the proper complete install/upgrade of pyATS. 
• Get the right testbed file for the type of version your running.
• If the device has telnet and ssh enabled, then you need to specify both ssh and telnet details of the device in the testbed file or it will not connect.

Some links:

pyATS getting started 

Getting started with pyATS and Genie 

[gdjundev]

In my case WSL restart solved problem