Unable to connect to IOS-XR 6.5.2 with Paramiko.

[dannew]

Hi!

After upgrade to IOS-XR 6.5.2 none of my automations are able to connect to to the router. Both my ncclient and Netmiko implementations have stopped working.

Paramiko debug log ends with :

DEB [20190408-18:11:41.008] thr=2 paramiko.transport: userauth is OK

The exception I get:

paramiko.ssh_exception.AuthenticationException: Authentication timeout.

ncclient output:

ncclient.transport.errors.AuthenticationError: AuthenticationException('Authentication timeout.',)

What happened to the SSH server implementation in XR 6.5?

Has anyone else managed to get this working?

[smilstea]

What XR code were you on before?

What cipher is paramiko sending?

What error message / syslog is seen on the asr9k?

Thanks,

Sam

[dannew]

Complete paramiko debug log:

DEB [20190408-18:36:01.421] thr=2 paramiko.transport: starting thread (client mode): 0xcc71aa90
DEB [20190408-18:36:01.421] thr=2 paramiko.transport: Local version/idstring: SSH-2.0-paramiko_2.4.2
DEB [20190408-18:36:01.695] thr=2 paramiko.transport: Remote version/idstring: SSH-1.99-Cisco-2.0
INF [20190408-18:36:01.695] thr=2 paramiko.transport: Connected (version 1.99, client Cisco-2.0)
DEB [20190408-18:36:01.701] thr=2 paramiko.transport: kex algos:['ecdh-sha2-nistp521', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp256', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] client mac:['hmac-sha2-512', 'hmac-sha2-256', 'hmac-sha1'] server mac:['hmac-sha2-512', 'hmac-sha2-256', 'hmac-sha1'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False
DEB [20190408-18:36:01.701] thr=2 paramiko.transport: Kex agreed: ecdh-sha2-nistp256
DEB [20190408-18:36:01.701] thr=2 paramiko.transport: HostKey agreed: ssh-rsa
DEB [20190408-18:36:01.701] thr=2 paramiko.transport: Cipher agreed: aes128-ctr
DEB [20190408-18:36:01.701] thr=2 paramiko.transport: MAC agreed: hmac-sha2-256
DEB [20190408-18:36:01.702] thr=2 paramiko.transport: Compression agreed: none
DEB [20190408-18:36:01.744] thr=2 paramiko.transport: kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>
DEB [20190408-18:36:01.745] thr=2 paramiko.transport: Switch to new keys ...
DEB [20190408-18:36:01.747] thr=2 paramiko.transport: userauth is OK

debug ssh server:

RP/0/RSP0/CPU0:Apr 8 19:06:35.915 CEST: SSHD_[65918]: =====Authentication AAA======
RP/0/RSP0/CPU0:Apr 8 19:06:35.915 CEST: SSHD_[65918]: sshd_aaa_authenticate: Entering, method:PASSWORD
RP/0/RSP0/CPU0:Apr 8 19:06:35.915 CEST: SSHD_[65918]: (sshd_aaa_authenticate:1679) Not in kex or user auth or interactive shell - ALARM NOT ACTIVE - not clearing alarm
RP/0/RSP0/CPU0:Apr 8 19:06:35.925 CEST: SSHD_[65918]: (sshd_aaa_authenticate:1683) Not in kex or user auth or interactive shell - ALARM NOT ACTIVE - not setting alarm
RP/0/RSP0/CPU0:Apr 8 19:06:35.925 CEST: SSHD_[65918]: (sshd_aaa_authenticate:1687) Not in kex or user auth or interactive shell - ALARM NOT ACTIVE - not clearing alarm
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: (sshd_aaa_authenticate:1690) Not in kex or user auth or interactive shell - ALARM NOT ACTIVE - not setting alarm
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: sshd_aaa_authenticate: Failed to authenticate(reason:not set)
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: In ssh error setting code 26
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: update_sshd_exit_reason: exit reason updated 26
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: (sshd_authenticate_internal:2171) Not in kex or user auth or interactive shell - ALARM NOT ACTIVE - not clearing alarm
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: (sshd_authenticate_internal:2173) Not in kex or user auth or interactive shell - ALARM NOT ACTIVE - not setting alarm
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: Failed to authenticate user
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: error writing bytes to sockfd
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: In ssh error setting code 24
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: update_sshd_exit_reason: exit reason updated 24
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: (sshd_authenticate) password rc:2
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: [TID=1] In cleanup code, pid:17215870, sig rcvd:0, state:5
RP/0/RSP0/CPU0:Apr 8 19:08:35.946 CEST: SSHD_[65918]: (ssh_cleanup:567) Not in kex or user auth or interactive shell - ALARM NOT ACTIVE - not clearing alarm
RP/0/RSP0/CPU0:Apr 8 19:08:35.947 CEST: SSHD_[65918]: ssh Cleanup: No open Channels to close
RP/0/RSP0/CPU0:Apr 8 19:08:35.947 CEST: SSHD_[65918]: [TID=1] Closing connection to 172.20.115.32
RP/0/RSP0/CPU0:Apr 8 19:08:35.947 CEST: SSHD_[65918]: [TID=1] Keys are zeroised
RP/0/RSP0/CPU0:Apr 8 19:08:35.948 CEST: SSHD_[65918]: [TID=1] Inside ttylist_cleanup FUNC
RP/0/RSP0/CPU0:Apr 8 19:08:35.948 CEST: SSHD_[65918]: [TID=1] SSH ttylist has not been cleaned up
RP/0/RSP0/CPU0:Apr 8 19:08:36.076 CEST: SSHD_[1211]: Proces with pid: 17215870 exited with waitpid exit_stat=0x0
RP/0/RSP0/CPU0:Apr 8 19:08:36.076 CEST: SSHD_[1211]: CLNT SES REL: Active Entries=2, Client PID=17215870
RP/0/RSP0/CPU0:Apr 8 19:08:36.076 CEST: SSHD_[1211]: Releasing client Session id: 311,index=60
RP/0/RSP0/CPU0:Apr 8 19:08:36.076 CEST: SSHD_[1211]: Not freeing first entry, freeing last entry
RP/0/RSP0/CPU0:Apr 8 19:08:36.076 CEST: SSHD_[1211]: Non-NULL Last History entry
RP/0/RSP0/CPU0:Apr 8 19:08:36.076 CEST: SSHD_[1211]: SIGCHLD: Active clients=1, session count=1
RP/0/RSP0/CPU0:Apr 8 19:08:36.076 CEST: SSHD_[1211]: Exit status of the process pid 17215870 : 0

As the client times out and gives up, and the router is waiting for the client, the client has a shorter timeout and the connection is already gone when the server tries to say something.

[dannew]

And last known good version is 4.3.4

[tkarnani · Accepted solution]

There is an issue in 6.5.2, would you be able to install this SMU and test ?

asr9k-px-6.5.2.CSCvo17475.tar

Thanks

[dannew]

Thank you very much for your quick and accurate help!

I have now verified the SMU on our lab router and it fixes my issue.

But as the SMU requires reboot, it will take some time to implement in our production routers, is there a way to create a local user for the automation user that overrides the tacacs-user?

[tkarnani]

i'm glad to hear this works.

i will tell you that i have broken more AAA than i have fixed :)

in theory you can setup aaa to prefer local login over tacacs

aaa authentication login ssh local group radius group tacacs+

maybe this will work as a temporary work around? (commit confirmed is your friend here)

[dannew]

Well, that was terrible.

Local user works, but not TACACS. I _think_ it only tries the first choice unless it "fails", which local authentication rarely does.

I'll keep digging to see if I can find something that works.

[dannew]

Unfortunately there seems to be no way to accept BOTH tacacs and local users at the same time. I will just have to wait until we can add the SMU to the routers.