ssh to leaf from apic

[fd_case17]

Hi,

have a fabric up & running,

i can ssh to all leaf & spine from controller using infra vlan.

but for 2 leafs it's not working , ping is ok,

do I have to activate something on these chassis

Do I need tac support to activate ssh deamon ?

thanks for your support.

FD.

[Tomas de Leon]

Please clarify:

• SSH or ATTACH to "all" leaf & spines from APIC work (no issues)
• SSH to OOB or INB management fails? To all leaf\Spines or just some leaf nodes?

Please check the following:

• from APIC, "acidiag fnvread".  Check "state" for leaf\spine nodes
• "ps -aux | grep ssh" on the problem leaf nodes (console access)
• If issues with management interfaces, make sure each Node (APIC, Leaf, & Spine) have Node Management addresses configured and applied to the all nodes.
• Also, verify that under the fabric policies\pod policies\management access that SSH is enabled.

Thanks

T.

[fd_case17]

Hi,

Please clarify:

• SSH or ATTACH to "all" leaf & spines from APIC work (no issues except for two leafs)
• SSH to OOB or INB management fails? To all leaf\Spines or just some leaf nodes? ( oob /INb not yet configured, OOB only configured for APIC, only infra vlan is configured,  )

will test tomorrow,

thanks!

[fd_case17]

Hi,

on leaf with ssh ok:

root      3612  0.0  0.0   5968   968 ?        Ss   Apr27   0:04 /usr/sbin/sshd -f /etc/ssh/sshd_config_external
root      3614  0.0  0.0   5968   968 ?        Ss   Apr27   0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config_local
root      4881  0.0  0.0 584836  8008 ?        Ss   Apr27   0:02 /isan/bin/psshelper -s 219
root      4888  0.0  0.0 584836  7940 ?        Ss   Apr27   0:02 /isan/bin/psshelper -s 398
root     28768  0.1  0.0   6628  2588 ?        Ss   12:53   0:00 sshd: admin [priv]                               
ishell   30297  0.0  0.0   6628  1372 ?        S    12:53   0:00 sshd: admin@pts/0                                
admin    31103  0.0  0.0   2284   624 pts/0    S+   12:54   0:00 grep ssh

on leaf with ssh ko:

root      5271  0.0  0.0 584836  8052 ?        Ss   Apr27   0:02 /isan/bin/psshelper -s 219
root      5277  0.0  0.0 584836  7928 ?        Ss   Apr27   0:02 /isan/bin/psshelper -s 398
admin    20947  0.0  0.0   2284   624 ttyS0    S+   12:53   0:00 grep ssh

I try to start service

# /etc/init.d/sshd start
*** Running INXOS PE IFC image  ***
sed: couldn't open temporary file /etc/ssh/sedbg9G4X: Read-only file system

Do i  need to contact Tac for this ?

thanks!

FD.

[fd_case17]

try to disabled/enabled via pod Policy but same result

[Robert Burns]

Have you tried rebooting one of the "problem" leafs? If you'd like to RCA this issue, grab a show tech support bundle for the problem leafs, if not, see if the reboot fixes the issue.

Robert

[fd_case17]

Hi,

regenerate ssh key pairs works for me

Thanks !

FD.

[Tomas de Leon]

Please run on the problem leaf:

leaf# acidiag restart xinetd

leaf# "ps -aux | grep ssh"

Thanks

T.

[Tomas de Leon · Accepted solution]

FD,

I believe I know what the issue is and the solution.  The solution will require ROOT access which can only be granted by a TAC engineer.

Please open a Cisco TAC Case -  Please use a description like

ACI Leaf node - "sshd" process not running.  Please check my rsa & dsa ssh keys

Here is a summary of issue and resolution:

PROBLEM DESCRIPTION:
--------------------
Customer can no longer SSH to the Leaf node(s) from the APIC (Infra-Band) or externally from another device to the OOB or In-Band management addresses.

PLAN OF ACTION:
---------------

- Since SSH is not working, access the Leaf node(s) via a "Console" Connection.

- Check & verify the "sshd" is indeed not working. Use the command: "ps aux | grep ssh"

Output from Working Node:

leaf1# ps aux | grep ssh
root 3632 0.0 0.0 5968 968 ? Ss Apr11 1:12 /usr/sbin/sshd -f /etc/ssh/sshd_config_external
root 3634 0.0 0.0 5968 968 ? Ss Apr11 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config_local
root 6347 0.0 0.0 584836 8148 ? Ss Apr11 0:35 /isan/bin/psshelper -s 219
root 6353 0.0 0.0 584836 7952 ? Ss Apr11 0:35 /isan/bin/psshelper -s 398
root 15348 0.3 0.0 6628 2580 ? Ss 18:03 0:00 sshd: admin [priv]
ishell 16073 0.0 0.0 6628 1412 ? S 18:03 0:00 sshd: admin@pts/0
admin 16891 0.0 0.0 2284 620 pts/0 S+ 18:04 0:00 grep ssh

Output from Non-Working Node:

leaf1# ps aux | grep ssh
root 6347 0.0 0.0 584836 8148 ? Ss Apr11 0:35 /isan/bin/psshelper -s 219
root 6353 0.0 0.0 584836 7952 ? Ss Apr11 0:35 /isan/bin/psshelper -s 398
admin 16891 0.0 0.0 2284 620 pts/0 S+ 18:04 0:00 grep ssh

* Notice "sshd" is down.

- So we need to restart "sshd". Normally, you could use the "acidiag restart xinetd" to restart "sshd". Try this command first and then repeat "ps aux | grep ssh" to see if "sshd" restarted.

If "acidiag restart xinetd" fails to restart "sshd" or you get the following error when running the command:

"leaf1# acidiag restart xinetd

404 Not Found
Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again."

Then, you woud have to look into different causes.

- Next Step will be to access "root" user to try to manually restart "sshd". You will need to open a Cisco TAC Case so that a Cisco Support Engineer can provide you a temporary "root" password and they can assist you with resolving this issue.

* Access the Leaf or another node in the fabric as "admin" user
* run "acidiag dbgtoken". This command is used to generate a temporary password token to be used with ROOT password tool. Get ROOT Password from the INSBU Tool
http://git.insieme.local/cgi-bin/generateRootPassword.py

* Login in to problem Leaf Node as "root" (i.e. since ssh is not working, you must use the console access). Use the root password string from the tool for the password.

- As root, try restarting "sshd" manually. Use command: "/etc/init.d/sshd start"

leaf1# /etc/init.d/sshd restart

*** Running INXOS PE IFC image ***
Restoring saved ssh keys
key_load_public: invalid format
Could not load host key: /etc/ssh/ssh_host_rsa_key
key_load_public: invalid format
Could not load host key: /etc/ssh/ssh_host_dsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key
Disabling protocol version 2. Could not load host key
sshd: no hostkeys available -- exiting.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note: this points to the issue why SSH is failing. The ssh keys are invalid and cannot be loaded. As a result, the ssh keys need to be fixed.

- As "root" user run the following commands

Customer could not SSH into Leaf node(s) because the SSH keys were 0 byte files.
Found that normally after a reload, the Leaf node(s) will copy the SSH keys from /mnt/pss/ssh/ to /etc/ssh/.

(root)leaf1# cd /etc/ssh
(root)leaf1# mv ssh_host_dsa_key ssh_host_dsa_key.old
(root)leaf1# mv ssh_host_dsa_key.pub ssh_host_dsa_key.pub.old
(root)leaf1# mv ssh_host_rsa_key ssh_host_rsa_key.old
(root)leaf1# mv ssh_host_rsa_key.pub ssh_host_rsa_key.pub.old

Generate new keys for both DSA and RSA using (no passphrase):
(leave the passphrase blank)

(root)leaf1# ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
(root)leaf1# ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key

(root)leaf1# cp /etc/ssh/ssh_host_dsa_key /mnt/pss/ssh/ssh_host_dsa_key
(root)leaf1# cp /etc/ssh/ssh_host_dsa_key.pub /mnt/pss/ssh/ssh_host_dsa_key.pub
(root)leaf1# cp /etc/ssh/ssh_host_rsa_key /mnt/pss/ssh/ssh_host_rsa_key
(root)leaf1# cp /etc/ssh/ssh_host_rsa_key.pub /mnt/pss/ssh/ssh_host_rsa_key.pub

- Reload Leaf node(s)

- Check & verify that SSH processes are running and that SSH is working on the leaf; verify that we can SSH from the APIC to the leaf node(s)

- Check & verify the "sshd" is working. Use the command: "ps aux | grep ssh". Also verify you can ssh from APIC to leaf node(s).

Output from Leaf node after Reload:

leaf1# ps aux | grep ssh
root 3632 0.0 0.0 5968 968 ? Ss Apr11 1:12 /usr/sbin/sshd -f /etc/ssh/sshd_config_external
root 3634 0.0 0.0 5968 968 ? Ss Apr11 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config_local
root 6347 0.0 0.0 584836 8148 ? Ss Apr11 0:35 /isan/bin/psshelper -s 219
root 6353 0.0 0.0 584836 7952 ? Ss Apr11 0:35 /isan/bin/psshelper -s 398
root 15348 0.3 0.0 6628 2580 ? Ss 18:03 0:00 sshd: admin [priv]
ishell 16073 0.0 0.0 6628 1412 ? S 18:03 0:00 sshd: admin@pts/0
admin 16891 0.0 0.0 2284 620 pts/0 S+ 18:04 0:00 grep ssh

Note: If SSH continues to fail after reloading Leaf node(s) but sshd is running as expected on the Leaf node(s); You will most likely need to remove the "key & IP address" from the APICs /home/admin/.ssh/known_hosts

Thanks

T.