04-27-2016 12:57 PM - edited 03-01-2019 04:57 AM
Hi,
have a fabric up & running,
i can ssh to all leaf & spine from controller using infra vlan.
but for 2 leafs it's not working , ping is ok,
do I have to activate something on these chassis
Do I need tac support to activate ssh deamon ?
thanks for your support.
FD.
Solved! Go to Solution.
04-28-2016 01:52 PM
FD,
I believe I know what the issue is and the solution. The solution will require ROOT access which can only be granted by a TAC engineer.
Please open a Cisco TAC Case - Please use a description like
ACI Leaf node - "sshd" process not running. Please check my rsa & dsa ssh keys
Here is a summary of issue and resolution:
PROBLEM DESCRIPTION:
--------------------
Customer can no longer SSH to the Leaf node(s) from the APIC (Infra-Band) or externally from another device to the OOB or In-Band management addresses.
PLAN OF ACTION:
---------------
- Since SSH is not working, access the Leaf node(s) via a "Console" Connection.
- Check & verify the "sshd" is indeed not working. Use the command: "ps aux | grep ssh"
Output from Working Node:
leaf1# ps aux | grep ssh
root 3632 0.0 0.0 5968 968 ? Ss Apr11 1:12 /usr/sbin/sshd -f /etc/ssh/sshd_config_external
root 3634 0.0 0.0 5968 968 ? Ss Apr11 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config_local
root 6347 0.0 0.0 584836 8148 ? Ss Apr11 0:35 /isan/bin/psshelper -s 219
root 6353 0.0 0.0 584836 7952 ? Ss Apr11 0:35 /isan/bin/psshelper -s 398
root 15348 0.3 0.0 6628 2580 ? Ss 18:03 0:00 sshd: admin [priv]
ishell 16073 0.0 0.0 6628 1412 ? S 18:03 0:00 sshd: admin@pts/0
admin 16891 0.0 0.0 2284 620 pts/0 S+ 18:04 0:00 grep ssh
Output from Non-Working Node:
leaf1# ps aux | grep ssh
root 6347 0.0 0.0 584836 8148 ? Ss Apr11 0:35 /isan/bin/psshelper -s 219
root 6353 0.0 0.0 584836 7952 ? Ss Apr11 0:35 /isan/bin/psshelper -s 398
admin 16891 0.0 0.0 2284 620 pts/0 S+ 18:04 0:00 grep ssh
* Notice "sshd" is down.
- So we need to restart "sshd". Normally, you could use the "acidiag restart xinetd" to restart "sshd". Try this command first and then repeat "ps aux | grep ssh" to see if "sshd" restarted.
If "acidiag restart xinetd" fails to restart "sshd" or you get the following error when running the command:
"leaf1# acidiag restart xinetd
404 Not Found
Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again."
Then, you woud have to look into different causes.
- Next Step will be to access "root" user to try to manually restart "sshd". You will need to open a Cisco TAC Case so that a Cisco Support Engineer can provide you a temporary "root" password and they can assist you with resolving this issue.
* Access the Leaf or another node in the fabric as "admin" user
* run "acidiag dbgtoken". This command is used to generate a temporary password token to be used with ROOT password tool. Get ROOT Password from the INSBU Tool
http://git.insieme.local/cgi-bin/generateRootPassword.py
* Login in to problem Leaf Node as "root" (i.e. since ssh is not working, you must use the console access). Use the root password string from the tool for the password.
- As root, try restarting "sshd" manually. Use command: "/etc/init.d/sshd start"
leaf1# /etc/init.d/sshd restart
*** Running INXOS PE IFC image ***
Restoring saved ssh keys
key_load_public: invalid format
Could not load host key: /etc/ssh/ssh_host_rsa_key
key_load_public: invalid format
Could not load host key: /etc/ssh/ssh_host_dsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key
Disabling protocol version 2. Could not load host key
sshd: no hostkeys available -- exiting.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note: this points to the issue why SSH is failing. The ssh keys are invalid and cannot be loaded. As a result, the ssh keys need to be fixed.
- As "root" user run the following commands
Customer could not SSH into Leaf node(s) because the SSH keys were 0 byte files.
Found that normally after a reload, the Leaf node(s) will copy the SSH keys from /mnt/pss/ssh/ to /etc/ssh/.
(root)leaf1# cd /etc/ssh
(root)leaf1# mv ssh_host_dsa_key ssh_host_dsa_key.old
(root)leaf1# mv ssh_host_dsa_key.pub ssh_host_dsa_key.pub.old
(root)leaf1# mv ssh_host_rsa_key ssh_host_rsa_key.old
(root)leaf1# mv ssh_host_rsa_key.pub ssh_host_rsa_key.pub.old
Generate new keys for both DSA and RSA using (no passphrase):
(leave the passphrase blank)
(root)leaf1# ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
(root)leaf1# ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
(root)leaf1# cp /etc/ssh/ssh_host_dsa_key /mnt/pss/ssh/ssh_host_dsa_key
(root)leaf1# cp /etc/ssh/ssh_host_dsa_key.pub /mnt/pss/ssh/ssh_host_dsa_key.pub
(root)leaf1# cp /etc/ssh/ssh_host_rsa_key /mnt/pss/ssh/ssh_host_rsa_key
(root)leaf1# cp /etc/ssh/ssh_host_rsa_key.pub /mnt/pss/ssh/ssh_host_rsa_key.pub
- Reload Leaf node(s)
- Check & verify that SSH processes are running and that SSH is working on the leaf; verify that we can SSH from the APIC to the leaf node(s)
- Check & verify the "sshd" is working. Use the command: "ps aux | grep ssh". Also verify you can ssh from APIC to leaf node(s).
Output from Leaf node after Reload:
leaf1# ps aux | grep ssh
root 3632 0.0 0.0 5968 968 ? Ss Apr11 1:12 /usr/sbin/sshd -f /etc/ssh/sshd_config_external
root 3634 0.0 0.0 5968 968 ? Ss Apr11 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config_local
root 6347 0.0 0.0 584836 8148 ? Ss Apr11 0:35 /isan/bin/psshelper -s 219
root 6353 0.0 0.0 584836 7952 ? Ss Apr11 0:35 /isan/bin/psshelper -s 398
root 15348 0.3 0.0 6628 2580 ? Ss 18:03 0:00 sshd: admin [priv]
ishell 16073 0.0 0.0 6628 1412 ? S 18:03 0:00 sshd: admin@pts/0
admin 16891 0.0 0.0 2284 620 pts/0 S+ 18:04 0:00 grep ssh
Note: If SSH continues to fail after reloading Leaf node(s) but sshd is running as expected on the Leaf node(s); You will most likely need to remove the "key & IP address" from the APICs /home/admin/.ssh/known_hosts
Thanks
T.
04-27-2016 01:27 PM
Please clarify:
Please check the following:
Thanks
T.
04-27-2016 01:35 PM
Hi,
Please clarify:
will test tomorrow,
thanks!
04-28-2016 05:57 AM
Hi,
on leaf with ssh ok:
root 3612 0.0 0.0 5968 968 ? Ss Apr27 0:04 /usr/sbin/sshd -f /etc/ssh/sshd_config_external
root 3614 0.0 0.0 5968 968 ? Ss Apr27 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config_local
root 4881 0.0 0.0 584836 8008 ? Ss Apr27 0:02 /isan/bin/psshelper -s 219
root 4888 0.0 0.0 584836 7940 ? Ss Apr27 0:02 /isan/bin/psshelper -s 398
root 28768 0.1 0.0 6628 2588 ? Ss 12:53 0:00 sshd: admin [priv]
ishell 30297 0.0 0.0 6628 1372 ? S 12:53 0:00 sshd: admin@pts/0
admin 31103 0.0 0.0 2284 624 pts/0 S+ 12:54 0:00 grep ssh
on leaf with ssh ko:
root 5271 0.0 0.0 584836 8052 ? Ss Apr27 0:02 /isan/bin/psshelper -s 219
root 5277 0.0 0.0 584836 7928 ? Ss Apr27 0:02 /isan/bin/psshelper -s 398
admin 20947 0.0 0.0 2284 624 ttyS0 S+ 12:53 0:00 grep ssh
I try to start service
# /etc/init.d/sshd start
*** Running INXOS PE IFC image ***
sed: couldn't open temporary file /etc/ssh/sedbg9G4X: Read-only file system
Do i need to contact Tac for this ?
thanks!
FD.
04-28-2016 06:04 AM
try to disabled/enabled via pod Policy but same result
04-28-2016 06:15 AM
Have you tried rebooting one of the "problem" leafs? If you'd like to RCA this issue, grab a show tech support bundle for the problem leafs, if not, see if the reboot fixes the issue.
Robert
05-02-2016 01:02 PM
Hi,
regenerate ssh key pairs works for me
Thanks !
FD.
04-28-2016 07:14 AM
Please run on the problem leaf:
leaf# acidiag restart xinetd
leaf# "ps -aux | grep ssh"
Thanks
T.
04-28-2016 01:52 PM
FD,
I believe I know what the issue is and the solution. The solution will require ROOT access which can only be granted by a TAC engineer.
Please open a Cisco TAC Case - Please use a description like
ACI Leaf node - "sshd" process not running. Please check my rsa & dsa ssh keys
Here is a summary of issue and resolution:
PROBLEM DESCRIPTION:
--------------------
Customer can no longer SSH to the Leaf node(s) from the APIC (Infra-Band) or externally from another device to the OOB or In-Band management addresses.
PLAN OF ACTION:
---------------
- Since SSH is not working, access the Leaf node(s) via a "Console" Connection.
- Check & verify the "sshd" is indeed not working. Use the command: "ps aux | grep ssh"
Output from Working Node:
leaf1# ps aux | grep ssh
root 3632 0.0 0.0 5968 968 ? Ss Apr11 1:12 /usr/sbin/sshd -f /etc/ssh/sshd_config_external
root 3634 0.0 0.0 5968 968 ? Ss Apr11 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config_local
root 6347 0.0 0.0 584836 8148 ? Ss Apr11 0:35 /isan/bin/psshelper -s 219
root 6353 0.0 0.0 584836 7952 ? Ss Apr11 0:35 /isan/bin/psshelper -s 398
root 15348 0.3 0.0 6628 2580 ? Ss 18:03 0:00 sshd: admin [priv]
ishell 16073 0.0 0.0 6628 1412 ? S 18:03 0:00 sshd: admin@pts/0
admin 16891 0.0 0.0 2284 620 pts/0 S+ 18:04 0:00 grep ssh
Output from Non-Working Node:
leaf1# ps aux | grep ssh
root 6347 0.0 0.0 584836 8148 ? Ss Apr11 0:35 /isan/bin/psshelper -s 219
root 6353 0.0 0.0 584836 7952 ? Ss Apr11 0:35 /isan/bin/psshelper -s 398
admin 16891 0.0 0.0 2284 620 pts/0 S+ 18:04 0:00 grep ssh
* Notice "sshd" is down.
- So we need to restart "sshd". Normally, you could use the "acidiag restart xinetd" to restart "sshd". Try this command first and then repeat "ps aux | grep ssh" to see if "sshd" restarted.
If "acidiag restart xinetd" fails to restart "sshd" or you get the following error when running the command:
"leaf1# acidiag restart xinetd
404 Not Found
Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again."
Then, you woud have to look into different causes.
- Next Step will be to access "root" user to try to manually restart "sshd". You will need to open a Cisco TAC Case so that a Cisco Support Engineer can provide you a temporary "root" password and they can assist you with resolving this issue.
* Access the Leaf or another node in the fabric as "admin" user
* run "acidiag dbgtoken". This command is used to generate a temporary password token to be used with ROOT password tool. Get ROOT Password from the INSBU Tool
http://git.insieme.local/cgi-bin/generateRootPassword.py
* Login in to problem Leaf Node as "root" (i.e. since ssh is not working, you must use the console access). Use the root password string from the tool for the password.
- As root, try restarting "sshd" manually. Use command: "/etc/init.d/sshd start"
leaf1# /etc/init.d/sshd restart
*** Running INXOS PE IFC image ***
Restoring saved ssh keys
key_load_public: invalid format
Could not load host key: /etc/ssh/ssh_host_rsa_key
key_load_public: invalid format
Could not load host key: /etc/ssh/ssh_host_dsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key
Disabling protocol version 2. Could not load host key
sshd: no hostkeys available -- exiting.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note: this points to the issue why SSH is failing. The ssh keys are invalid and cannot be loaded. As a result, the ssh keys need to be fixed.
- As "root" user run the following commands
Customer could not SSH into Leaf node(s) because the SSH keys were 0 byte files.
Found that normally after a reload, the Leaf node(s) will copy the SSH keys from /mnt/pss/ssh/ to /etc/ssh/.
(root)leaf1# cd /etc/ssh
(root)leaf1# mv ssh_host_dsa_key ssh_host_dsa_key.old
(root)leaf1# mv ssh_host_dsa_key.pub ssh_host_dsa_key.pub.old
(root)leaf1# mv ssh_host_rsa_key ssh_host_rsa_key.old
(root)leaf1# mv ssh_host_rsa_key.pub ssh_host_rsa_key.pub.old
Generate new keys for both DSA and RSA using (no passphrase):
(leave the passphrase blank)
(root)leaf1# ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
(root)leaf1# ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
(root)leaf1# cp /etc/ssh/ssh_host_dsa_key /mnt/pss/ssh/ssh_host_dsa_key
(root)leaf1# cp /etc/ssh/ssh_host_dsa_key.pub /mnt/pss/ssh/ssh_host_dsa_key.pub
(root)leaf1# cp /etc/ssh/ssh_host_rsa_key /mnt/pss/ssh/ssh_host_rsa_key
(root)leaf1# cp /etc/ssh/ssh_host_rsa_key.pub /mnt/pss/ssh/ssh_host_rsa_key.pub
- Reload Leaf node(s)
- Check & verify that SSH processes are running and that SSH is working on the leaf; verify that we can SSH from the APIC to the leaf node(s)
- Check & verify the "sshd" is working. Use the command: "ps aux | grep ssh". Also verify you can ssh from APIC to leaf node(s).
Output from Leaf node after Reload:
leaf1# ps aux | grep ssh
root 3632 0.0 0.0 5968 968 ? Ss Apr11 1:12 /usr/sbin/sshd -f /etc/ssh/sshd_config_external
root 3634 0.0 0.0 5968 968 ? Ss Apr11 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config_local
root 6347 0.0 0.0 584836 8148 ? Ss Apr11 0:35 /isan/bin/psshelper -s 219
root 6353 0.0 0.0 584836 7952 ? Ss Apr11 0:35 /isan/bin/psshelper -s 398
root 15348 0.3 0.0 6628 2580 ? Ss 18:03 0:00 sshd: admin [priv]
ishell 16073 0.0 0.0 6628 1412 ? S 18:03 0:00 sshd: admin@pts/0
admin 16891 0.0 0.0 2284 620 pts/0 S+ 18:04 0:00 grep ssh
Note: If SSH continues to fail after reloading Leaf node(s) but sshd is running as expected on the Leaf node(s); You will most likely need to remove the "key & IP address" from the APICs /home/admin/.ssh/known_hosts
Thanks
T.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide