Re: NSO 5.4.2 - restart with packages reload gets timed-out, and socket binding error

swilsons · ‎02-19-2021

When trying to restart NSO with some functional packs (approx 20+) its getting timed-out. Some times getting socket binding error.

Time-out issue:

>>>>>

[root@localhost nso-5.4.2-tsdn-2.0.0-M8]# sudo /etc/init.d/ncs stop

Stopping ncs (via systemctl): Warning: ncs.service changed on disk. Run 'systemctl daemon-reload' to reload units.

[ OK ]

[root@localhost nso-5.4.2-tsdn-2.0.0-M8]# sudo cp core-fp-packages/*.tar.gz /var/opt/ncs/packages

[root@localhost nso-5.4.2-tsdn-2.0.0-M8]# sudo cp example-packages/*.tar.gz /var/opt/ncs/packages

[root@localhost nso-5.4.2-tsdn-2.0.0-M8]# sudo /etc/init.d/ncs restart-with-package-reload

Restarting ncs (via systemctl): Warning: ncs.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Job for ncs.service failed because a timeout was exceeded. See "systemctl status ncs.service" and "journalctl -xe" for details.

[FAILED]

>>>>>

Tried to increase the timeout in ncs.conf like following, but it didnt help

<cli>

<idle-timeout>PT120M</idle-timeout>

<command-timeout>infinity</command-timeout>

<ssh>

</ssh>

</cli>

<ssh>

<client-alive-interval>infinity</client-alive-interval>

<client-alive-count-max>5</client-alive-count-max>

</ssh>

2. Socket binding issue:

This is always seen after the timeout error. The NSO is not getting stopped, it's in running state for ever.

>>>>>

module/nodes/node/config/auto-rese... not exist

Feb 17 14:28:26 localhost.localdomain ncs[12356]: - Cannot bind to internal socket 127.0.0.1:4569 : address already in use

Feb 17 14:28:26 localhost.localdomain ncs[12343]: Starting ncs: Cannot bind to internal socket 127.0.0.1:4569 : address already in use

Feb 17 14:28:26 localhost.localdomain ncs[12343]: Daemon died status=20

Feb 17 14:28:26 localhost.localdomain systemd[1]: ncs.service: control process exited, code=exited status=20

Feb 17 14:28:26 localhost.localdomain systemd[1]: Failed to start LSB: NCS.

>>>>>

Couldn't stop NSO

root@localhost ncs]# netstat -lntp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:2022 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 127.0.0.1:2023 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 0.0.0.0:2024 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1149/sshd

tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 127.0.0.1:4569 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1372/master

tcp6 0 0 127.0.0.1:41003 :::* LISTEN 2232/java

tcp6 0 0 127.0.0.1:9901 :::* LISTEN 1815/java

tcp6 0 0 127.0.0.1:9902 :::* LISTEN 1815/java

tcp6 0 0 :::22 :::* LISTEN 1149/sshd

tcp6 0 0 ::1:25 :::* LISTEN 1372/master

[root@localhost ncs]#

[root@localhost ncs]# /etc/init.d/ncs stop

Stopping ncs (via systemctl): [ OK ]

[root@localhost ncs]# netstat -lntp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:2022 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 127.0.0.1:2023 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 0.0.0.0:2024 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1149/sshd

tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 127.0.0.1:4569 0.0.0.0:* LISTEN 1203/ncs.smp

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1372/master

tcp6 0 0 127.0.0.1:41003 :::* LISTEN 2232/java

tcp6 0 0 127.0.0.1:9901 :::* LISTEN 1815/java

tcp6 0 0 127.0.0.1:9902 :::* LISTEN 1815/java

tcp6 0 0 :::22 :::* LISTEN 1149/sshd

tcp6 0 0 ::1:25 :::* LISTEN 1372/master

[root@localhost ncs]# /etc/init.d/ncs status | grep status

status: started

http://cisco.com/yang/nso/custom-template-status prefix:ct-status exported to: all

http://com/cisco/cfp-common-status-codes prefix:cfp-common-status-codes exported to: all

module: core-fp-common-status-codes

namespace: http://com/cisco/cfp-common-status-codes

prefix: cfp-common-status-codes

module: custom-template-status

namespace: http://cisco.com/yang/nso/custom-template-status

prefix: ct-status

sessionId=59 2021-02-18 01:13:15 admin@127.0.0.1 system/tcp com.cisco.corefpcommon.statuscodes.StatusCodes.startMaapiTransaction

tid=2384 db=running mode=read com.cisco.corefpcommon.statuscodes.StatusCodes.startMaapiTransaction

id=status-code-validation daemonId=7 daemonName=ncs-dp-2-core-fp-common:Statuscodesvalidationandcompletion

id=status-code-validation-py daemonId=17 daemonName=sc_val_daemon

db=running id=28 priority=1 path=/ncs:devices/device/live-status-protocol/device-type

[root@localhost ncs]#

kwanchai.szd · ‎05-01-2021

I found a similar case.What I do is, I go into fixing the error manually, or delete the one that caused the error.

according to https://blog.devopssimplified.com/Cisco-NSO-Common-Error-Msg-and-Fix

Can follow up with session "You can delete an entire package our from the NSO by deleting the package from the packages dir followed by a packages reload . This needs to be forced for the deletio"

BTW, I have some questions about how to download >> nso-5.4.2-tsdn-2.0.0. Can you share how you download it?

Thank you in advance.

vleijon · ‎05-01-2021

The TSDN cfp requires a separate license. When you purchase it you will be provided access instructions, please talk to your account team for details.