NSO live status commands in parallel

[Raja.Selvaraj]

Hi,

I am trying to write a restconf code to run parallel show/any commands to read the data from same device(ios-xr) via live-status. I am facing the connectivity issues constantly when i perform this in parallel and no issues if i do it in serial. Can someone suggest how to make these calls in parallel via NSO and that reduces the wait time if we try with serial.

//Sample restconf payload 

let params = {

path: `/restconf/ds/ietf-datastores:operational/tailf-ncs:devices/device=${deviceName}/live-status/tailf-ned-cisco-ios-xr-stats:exec/any`,

method: "POST",

accept: "application/yang-data+json",

contentType: "application/yang-data+json"

};

//Error observed
Failed to connect to device sxxx-e-100: connection refused: NEDCOM CONNECT: Connection reset in new state

[snovello]

Looks like the device is refusing to open a connection. Some ,I think most network devices protect themselves from CPU overloads by limiting the number of management sessions they handle in parallel. If many of your requests are hitting the same device this may be your issue. You just have to find how to reconfigure your device to accept more parallel sessions so it can run your tests.

[Raja.Selvaraj]

Thanks for your input. Are you saying I need to reconfigure the device in the NSO to accept more connections. In fact, do we have settings in the NSO while configuring the device or something to do with NED settings?

[snovello]

Hello,
To avoid the error you see, you should modify the configuration on the device itself, I am not talking about an NSO feature. I don’t know your device but for example if it was an IOS-XE device you would change the value of session-limit
See here
https://www.cisco.com/c/en/us/td/docs/ios/termserv/configuration/guide/xe_16/tsv-xe-book/tsv_term_op_char_dialin.html#pgfId-1001125
Also the number of vtys in the IOS-XE configuration will act as a limit on the number of sessions.

As an aside NSO will by default let you attempt to make as many parallel sessions as it can, eventually it will hit some thread limit but that will just mean there is a limit to parallelism, not an error like you see, also you can configure NSO to limit the number of parallel sessions to a device, which can avoid such errors if you have code that is attempting to make many accesses in parallel.

[huayyang]

In NSO-5.5+, there's a session-limits setting to limit how many concurrent sessions NSO can create towards a device(look for ENG-3837 in CHANGES), but what you see indicates it's the device that refuses the connection from NSO.

[Raja.Selvaraj]

Thanks. How do I confirm this error from TACAS or reaching the device SSH connection limit. All I can see from the trace is NED exception.

-- SSH failed, closing connection

-- [22-Aug-2023::19:36:05.679] progress: usid=98468 thandle=0: ncs: connect: cisco-iosxr: connecting error [30 ms]

-- ERROR:

com.tailf.ned.NedException: NEDCOM CONNECT: Connection reset in new state

        at com.tailf.packages.ned.nedcom.connector.CliConnectorNedUtils.connectorConnectDevice(CliConnectorNedUtils.java:58)

        at com.tailf.packages.ned.nedcom.connector.CliConnectorNedUtils.connectorConnectDevice(CliConnectorNedUtils.java:46)

        at com.tailf.packages.ned.iosxr.IosxrNedCli.connectDevice(IosxrNedCli.java:779)

        at com.tailf.packages.ned.nedcom.NedComCliBase.newConnection(NedComCliBase.java:2541)

        at com.tailf.ned.NedWorker.dorun(NedWorker.java:1577)

        at com.tailf.ned.NedWorker.run(NedWorker.java:317)

Caused by: com.tailf.packages.ned.nedcom.connector.CliException: Connection reset in new state

        at com.tailf.packages.ned.nedcom.connector.CliInteractor.action(CliInteractor.java:719)

        at com.tailf.packages.ned.nedcom.connector.CliInteractor.access$600(CliInteractor.java:36)

        at com.tailf.packages.ned.nedcom.connector.CliInteractor$State.runState(CliInteractor.java:616)

        at com.tailf.packages.ned.nedcom.connector.CliInteractor$State.access$100(CliInteractor.java:532)

        at com.tailf.packages.ned.nedcom.connector.CliInteractor.run(CliInteractor.java:521)

        at com.tailf.packages.ned.nedcom.connector.CliConnectorNedUtils.doConnectorConnectDevice(CliConnectorNedUtils.java:209)

        at com.tailf.packages.ned.nedcom.connector.CliConnectorNedUtils.connectorConnectDevice(CliConnectorNedUtils.java:53)

        ... 5 more

[huayyang]

you need to look at logs on device, I'm no device expert, but a colleague mentioned "debug ssh server" command on device cli