Intermittently getting 'Failed to connect to ConfD: Connection refused' immediately after restarting NSO

[kevchian]

Hi,

I have a Python script that makes use of the NSO Python APIs (MAAPI and Maagic APIs) to query for something in the CDB.

The preconditions for running the script:

• NSO is already installed (system installation) on the host. This script is executed on the same host where NSO is installed.
• NSO may be running, or it may be stopped. A separate script ensures that NSO is running by restarting NSO (via /etc/init.d/ncs restart) immediately before invoking my script.
• System-wide environment variables (e.g. PYTHONPATH) must be loaded by sourcing /etc/profile.d/ncs.sh or /opt/ncs/current/ncsrc

When the script is run, I occasionally encounter the following stack trace originating from a single_read_trans call:

with maapi.single_read_trans(user, 'system') as t:
    ...

The stack trace:

SSH CMD FAILED (Exit Status 1): 'ssh source /opt/ncs/current/ncsrc; python /opt/cisco/nso/test/scripts/release_updater/fp_check.py --user admin --list' returned:
Traceback (most recent call last):
  File "/opt/cisco/nso/test/scripts/release_updater/fp_check.py", line 100, in <module>
    main(args)
  File "/opt/cisco/nso/test/scripts/release_updater/fp_check.py", line 76, in main
    print_installed_function_packs(args.user)
  File "/opt/cisco/nso/test/scripts/release_updater/fp_check.py", line 27, in print_installed_function_packs
    with maapi.single_read_trans(user, 'system') as t:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/opt/ncs/ncs-4.6.1/src/ncs/pyapi/ncs/maapi.py", line 128, in single_read_trans
    with Maapi(ip, port, path) as m:
  File "/opt/ncs/ncs-4.6.1/src/ncs/pyapi/ncs/maapi.py", line 181, in __init__
    self.msock = connect(ip, port, path)
  File "/opt/ncs/ncs-4.6.1/src/ncs/pyapi/ncs/maapi.py", line 97, in connect
    _tm.maapi.connect(msock, ip, port)
_ncs.error.Error: system call failed (24): Failed to connect to ConfD: Connection refused

My first guess would be that the MAAPI library is trying to connect to ConfD before ConfD comes up (right after the NSO restart). Should I be polling for some process to be up before invoking my script? Any guidance/suggestions on this matter would be greatly appreciated.

[mvolf]

The error message is most likely misleading: the Python API uses existing C API and the C API is shared between ConfD and NSO and uses the word "ConfD" in its error messages. So I guess it is trying to say that NSO is not (fully) running yet, your script was too fast after the restart.

I do not know how the system installation and its init.d scripts look like, but NSO allows you to start it synchronously, the command does not return until NSO is fully running. Isn't something like this possible in the system installation?

[kevchian]

Thanks Martin. I suspected as much about the slightly misleading error messages, but it's reassuring to get confirmation on that.

You're also right about the command to start NSO being a synchronous one. Once the command to start NSO returns, NSO should be fully running.

With your help I think I figured it out. I think what happened in my case (it was on a CentOS machine using systemd as its init system) was that the NSO systemd unit file ncs.service "changed on disk" some time after starting NSO. So I needed to run systemctl daemon-reload before /etc/init.d/ncs restart in order for NSO to restart properly. Since I was previously not restarting NSO properly, the Python MAAPI API was unable to connect to NSO, which led to the error.