NETCONF notification stream forwarding in LSA environment

[JakubHolcman]

Hello all,

I am working on self-made NSO NETCONF notifications in LSA environment (RFS node - CFS node).
There are several difficulties I faced so far. For the context I am using Python, because my company wants to use python.

1. There is no high-level support for notifications in python.
2. You can't access Notification node with maagic implementation (the node will be of type "None"), but there is possibility to access it with low-level _ncs.maapi.
3. There is no possibility to forward custom created stream to northbound (like pre-configured stream "device-notifications" it does) of CFS node when RFS node generates an notification towards CFS.

Python sender function:

def send(self):
        notification_socket = socket.socket()

        try:
            # init daemon
            ctx = _ncs.dp.init_daemon(f"netconf-notification-{self.qual_name}")

            # making a socket
            _ncs.dp.connect(
                dx=ctx,
                sock=notification_socket,
                type=_ncs.dp.CONTROL_SOCKET,
                ip="127.0.0.1",
                port=_ncs.PORT,
            )

            # register the stream
            livectx = _ncs.dp.register_notification_stream(
                ctx,
                None,
                notification_socket,
                self._stream,
            )

            # time
            now = datetime.now(timezone.utc)  # timezone must be set correctly!!!
            time = _ncs.DateTime(
                now.year,
                now.month,
                now.day,
                now.hour,
                now.minute,
                now.second,
                now.microsecond,
                int(now.strftime("%z")[:3]),
                int(now.strftime("%z")[3:]),
            )

            self.log.debug(f"Generating notification message...")

            message = self._create_xml()

            self.log.debug(f"Notification message is set to={message}")
            self.log.debug(f"Sending notification with={livectx=}, {time=}, {message=}...")

            # sending the notification
            _ncs.dp.notification_send(
                livectx,
                time,
                message,
            )

            self.log.debug(f"Message sent...")

        except Exception as e:
            self.log.info(e)

        finally:
            notification_socket.close()

Example of underlying YANG on RFS node:

module device-status-notifications {
  yang-version 1.1;
  namespace "http://example.com/device-status-notifications";
  prefix status-notifi;

  import tailf-common {
    prefix tailf;
  }

  notification reachability-status {
    leaf customer-name {
      type string;
    }
    leaf site-id {
      type string;
    }
    leaf device-id {
      type string;
    }
    leaf device-uuid {
      type string;
    }
    leaf host-name {
      type string;
    }
    leaf status {
      type string;
    }
  }
}

As "dirty-fix" I tought about listening on stream with an kicker on CFS node and re-sending the same notification from CFS (on the same stream). I don't really like this solution because I have to develop 2 YANG models (for RFC and CFS) with notification in it and also implement 2 senders in python (on RFS and CFS).

Do you have different ideas / approaches to solve this?

P.S. I also realised that python has the worst API support of the main 3 APIs (C, Java, Python)

This post should be seen as feature request / discussion thread.

[cohult]

Hello,

I believe that you can set /cluster/device-notifications/enabled to forward device notifications.
From the $NCS_DIR/src/ncs/yang//tailf-ncs-cluster.yang notifications container description:
"NETCONF notification streams used by NCS nodes in a clustered environment. A top NCS node will automatically subscribe to these streams for all remote NCS nodes.The 'ncs-events' stream is used by the commit queue.  The 'device-notifications' stream is used to forward NETCONF notifications originating from devices managed by remote NCS nodes.";

Note that you can use the high-level API to handle sockets for notification sending. Example:

import ncs
import _ncs

d = ncs.dp.Daemon("change-notifier", port=port)
nctx = _ncs.dp.register_notification_stream(d.ctx(), None,
ncs.dp.take_worker_socket(d,
'notif',
'notif-key'),
'some_state')
d.start()
...
_tm.dp.notification_send(nctx, time, message)

Regarding "Python has the worst API support", I agree that a high-level API for sending notifications would be helpful not to bother with time stamps and tag value XML-like details.

Do you have any other input on what's missing and should be supported in the Python API?

[JakubHolcman]

Hello,
thank you for your answer. Device-Notifications Stream can be successfully forwarded (Notifi -> RFS -> CFS -> Management Systems) but custom streams like "my-stream" are not forwarded this way.
If RFS sends an notification on "my-stream" CFS-Node gets it successfully but management systems that subscribe to CFS "my-stream" do not recive the notification (RFS --Notifi--> CFS --X--> Management Systems).
Is there any way to forward "my-stream" like "device-notifications" stream?

[cohult]

Hello,
It works, but what you probably missed, since it is undocumented and no examples show it, is that you need to load the device NED package(s) with the CFS NSO node, too (not just the RFS nodes). The NED should not be configured on the CFS node. It should just be loaded so the CFS NSO has the FXS from the YANG model with the notification loaded. Otherwise, you will get a "connecting" status for the device-notifications stream:

admin@upper-nso# show cluster           
                                                                                 RECEIVED       
NAME         NAME                  STATUS      LAST EVENT                        NOTIFICATIONS  
------------------------------------------------------------------------------------------------
lower-nso-1  device-notifications  connecting  0000-01-01T00:00:00-00:00         0              
             ncs-events            up          2023-11-28T18:52:50.312666+00:00  9              
lower-nso-2  device-notifications  connecting  0000-01-01T00:00:00-00:00         0              
             ncs-events            up          2023-11-28T18:52:50.310534+00:00  9     

... and a CFS node error log similar to this:

$ ncs --printlog upper-nso/logs/ncserr.log.1
28-Nov-2023::19:52:50.271 <0.433.0> <0.1790.0> ncs_cluster_nc_notifs:1246: cluster: add error code for: {ncs_cluster_nc_notifs,
                                                                                                         {unknown_notification,
                                                                                                          "http://example.com/device-status-notifications",
                                                                                                          "reachability-status"}} (<<"lower-nso-1">> <<"admin">>)

[{ncs_cluster_nc_notifs,mk_session_error,2,
                        [{file,"ncs_cluster_nc_notifs.erl"},{line,1247}]},
 {ncs_cluster_nc_notifs,subscriber_loop,2,
                        [{file,"ncs_cluster_nc_notifs.erl"},{line,711}]},
 {ncs_cluster_nc_notifs,subscriber,3,
                        [{file,"ncs_cluster_nc_notifs.erl"},{line,658}]},
 {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]
28-Nov-2023::19:52:50.274 <0.433.0> <0.1783.0> ncs_cluster_nc_notifs:1246: cluster: add error code for: {ncs_cluster_nc_notifs,
                                                                                                         {unknown_notification,
                                                                                                          "http://example.com/device-status-notifications",
                                                                                                          "reachability-status"}} (<<"lower-nso-2">> <<"admin">>)

[{ncs_cluster_nc_notifs,mk_session_error,2,
                        [{file,"ncs_cluster_nc_notifs.erl"},{line,1247}]},
 {ncs_cluster_nc_notifs,subscriber_loop,2,
                        [{file,"ncs_cluster_nc_notifs.erl"},{line,711}]},
 {ncs_cluster_nc_notifs,subscriber,3,
                        [{file,"ncs_cluster_nc_notifs.erl"},{line,658}]},
 {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]

I will add a ticket to create some documentation and an example that shows how to set it up to avoid wasting more time for users. Also, the error should be handled better so the user is notified of the issue somehow.

[JakubHolcman]

Hello,
thank you for your answer. I have created new environment and loaded NED on CFS as you described in your post.
Now I have the issue that everything is up but CFS gets no notifications (without configuring subscriber under "devices/device/netconf-notifications/subscription"). With an exclusive subscription CFS recives the notifications but doesn't re-send them on its norhboud -> the tool "netconf-console2" didn't recive any notification from CFS on "vedge-reachability-status" / "my-stream" stream.

cloud@ncs> show cluster
                                                                    RECEIVED       
NAME       NAME                  STATUS  LAST EVENT                 NOTIFICATIONS  
-----------------------------------------------------------------------------------
rfs_sdwan  device-notifications  up      0000-01-01T00:00:00-00:00  0              
           ncs-events            up      0000-01-01T00:00:00-00:00  0

 I don't see the custom stream
There are also no errors in ncserr.log.1

cloud@ncs> show devices device rfs_sdwan notifications 
                                                                                                                                                                           REPLAY  
                                                                                                                                                                           LOG     
                                                                                                                                REPLAY                                     AGED    
NAME                       DESCRIPTION                                                                                          SUPPORT  REPLAY LOG CREATION TIME          TIME    
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
NETCONF                    default NETCONF event stream                                                                         false    -                                 -       
ncs-alarms                 NCS alarms according to tailf-ncs-alarms.yang                                                        false    -                                 -       
ncs-events                 NCS event according to tailf-ncs-devices.yang                                                        true     2023-11-29T12:47:35.900918+00:00  -       
kicker-events              NCS event according to tailf-kicker.yang                                                             true     2023-11-29T12:47:35.911359+00:00  -       
device-notifications       NCS events forwarded from devices                                                                    true     2023-11-29T12:47:35.915956+00:00  -       
service-state-changes      Service state changes according to
        tailf-ncs-plan.yang and tailf-ncs-services.yang
          true     2023-11-29T12:47:35.920803+00:00  -       
vedge-reachability-status  NETCONF notifications about vedge reachability status generated by rfs-node                          false    -                                 -       

NAME               URI                                   
---------------------------------------------------------
test-notification  http://example.com/rfs-notifications 

cloud@ncs> show packages package oper-status 
                                                                                                          PACKAGE                          
                            PROGRAM                                                                       META     FILE                    
                            CODE     JAVA           PYTHON         BAD NCS  PACKAGE  PACKAGE  CIRCULAR    DATA     LOAD   ERROR            
NAME                    UP  ERROR    UNINITIALIZED  UNINITIALIZED  VERSION  NAME     VERSION  DEPENDENCY  ERROR    ERROR  INFO   WARNINGS  
-------------------------------------------------------------------------------------------------------------------------------------------
bgworker                X   -        -              -              -        -        -        -           -        -      -      -         
observability-exporter  X   -        -              -              -        -        -        -           -        -      -      -         
rfs-notifications-ned   X   -        -              -              -        -        -        -           -        -      -      -         
tailf-hcc               X   -        -              -              -        -        -        -           -        -      -      -         

Is there anything I am missing at this point?

[cohult]

Hi @JakubHolcman 

Example ncs.conf on CFS and RFS nodes:

<notifications>
<event-streams>
...
<stream>
<name>device-notifications</name>
<description>NSO events forwarded from devices</description>
<replay-support>true</replay-support>
<builtin-replay-store>
<enabled>true</enabled>
<dir>./state</dir>
<max-size>S10M</max-size>
<max-files>50</max-files>
</builtin-replay-store>
</stream>
...
</event-streams>
</notifications>

RFS node(s):

ncs_cli --port $MY_RFS_NC_PORT -u admin -C << EOF
config
devices device my-dev0 netconf-notifications subscription some-name stream my-stream-name local-user my-user
devices device my-dev1 netconf-notifications subscription some-name stream my-stream-name local-user my-user
commit
EOF

CFS node:

ncs_cli --port $MY_CFS_IPC_PORT -n -u admin -C << EOF
config
cluster commit-queue enabled
cluster device-notifications enabled
commit
EOF

Verify RFS node device notifications:

netconf-console --port=$MY_RFS_NC_PORT --create-subscription=device-notifications

CFS:

netconf-console --port=$MY_CFS_NC_PORT --create-subscription=device-notifications

[JakubHolcman]

Hi,
so I get it, ok that works.
The final Question: Is it possible to have multiple streams in cluster configuration like below?
The main point of this is: I / we don't want to use "device-notifications" stream, instead we would like to listen on an exclusive stream for that specific notification.

cloud@ncs> show cluster
                                                                    RECEIVED       
NAME       NAME                  STATUS  LAST EVENT                 NOTIFICATIONS  
-----------------------------------------------------------------------------------
rfs_sdwan  device-notifications  up      0000-01-01T00:00:00-00:00  0              
           ncs-events            up      0000-01-01T00:00:00-00:00  0
           vedge-reachability    up      0000-01-01T00:00:00-00:00  0
           vmanage-status        up      0000-01-01T00:00:00-00:00  0
           management-info       up      0000-01-01T00:00:00-00:00  0

I attach an image with final architecture I / we want to have. Maybe this would help to understand my case better
Btw: both CFS and RFS will have "reachability-stream" defined in their ncs.conf file.
What I want to do

[cohult · Accepted solution]

Hi,
There is no built-in support for forwarding notifications other than the device-notifications stream. So I would suggest something like attached image:

Note: Only the RFS will have the "reachability-stream" defined in its ncs.conf file

If you need the CFS node to forward and expose a reachability stream (not the device-notifications stream) to the Management-System, you need an application in a package on the CFS that does the forwarding.

Also, note that the Management-System can create a RESTCONF subscription for the CFS device-notifications stream if REST is preferred over NETCONF.

[JakubHolcman]

Hi,
thank you for all your help and explanations.
Would be nice if NSO would support custom stream forwarding in future
Again thank you very much.

Jakub H.