Re: Copy input ActionParams into Service Nodes

Gastonper · ‎06-26-2024

Hi,

I want to know if it is possible in Python Maagic/Maapi to copy the inputs of an action directly into the schema tree of a service.

For example, if I have a grouping in Yang that is used by both my service and the action:

grouping items {
  leaf A {...}

  list B {
    ...
    
    container C {
      leaf D {..}
    }
  }
}

augment /ncs:services/ {
  list my-service {
    ...
    uses items;
  }
}

tailf:action my-action {
  input {
    uses items:
  }
}

Since both my service and action use items and have the same nodes, I want to create an instance of my service using the action inputs without having to explicitly iterate and assign leaf by leaf.

I want to do it in such a way that if a leaf is added to the grouping items in the future, the Python code doesn't need to change.

Thanks

james-henderson · ‎06-26-2024

I believe you can accomplish what you want using a recursive algorithm to iterate through the source tree, writing everything out into the destination tree. Both ActionParams and your service node will be a Node in maapi terms, and both will have _name and _children (accessed with dir(node)) as elements. You need to iterate through all of the dir(node) entries of the source node, then create entries of the correct type for each one. Doing this correctly would be a bit of a pain, but fortunately there is already some open source software to do just that:

https://gitlab.com/nso-developer/maagic-copy/-/blob/master/packages/maagic-copy/python/maagic_copy/maagic_copy.py?ref_type=heads

I think this will work for your use case, or it will work with a bit of modification. Either way it should give you a head start.

I am also interested to know exactly what the use case is here. What is the action doing that just creating the service wouldn't do? Generally the service itself should be the entry point for this sort of data, but I can imagine some use cases where this sort of design might make sense.

Gastonper · ‎06-26-2024

Thanks for the answer.

I want to have a custom API with specific output responses, error management, log traceability, etc. The service itself would not give me that. Using the action as an entry point gives me the ability to have custom output responses and generate random trace-IDs for logs.

huayyang · ‎07-17-2024

I don't think there's a single function in the pyapi that does what you want, but maybe you can do it in python with help of ActionParams class?

However I think your use case(having a customer api with customizable error management) is reasonable, you can write a feature request via aha(though there's no guarantee it'll be accepted)

james-henderson · ‎07-26-2024

@Gastonper

It sounds like this would be good feedback for the NSO product team. I'm guessing specific output responses are modelled? Do you have an example? I'm curious to see what it looks like.

The idea of traceable logs is interesting too. I know you can set commit labels and comments, but I don't think that will let you get every log generated by the commit.

Gastonper · ‎07-26-2024

I modeled the output as follows:

grouping output-actions {
	leaf user-id {
		description "user-id associated with the connection to ncs.";
		type string;
	}

	leaf trace-id {
		description "trace-id of the operation.";
		type string;
	}

	leaf status {
		description "Status";
		type string;
	}

	leaf status-code {
		description "Code associated with the status";
		type uint16;
	}

	leaf error-message {
		description "Descriptive text string";
		type string;
	}
}

Here is an example:

{
    "pon2-onu:output": {
        "user-id": "15387",
        "trace-id": "6bfb5fc6-1d67-409e-8871-263e70881260",
        "status": "completed",
        "status-code": 200
    }
}

With the trace-id I can check the logs for information about the operation:

tail -f ncs-python-vm-cdb-manager.log | grep "6bfb5fc6-1d67-409e-8871-263e70881260"

<INFO> 26-Jul-2024::16:07:08.84 cdb-manager ncs-dp-43064-cdb-manager:main:0-2-usid-15390-single-info: - {'username': 'root', 'user-id': 15387, 'type': 'action', 'trace-id': '6bfb5fc6-1d67-409e-8871-263e70881260', 'context': 'rest', 'name': 'create-onu', 'path': '/ncs:services/olt:pon2/olt{CEYD-01M}', 'inputs': {'chasis-number': 1, 'shelf-number': 0, 'slot-number': 2, 'port-number': 1, 'onuid': 21, 'access-card-type': 'GPON', 'technology': 'GPON', 'type': 'ONUE4P2W4', 'authentication-mode': 'serial-number', 'password': None, 'serial-number': '485754433A47079F', 'label': None, 'management': {'ipv4': {'address': None, 'mask': None, 'gateway': None}}}}

<INFO> 26-Jul-2024::16:09:28.955 cdb-manager ncs-dp-43064-cdb-manager:main:0-3-usid-15435-single-info: - {'username': 'root', 'user-id': 15433, 'type': 'data-change', 'trace-id': '6bfb5fc6-1d67-409e-8871-263e70881260', 'context': 'rest', 'transaction': 188429, 'commit-params': ['no-out-of-sync-check', 'trace-id', 'commit-queue', 'sync', 'sync', 'error-option', 'commit-queue'], 'changes': [{'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}', 'type': 'CREATED', 'old-value': 'None', 'new-value': 'None'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/id', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '21'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/serial-number', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '485754433A47079F'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/technology', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'enum<0>'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/access-card-type', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'enum<0>'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/authentication-mode', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'enum<1>'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/type', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'ONUE4P2W4'}]}

<INFO> 26-Jul-2024::16:09:29.52 cdb-manager ncs-dp-43064-cdb-manager:main:0-3-usid-15439-single-info: - {'username': 'root', 'user-id': 15433, 'type': 'data-change', 'trace-id': '6bfb5fc6-1d67-409e-8871-263e70881260', 'context': 'rest', 'transaction': 188429, 'commit-params': ['no-out-of-sync-check', 'trace-id', 'commit-queue', 'sync', 'sync', 'error-option', 'commit-queue'], 'changes': [{'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}', 'type': 'CREATED', 'old-value': 'None', 'new-value': 'None'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/portid', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '1'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/ontid', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '21'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/wifi{2}', 'type': 'CREATED', 'old-value': 'None', 'new-value': 'None'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/wifi{2}/id', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '2'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/wifi{2}/operational-state', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'enum<0>'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/wifi{3}', 'type': 'CREATED', 'old-value': 'None', 'new-value': 'None'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/wifi{3}/id', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '3'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/wifi{3}/operational-state', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'enum<0>'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/wifi{4}', 'type': 'CREATED', 'old-value': 'None', 'new-value': 'None'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/wifi{4}/id', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '4'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/port/attribute{1 21}/wifi{4}/operational-state', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'enum<0>'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/add{1 21}', 'type': 'CREATED', 'old-value': 'None', 'new-value': 'None'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/add{1 21}/portid', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '1'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/add{1 21}/ontid', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '21'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/add{1 21}/manage-mode', 'type': 'CREATED', 'old-value': 'None', 'new-value': 'None'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/add{1 21}/omci', 'type': 'CREATED', 'old-value': 'None', 'new-value': 'None'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/add{1 21}/profile-id', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '17'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/add{1 21}/sn-auth', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '485754433A47079F'}, {'path': '/ncs:devices/device{CEYD-01M}/config/hias:interface/gpon{0/2}/ont/add{1 21}/desc', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'CEYD-01M_0/2/1/21'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}', 'type': 'CREATED', 'old-value': 'None', 'new-value': 'None'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/id', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '21'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/serial-number', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': '485754433A47079F'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/technology', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'enum<0>'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/access-card-type', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'enum<0>'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/authentication-mode', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'enum<1>'}, {'path': '/ncs:services/olt:pon2/olt{CEYD-01M}/access-port{1 0 2 1}/pon2-onu:onu{21}/type', 'type': 'VALUE_SET', 'old-value': 'None', 'new-value': 'ONUE4P2W4'}]}

In the logs I get:

The name and inputs of the action.
The nodes of /servcies that have changed/created/deleted (pre-modification of the service)
The nodes of /devices that have changed/created/deleted (post-modification of the service)

With the same trace-id I also get the commands that NSO send to the device:

>> 26-Jul-2024::16:09:29.276 user: root/15433 thandle 188429 hostname 5b14f8b5989e device CEYD-01M trace-id=6bfb5fc6-1d67-409e-8871-263e70881260 PREPARE 0:
interface gpon 0/2
 ont add 1 21 sn-auth 485754433A47079F profile-id 17 desc CEYD-01M_0/2/1/21 manage-mode omci
 ont port attribute 1 21 wifi 2 operational-state off
 ont port attribute 1 21 wifi 3 operational-state off
 ont port attribute 1 21 wifi 4 operational-state off
!

In case of error in the device, the output is something like this:

{
    "pon2-onu:output": {
        "user-id": "15447",
        "trace-id": "64f286e6-08be-4869-9099-7fd6a5980414",
        "status": "failed",
        "status-code": 400,
        "error-message": "El número de serie '485754433A47079F' está repetido"
    }
}

james-henderson · ‎07-26-2024

What if instead of using an action to use the service, you included an action in the service to query after a problem?

You could store information about each request in a list under the service using pre_mod and post_mod to create a unique ID, which would be tied to all the logs generated. That ID would be stored in a list. Then you could add an action "history", to the service, something like:

pon2 <id> history latest
  trace-id 64f286e6-08be-4869-9099-7fd6a5980414
  user-id"15447
  status failed
  status-code 400
  error-message "El número de serie '485754433A47079F' está repetido"
  logs

pon2 <id> history latest logs
  ... (same info as above)
  logs "log lines here...." (but also include all the logs in the output)

This grouping and actions plus pre/post callbacks could be applied for all your services to create a uniform interface for the northbound system to display all this additional information.

This is just an idea, not sure if this is the best way to do this, and I know the design could be improved. I can ask/look around and see if someone I know has implemented something similar if you are interested.

Gastonper · ‎07-26-2024

That's a really good idea. I would argue that this would make my system asynchronous, but it's a pretty good alternative, thanks

huayyang · ‎07-26-2024

1 for config changes pushed to device, there's /ncs-config/logs/audit-network-log, so you don't need to write your own log

2 if you use an action to process logs, you need to consider in rest client <-> nso <-> action callback communication, the payload in nso <-> action callback is not streamed, which means you probably want to stress/scale test your solution

james-henderson · ‎07-26-2024

@Gastonper

You may also be able to give an ID in the error message depending on when and how the error came about. That ID could be used to query the history subsystem of the service (if such a system existed). That might fix the async issue that you were thinking about.