NSO Issue with Admin Down State When Changing Interface from L2 to L3

[ndmitri]

Hello Cisco Community,

I’m currently working on a network automation project using Cisco NSO and Cisco NX-OS devices (version 10.3(1) [Feature Release]). I’m encountering an issue when changing an interface from Layer 2 to Layer 3 via NETCONF. After making the change, the interface goes into an “admin down” state, and I’m struggling to bring it back up programmatically.

Project Setup:

• NSO Version: 6.2

• Device: Cisco switch running NX-OS 10.3(1) [Feature Release]

• Automation Tool: Cisco NSO with Python callbacks and NETCONF

• Objective: Change an interface from Layer 2 to Layer 3 and ensure it remains “admin up” after the change.

Issue Description:

When I change an interface from Layer 2 to Layer 3 using NETCONF, the interface automatically transitions to an “admin down” state. I have implemented logic in NSO to detect this transition and attempt to bring the interface back up, but it isn’t working as expected.

Code Implementation:

Here’s a summary of my NSO service callback logic:

1. Detection of Layer Change:

• I detect when an interface’s layer is changed from Layer 2 to Layer 3.

• I set an operational flag (needs_admin_up) to indicate that the interface needs to be brought up after the change.

2. Post-Modification Callback:

• In the cb_post_modification method, I check the needs_admin_up flag.

• If the flag is set, I attempt to bring the interface back up by setting adminSt to “up”.

.post_modification
def cb_post_modification(self, tctx, op, kp, root, proplist):
    self.log.info('Service post-modification(service=', kp, ')')

    service = ncs.maagic.get_node(root, kp)
    self.log.info(f'needs_admin_up: {service.needs_admin_up}')

    if getattr(service, 'needs_admin_up', False):
        self.log.info(f'Bringing interface {service.interface} on device {service.device} back up.')

        # Attempt to set the interface admin state to up
        device_config = root.devices.device[service.device].config
        device_if = device_config.top__System.intf_items.phys_items.PhysIf_list[service.interface]
        device_if.adminSt = "up"

        self.log.info(f'Interface {service.interface} on device {service.device} is now up.')

Logs:

The logs indicate that the logic to bring the interface up is being executed:

<INFO> 14-Aug-2024::01:27:28.628 phys-if ncs-dp-233251-phys-if:phys_if:0-1-th-67134: - needs_admin_up: True
<INFO> 14-Aug-2024::01:27:28.628 phys-if ncs-dp-233251-phys-if:phys_if:0-1-th-67134: - Bringing interface eth1/12 on device SPN01.FAB00.EWR00 back up.
<INFO> 14-Aug-2024::01:27:28.640 phys-if ncs-dp-233251-phys-if:phys_if:0-1-th-67134: - Interface eth1/12 on device SPN01.FAB00.EWR00 is now up.

Problem:

Despite the logs indicating that the interface is being brought up, the interface remains in an “admin down” state on the device. I don’t see any transactions being committed that would push the adminSt change to the device.

What I’ve Tried:

• Direct assignment: Setting device_if.adminSt = "up" directly in Python.

• Using templates: Applying a template in the cb_post_modification callback to set the adminSt value.

• Manual verification: Manually checking the interface state via CLI, which confirms the interface remains down.

• Reviewing logs: No clear errors or warnings in NSO logs that would indicate why the change isn’t being applied.

Question:

Has anyone encountered a similar issue when changing interface modes via NETCONF on NX-OS? What could be causing the interface to remain in an “admin down” state despite my attempts to bring it back up? Are there any specific NX-OS quirks or additional configurations required when making such changes?

Any insights or suggestions would be greatly appreciated!

Thank you!

[Marcel Zehnder]

Hi @ndmitri Does it work, if you set the adminSt in the create callback (cb_create) instead of the post-modification callback? Or do you encounter the same issue?

[snovello]

Hello,

The post modification callback makes its changes in the same transaction as the create, the differences is that you are no longer in the scope of fastmap, so any changes made are not undone when the service is removed and generally anything configured in post modification is not configuration owned by the service, so there the attribute to point back to the service is not set for the adminSt leaf.

This is why you see the code running but it has no effect. The code runs, sets the adminSt to up in cdb but the merge operation will see it is already up and so no change is made.

What you need  is something running after the transaction has been sent to the device and succesfully terminated that starts a new transaction. First, since the device has changed adminSt on its own without you asking for it, the cdb does not know it is down. You need to do a sync-from to read that state in. Second you need to change the value of the leaf.

To do that you first need to create an action, typically you add it to the yang model inside your service, then you need to trigger it. There are 2 methods I would recommend, a kicker triggered on any change to the service could be used, or you could implement a nano-service with 2 states. I recommend to use the kicker for this. It is conceptually simpler, and since nano-services are built on top of kickers it helps to understand the kicker before building using a nano-service.

To be more efficient the action could look at the needs-change flag, or do just a partial sync from to just read the state of the leaf and only make the change it if the leaf is down.

[ndmitri]

@snovello I will give it a try and let you know the result. Thank you.

[ndmitri · Accepted solution]

@snovello,

Thank you for your detailed explanation and recommendations regarding the post-modification callback and the behavior of adminSt in NSO. I wanted to share the approach we ultimately decided to take and the rationale behind it.

Instead of relying on the implementing actions and kickers, we opted to use a CDB subscriber to monitor changes to the layer leaf of the interface configuration. By subscribing to changes in the layer attribute, we were able to detect when an interface transitions from Layer 2 to Layer 3 (or vice versa).

When such a change is detected, the subscriber triggers a sequence of operations to explicitly set the adminSt to “down” and then back to “up” in a new transaction. This ensures that the interface is correctly brought back up after the layer transition, regardless of the automatic state changes made by the device itself.

Why We Chose This Route:

1. Direct and Immediate Reaction: The CDB subscriber allows us to react directly and immediately to changes in the layer attribute. This approach ensures that the necessary adjustments to the adminSt are made as soon as the configuration change is detected.

2. Simplicity and Maintainability: Implementing the solution via a subscriber was conceptually straightforward and aligned well with our existing workflow. It allowed us to avoid additional complexity associated with creating and managing actions and kickers, which might have required more intricate coordination and testing.

3. Consistency in Transactions: By handling the adminSt adjustment within a new transaction triggered by the subscriber, we ensured that the changes were consistently applied in the desired order, avoiding potential issues with synchronization or state mismatches.

4. Flexibility for Future Adjustments: The subscriber approach provides a flexible foundation that can easily be extended or modified in the future if additional monitoring or reactions are needed for other configuration changes.

Overall, the CDB subscriber provided a clean, efficient, and reliable way to achieve our goals without the need for additional mechanisms like actions or kickers. This decision also simplified the implementation, making it easier to manage and maintain in the long term.

Thanks again for your input and guidance. It was invaluable in helping us explore the different options and ultimately arrive at a solution that works well for our use case.