Terrafrom [BUG] - Resource "nxos_save_config" not working

[krzysztofmaciejewskiit]

Version: 0.5.7

I want to save the configuration of the device with Terraform.

Logs when creating the resource that should be responsible for this.

The result on the device, which suggests an unsaved configuration. After saving manually, WARNING does not appear.

However, when I do this with the "nxos_rest" resource it saves after the second run of Terraform (it doesn't after the first - not sure why, but at least it works). The content of the resource "nxos_rest" "save" is literally a rewritten request by the API, which you can see in the debug.

API request:

[krzysztofmaciejewskiit]

From the perspective of the switch, about running the"debug nxapi-server logs" command, you can see that something interrupts the save.

[mfr-6]

hi @krzysztofmaciejewskiit 

I'm not sure if this is an issue with TF/Provider itself. I just sent a POST request to sandbox nxos with a payload from the docs here -  and it works fine - configuration is saved.

Let's break that down:

First - i just modified the configuration and then verified if there is a prompt about unsaved configuration when attempting to reboot the switch.

Enter configuration commands, one per line. End with CNTL/Z.
nxos(config)# int eth1/12
nxos(config-if)# description before_tf_save
nxos(config-if)# end
nxos# reload
!!!WARNING! there is unsaved configuration!!!
This command will reboot the system. (y/n)? [n] n
nxos#

There we go - we have unsaved configuration now.

To make it even more clear - I checked if startup config didn't contain new interface description:

nxos# sh startup-config interface eth1/12

!Command: show startup-config interface Ethernet1/12
!Time: Sat Feb 15 11:00:48 2025
!Startup config saved at: Sat Feb 15 10:47:07 2025

version 10.3(3) Bios:version

interface Ethernet1/12
description www
shutdown

nxos#

I created nxos_save_config resource and create a plan and apply - tf workflow executed flawlessly.

(main) expert@expert-cws:~/tf$ terraform plan

Terraform used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# nxos_save_config.cfg_saved will be created
+ resource "nxos_save_config" "cfg_saved" {
+ save = true
}

Plan: 1 to add, 0 to change, 0 to destroy.

───────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
(main) expert@expert-cws:~/tf$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# nxos_save_config.cfg_saved will be created
+ resource "nxos_save_config" "cfg_saved" {
+ save = true
}

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

nxos_save_config.cfg_saved: Creating...
nxos_save_config.cfg_saved: Creation complete after 1s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
(main) expert@expert-cws:~/tf$

Coming back to nexus - i verified the startup config and as expected - the config was saved - now eth1/12 interface has new description.

nxos# sh startup-config interface eth1/12

!Command: show startup-config interface Ethernet1/12
!Time: Sat Feb 15 11:01:40 2025
!Startup config saved at: Sat Feb 15 11:01:24 2025

version 10.3(3) Bios:version

interface Ethernet1/12
description before_tf_save
shutdown

nxos#

Before excercise with TF - I sent POST request using curl with the payload from the NXAPI docs to double-check if it works on sandbox nxos and I confirmed it worked. I was thinking that there could be a bug in terraform nxos provider, but for me everything is fine.

Although - from the nxapi debug output - I see an issue with a particular error 0x4370002C.

Could you please have a look at output from the following commands?

show system error-id 0x4370002C
show nxapi retries

I suspect that a particular feature on your nexus that is enabled prevents you from saving the configuration properly via NXAPI. I got a similar issue with IOSXE where a particular configuration applied via RESTCONF couldn't be applied due to the wrong order (CLI prevented from doing so, but RESTCONF didn't). In my case - I tried to set hello timers on a interface level without having EIGRP AS created. hello timer command requires you to provide AS number. Configuration has been applied but DME crashed and I had to reload the router, well...

To sum it up - I don't think this is a TF Provider bug. Instead - have a look at your Nexus, i think this is the place where the root cause sits.

This looks like a bug - at least looks like it's similar to your case.
https://bst.cisco.com/quickview/bug/CSCwa58073

[krzysztofmaciejewskiit]

Hi Mateusz @mfr-6,
Thanks for your suggestions. I have attached the screenshot you requested below. The moment I type "show nxapi retries" is when I run "terraform apply -auto-approve" and I want to catch that moment you see in the screenshot. What information does this give us?


I am also uploading the software version below.

[mfr-6 · Accepted solution]

@krzysztofmaciejewskiit 

Most likely you are affected by a bug I've mentioned above and seems like BFD configuration on a interface eth1/10 crashes DME on a switch or ACL - based on the output you've provided.

This bug is resolved in the next minor release of the nxos you have -> 10.2(4) https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/102x/release-notes/cisco-nexus-9000-nxos-release-notes-1024.html

The workaround described under the bug entry is to reload ascii startup configuration -> "reload ascii" or to clear nxapi retries -> "clear nxapi retries". I wasn't able to reproduce that in my lab so I cannot say more about the impact.

Please let me know if that helps.