08-08-2024 06:57 AM - edited 08-08-2024 07:27 AM
Having a "clean" terraform directory (main.tf, terraform.tfvars, variables.tf) executes the command "terraform init; terraform apply -auto-approve", everything passes fine, adds the first part of the code marked "part-1", all the commands along with "no shutdown" are added. When I add the code "part-2", coemend "terraform apply -auto-approve" usually all commands except "no shutdown" are added, relatively rarely all will be added, so it happens quite randomly. When I let go of the same terraform again without changes it will add the command "no shutdown" correctly. Below I paste the code of "part-1" and "part-2" and the output from terraform.
Terraform code: main.tf
terraform {
required_providers {
nxos = {
source = "CiscoDevNet/nxos"
version = "0.5.3"
}
}
}
provider "nxos" {
username = var.nxos_username
password = var.nxos_password
url = var.nxos_url
}
/*============================================ common ============================================*/
resource "nxos_feature_lldp" "lldp" {
admin_state = "enabled"
}
/*=====================================*/
resource "nxos_rest" "cdpEntity" {
dn = "sys/cdp"
class_name = "cdpEntity"
}
//============================================== part-1 ==============================================
//============================================== part-1 ==============================================
//============================================== part-1 ==============================================
/*============================================ description & L3 ============================================*/
resource "nxos_physical_interface" "desc-L3" {
interface_id = "eth1/5"
description = "desc1"
layer = "Layer3"
admin_state = "up"
user_configured_flags = "admin_state"
}
/*============================================ lldp ============================================*/
resource "nxos_rest" "lldpInst" {
depends_on = [nxos_feature_lldp.lldp]
dn = "sys/lldp/inst"
class_name = "lldpInst"
children = [
{
rn = "if-[eth1/5]"
class_name = "lldpIf"
content = {
adminRxSt = "disabled",
adminTxSt = "disabled",
id = "eth1/5"
}
}
]
}
/*============================================ cdp ============================================*/
resource "nxos_rest" "cdpInst" {
depends_on = [nxos_rest.cdpEntity]
dn = "sys/cdp/inst"
class_name = "cdpInst"
children = [
{
rn = "if-[eth1/5]"
class_name = "cdpIf"
content = {
adminSt = "disabled",
id = "eth1/5"
}
}
]
}
//============================================== part-2 ==============================================
//============================================== part-2 ==============================================
//============================================== part-2 ==============================================
/*============================================ description & L3 ============================================*/
resource "nxos_physical_interface" "desc-L3v2" {
interface_id = "eth1/6"
description = "desc2"
layer = "Layer3"
admin_state = "up"
user_configured_flags = "admin_state"
}
/*============================================ lldp ============================================*/
resource "nxos_rest" "lldpInstv2" {
depends_on = [nxos_feature_lldp.lldp]
dn = "sys/lldp/inst"
class_name = "lldpInst"
children = [
{
rn = "if-[eth1/6]"
class_name = "lldpIf"
content = {
adminRxSt = "disabled",
adminTxSt = "disabled",
id = "eth1/6"
}
}
]
}
/*============================================ cdp ============================================*/
resource "nxos_rest" "cdpInstv2" {
depends_on = [nxos_rest.cdpEntity]
dn = "sys/cdp/inst"
class_name = "cdpInst"
children = [
{
rn = "if-[eth1/6]"
class_name = "cdpIf"
content = {
adminSt = "disabled",
id = "eth1/6"
}
}
]
}
Output:
Terraform will perform the following actions:
# nxos_physical_interface.desc-L3v2 will be created
+ resource "nxos_physical_interface" "desc-L3v2" {
+ access_vlan = "vlan-1"
+ admin_state = "up"
+ auto_negotiation = "on"
+ bandwidth = 0
+ delay = 1
+ description = "desc2"
+ duplex = "auto"
+ fec_mode = "auto"
+ id = (known after apply)
+ interface_id = "eth1/6"
+ layer = "Layer3"
+ link_debounce_down = 100
+ link_debounce_up = 0
+ link_logging = "default"
+ medium = "broadcast"
+ mode = "access"
+ mtu = 1500
+ native_vlan = "vlan-1"
+ speed = "auto"
+ speed_group = "auto"
+ trunk_vlans = "1-4094"
+ uni_directional_ethernet = "disable"
+ user_configured_flags = "admin_state"
}
Output form Nexus9000:
I have one more problem, because when I delete a part of the code in e.g. part two, responsible for cdp or lldp it says it is destroyed, but it stays on the Nexus configuration and I don't know why all the other stuff gets deleted and those 3 lines of code (seen on the switch) stay, even though the terraform logs clearly say "destroyed", but I will devote a second post on the community to this, because it seems to have nothing to do with each other.
https://community.cisco.com/t5/devnet-general-discussions/after-removing-resource-cdp-ldp-configuration-is-left-on-the/m-p/5157851#M2553
Solved! Go to Solution.
08-30-2024 01:28 AM - edited 08-30-2024 01:30 AM
I went through that issue and I did a small lab. I've got two news - good and bad one.
The good one is that this is not the Terraform's issue (nor nxos provider). The bad one is - this is the issue with NXAPI that NXOS TF Provider uses under the hood.
I used Sandbox NXOS to troubleshoot that. See my Tf code below:
terraform {
required_providers {
nxos = {
source = "CiscoDevNet/nxos"
version = "0.5.3"
}
}
}
provider "nxos" {
username = "x"
password = "x"
url = "https://sbx-nxos-mgmt.cisco.com"
}
resource "nxos_physical_interface" "e1_10_intf" {
interface_id = "eth1/10"
description = "TF_DESCR"
layer = "Layer3"
admin_state = "up"
}
Before I applied configuration I prepared two interfaces:
- Eth1/10 - I left it as it is by default, so no configuration has been added. By default interfaces is administratively up
nxos# sh int eth1/10 | g admin
admin state is up, Dedicated Interface
admin fec state is auto, oper fec state is off
nxos# sh run int eth1/10
!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:51:03 2024
!Time: Thu Aug 29 10:51:47 2024
version 10.3(3) Bios:version
interface Ethernet1/10
nxos#
- Eth1/11 - I left it as it is by default, but I typed "shutdown" command to disable it.
nxos# sh int eth1/11 | g admin
admin state is down, Dedicated Interface
admin fec state is auto, oper fec state is off
nxos# sh run int eth1/11
!Command: show running-config interface Ethernet1/11
!Running configuration last done at: Thu Aug 29 10:51:03 2024
!Time: Thu Aug 29 10:51:57 2024
version 10.3(3) Bios:version
interface Ethernet1/11
shutdown
I enabled terraform debug logging by setting env var "TF_LOG" to "DEBUG". I applied configuration with Terraform. TF Output stated it's all applied, but once i got to the command line - no "no shutdown" command was present:
nxos# sh run int eth1/10
!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:52:21 2024
!Time: Thu Aug 29 10:52:30 2024
version 10.3(3) Bios:version
interface Ethernet1/10
description TF_DESCR
no switchport
I applied changes again and this time - Terraform discovered there are differences with admin state. I applied that and this time command showed up in the config:
nxos# sh run int eth1/10
!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:56:04 2024
!Time: Thu Aug 29 10:56:30 2024
version 10.3(3) Bios:version
interface Ethernet1/10
description TF_DESCR
no switchport
no shutdown
Same happened to Eth1/11 when I tested it, so initial state before applying changes via Terraform doesn't matter in this case.
Due to the fact I enabled Terraform logging, I was able to see what API calls were sent to Nexus.
2024-08-30T09:39:58.347+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:58 [DEBUG] HTTP Request: POST, https://sbx-nxos-mgmt.cisco.com/api/aaaLogin.json
2024-08-30T09:39:59.368+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:59 [DEBUG] Exit from Do method
2024-08-30T09:39:59.368+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:59 [DEBUG] HTTP Request: POST, https://sbx-nxos-mgmt.cisco.com/api/mo/sys/intf/phys-[eth1/10].json, {{"l1PhysIf":{"attributes":{"id":"eth1/10","FECMode":"auto","accessVlan":"vlan-1","adminSt":"up","autoNeg":"on","bw":"0","delay":"1","descr":"TF_DESCR","duplex":"auto","layer":"Layer3","linkLog":"default","linkDebounce":"100","linkDebounceLinkUp":"0","medium":"broadcast","mode":"access","mtu":"1500","nativeVlan":"vlan-1","speed":"auto","speedGroup":"auto","trunkVlans":"1-4094","uniDirectionalEthernet":"disable","userCfgdFlags":""}}}}
I recovered Eth1/10 to its initial state before I run terraform, but this time I simply did two REST API calls directly to NXOS to isolate the issue and see if this is related to Terraform or not.
First call - Retrieving a token:
TOKEN=$(curl -s -k -X POST https://sbx-nxos-mgmt.cisco.com/api/aaaLogin.json --data-raw '{ "aaaUser" : { "attributes" : { "name" : "x", "pwd" : "x" } } }' | jq .imdata[0].aaaLogin.attributes.token -r)
Second - modifying Eth1/10 using the payload I got from Terraform logs and by supplying token I got in the previous step:
curl -k --globoff -X POST 'https://sbx-nxos-mgmt.cisco.com/api/mo/sys/intf/phys-[eth1/10].json' --header 'Content-Type: application/json' --header 'Cookie: APIC-cookie='$TOKEN --data-raw '{ "l1PhysIf": { "attributes": { "id": "eth1/10", "FECMode": "auto", "accessVlan": "vlan-1", "adminSt": "up", "autoNeg": "on", "bw": "0", "delay": "1", "descr": "VIA_CURL", "duplex": "auto", "layer": "Layer3", "linkLog": "default", "linkDebounce": "100", "linkDebounceLinkUp": "0", "medium": "broadcast", "mode": "access", "mtu": "1500", "nativeVlan": "vlan-1", "speed": "auto", "speedGroup": "auto", "trunkVlans": "1-4094", "uniDirectionalEthernet": "disable", "userCfgdFlags": "" } } }'
The result was the same when i did this with Terraform.
interface Ethernet1/10
description VIA_CURL
no switchport
But once I sent the same call again, then interface has been modified as expected, including "no shutdown" command:
interface Ethernet1/10
description VIA_CURL
no switchport
no shutdown
To sum it up - unfortunately it seems to be an issue with NXAPI itself. This is something that should be raised to TAC for further investigation, because we need to send the same two requests to NXAPI to get desired result.
08-30-2024 01:28 AM - edited 08-30-2024 01:30 AM
I went through that issue and I did a small lab. I've got two news - good and bad one.
The good one is that this is not the Terraform's issue (nor nxos provider). The bad one is - this is the issue with NXAPI that NXOS TF Provider uses under the hood.
I used Sandbox NXOS to troubleshoot that. See my Tf code below:
terraform {
required_providers {
nxos = {
source = "CiscoDevNet/nxos"
version = "0.5.3"
}
}
}
provider "nxos" {
username = "x"
password = "x"
url = "https://sbx-nxos-mgmt.cisco.com"
}
resource "nxos_physical_interface" "e1_10_intf" {
interface_id = "eth1/10"
description = "TF_DESCR"
layer = "Layer3"
admin_state = "up"
}
Before I applied configuration I prepared two interfaces:
- Eth1/10 - I left it as it is by default, so no configuration has been added. By default interfaces is administratively up
nxos# sh int eth1/10 | g admin
admin state is up, Dedicated Interface
admin fec state is auto, oper fec state is off
nxos# sh run int eth1/10
!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:51:03 2024
!Time: Thu Aug 29 10:51:47 2024
version 10.3(3) Bios:version
interface Ethernet1/10
nxos#
- Eth1/11 - I left it as it is by default, but I typed "shutdown" command to disable it.
nxos# sh int eth1/11 | g admin
admin state is down, Dedicated Interface
admin fec state is auto, oper fec state is off
nxos# sh run int eth1/11
!Command: show running-config interface Ethernet1/11
!Running configuration last done at: Thu Aug 29 10:51:03 2024
!Time: Thu Aug 29 10:51:57 2024
version 10.3(3) Bios:version
interface Ethernet1/11
shutdown
I enabled terraform debug logging by setting env var "TF_LOG" to "DEBUG". I applied configuration with Terraform. TF Output stated it's all applied, but once i got to the command line - no "no shutdown" command was present:
nxos# sh run int eth1/10
!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:52:21 2024
!Time: Thu Aug 29 10:52:30 2024
version 10.3(3) Bios:version
interface Ethernet1/10
description TF_DESCR
no switchport
I applied changes again and this time - Terraform discovered there are differences with admin state. I applied that and this time command showed up in the config:
nxos# sh run int eth1/10
!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:56:04 2024
!Time: Thu Aug 29 10:56:30 2024
version 10.3(3) Bios:version
interface Ethernet1/10
description TF_DESCR
no switchport
no shutdown
Same happened to Eth1/11 when I tested it, so initial state before applying changes via Terraform doesn't matter in this case.
Due to the fact I enabled Terraform logging, I was able to see what API calls were sent to Nexus.
2024-08-30T09:39:58.347+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:58 [DEBUG] HTTP Request: POST, https://sbx-nxos-mgmt.cisco.com/api/aaaLogin.json
2024-08-30T09:39:59.368+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:59 [DEBUG] Exit from Do method
2024-08-30T09:39:59.368+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:59 [DEBUG] HTTP Request: POST, https://sbx-nxos-mgmt.cisco.com/api/mo/sys/intf/phys-[eth1/10].json, {{"l1PhysIf":{"attributes":{"id":"eth1/10","FECMode":"auto","accessVlan":"vlan-1","adminSt":"up","autoNeg":"on","bw":"0","delay":"1","descr":"TF_DESCR","duplex":"auto","layer":"Layer3","linkLog":"default","linkDebounce":"100","linkDebounceLinkUp":"0","medium":"broadcast","mode":"access","mtu":"1500","nativeVlan":"vlan-1","speed":"auto","speedGroup":"auto","trunkVlans":"1-4094","uniDirectionalEthernet":"disable","userCfgdFlags":""}}}}
I recovered Eth1/10 to its initial state before I run terraform, but this time I simply did two REST API calls directly to NXOS to isolate the issue and see if this is related to Terraform or not.
First call - Retrieving a token:
TOKEN=$(curl -s -k -X POST https://sbx-nxos-mgmt.cisco.com/api/aaaLogin.json --data-raw '{ "aaaUser" : { "attributes" : { "name" : "x", "pwd" : "x" } } }' | jq .imdata[0].aaaLogin.attributes.token -r)
Second - modifying Eth1/10 using the payload I got from Terraform logs and by supplying token I got in the previous step:
curl -k --globoff -X POST 'https://sbx-nxos-mgmt.cisco.com/api/mo/sys/intf/phys-[eth1/10].json' --header 'Content-Type: application/json' --header 'Cookie: APIC-cookie='$TOKEN --data-raw '{ "l1PhysIf": { "attributes": { "id": "eth1/10", "FECMode": "auto", "accessVlan": "vlan-1", "adminSt": "up", "autoNeg": "on", "bw": "0", "delay": "1", "descr": "VIA_CURL", "duplex": "auto", "layer": "Layer3", "linkLog": "default", "linkDebounce": "100", "linkDebounceLinkUp": "0", "medium": "broadcast", "mode": "access", "mtu": "1500", "nativeVlan": "vlan-1", "speed": "auto", "speedGroup": "auto", "trunkVlans": "1-4094", "uniDirectionalEthernet": "disable", "userCfgdFlags": "" } } }'
The result was the same when i did this with Terraform.
interface Ethernet1/10
description VIA_CURL
no switchport
But once I sent the same call again, then interface has been modified as expected, including "no shutdown" command:
interface Ethernet1/10
description VIA_CURL
no switchport
no shutdown
To sum it up - unfortunately it seems to be an issue with NXAPI itself. This is something that should be raised to TAC for further investigation, because we need to send the same two requests to NXAPI to get desired result.
08-30-2024 03:14 AM
Thanks for such a comprehensive and precise translation.
Great that you presented these tests.
I got around this problem in a different way. I simply create two resources with a different name that do “no shutdown” (the catch is that both must have the same content, i.e. when you set “no switchport”, “description”, “no shutdown” on one, even though in theory you are only interested in letting “no shutdown” go one more time, you must also repeat “no switchport”, “description” in the other identical resource).
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide