cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1219
Views
3
Helpful
2
Replies

[Terraform] Problem with adding "no shutdown" to configuration

Having a "clean" terraform directory (main.tf, terraform.tfvars, variables.tf) executes the command "terraform init; terraform apply -auto-approve", everything passes fine, adds the first part of the code marked "part-1", all the commands along with "no shutdown" are added. When I add the code "part-2", coemend "terraform apply -auto-approve" usually all commands except "no shutdown" are added, relatively rarely all will be added, so it happens quite randomly. When I let go of the same terraform again without changes it will add the command "no shutdown" correctly. Below I paste the code of "part-1" and "part-2" and the output from terraform.

Terraform code: main.tf

terraform {
required_providers {
nxos = {
source = "CiscoDevNet/nxos"
version = "0.5.3"
}
}
}

provider "nxos" {
username = var.nxos_username
password = var.nxos_password
url = var.nxos_url
}

/*============================================ common ============================================*/

resource "nxos_feature_lldp" "lldp" {
admin_state = "enabled"
}

/*=====================================*/

resource "nxos_rest" "cdpEntity" {
dn = "sys/cdp"
class_name = "cdpEntity"
}

//============================================== part-1 ==============================================
//============================================== part-1 ==============================================
//============================================== part-1 ==============================================

/*============================================ description & L3 ============================================*/

resource "nxos_physical_interface" "desc-L3" {
interface_id = "eth1/5"
description = "desc1"
layer = "Layer3"
admin_state = "up"
user_configured_flags = "admin_state"
}

/*============================================ lldp ============================================*/

resource "nxos_rest" "lldpInst" {
depends_on = [nxos_feature_lldp.lldp]
dn = "sys/lldp/inst"
class_name = "lldpInst"
children = [
{
rn = "if-[eth1/5]"
class_name = "lldpIf"
content = {
adminRxSt = "disabled",
adminTxSt = "disabled",
id = "eth1/5"
}
}
]
}

/*============================================ cdp ============================================*/

resource "nxos_rest" "cdpInst" {
depends_on = [nxos_rest.cdpEntity]
dn = "sys/cdp/inst"
class_name = "cdpInst"
children = [
{
rn = "if-[eth1/5]"
class_name = "cdpIf"
content = {
adminSt = "disabled",
id = "eth1/5"
}
}
]
}

//============================================== part-2 ==============================================
//============================================== part-2 ==============================================
//============================================== part-2 ==============================================

/*============================================ description & L3 ============================================*/

resource "nxos_physical_interface" "desc-L3v2" {
interface_id = "eth1/6"
description = "desc2"
layer = "Layer3"
admin_state = "up"
user_configured_flags = "admin_state"
}

/*============================================ lldp ============================================*/

resource "nxos_rest" "lldpInstv2" {
depends_on = [nxos_feature_lldp.lldp]
dn = "sys/lldp/inst"
class_name = "lldpInst"
children = [
{
rn = "if-[eth1/6]"
class_name = "lldpIf"
content = {
adminRxSt = "disabled",
adminTxSt = "disabled",
id = "eth1/6"
}
}
]
}

/*============================================ cdp ============================================*/

resource "nxos_rest" "cdpInstv2" {
depends_on = [nxos_rest.cdpEntity]
dn = "sys/cdp/inst"
class_name = "cdpInst"
children = [
{
rn = "if-[eth1/6]"
class_name = "cdpIf"
content = {
adminSt = "disabled",
id = "eth1/6"
}
}
]
}

Output:

Terraform will perform the following actions:

# nxos_physical_interface.desc-L3v2 will be created
+ resource "nxos_physical_interface" "desc-L3v2" {
+ access_vlan = "vlan-1"
+ admin_state = "up"
+ auto_negotiation = "on"
+ bandwidth = 0
+ delay = 1
+ description = "desc2"
+ duplex = "auto"
+ fec_mode = "auto"
+ id = (known after apply)
+ interface_id = "eth1/6"
+ layer = "Layer3"
+ link_debounce_down = 100
+ link_debounce_up = 0
+ link_logging = "default"
+ medium = "broadcast"
+ mode = "access"
+ mtu = 1500
+ native_vlan = "vlan-1"
+ speed = "auto"
+ speed_group = "auto"
+ trunk_vlans = "1-4094"
+ uni_directional_ethernet = "disable"
+ user_configured_flags = "admin_state"
}

Output form Nexus9000:

krzysztofmaciejewskiit_0-1723125140530.png

I have one more problem, because when I delete a part of the code in e.g. part two, responsible for cdp or lldp it says it is destroyed, but it stays on the Nexus configuration and I don't know why all the other stuff gets deleted and those 3 lines of code (seen on the switch) stay, even though the terraform logs clearly say "destroyed", but I will devote a second post on the community to this, because it seems to have nothing to do with each other.
https://community.cisco.com/t5/devnet-general-discussions/after-removing-resource-cdp-ldp-configuration-is-left-on-the/m-p/5157851#M2553

krzysztofmaciejewskiit_1-1723125355034.pngkrzysztofmaciejewskiit_2-1723125364706.pngkrzysztofmaciejewskiit_3-1723125372631.png

 

 

 

1 Accepted Solution

Accepted Solutions

mfr-6
Spotlight
Spotlight

hi @krzysztofmaciejewskiit 

I went through that issue and I did a small lab. I've got two news - good and bad one.

The good one is that this is not the Terraform's issue (nor nxos provider). The bad one is - this is the issue with NXAPI that NXOS TF Provider uses under the hood.

I used Sandbox NXOS to troubleshoot that. See my Tf code below:

 

terraform {
  required_providers {
    nxos = {
      source  = "CiscoDevNet/nxos"
      version = "0.5.3"
    }
  }
}

provider "nxos" {
  username = "x"
  password = "x"
  url      = "https://sbx-nxos-mgmt.cisco.com"
}

resource "nxos_physical_interface" "e1_10_intf" {
  interface_id = "eth1/10"
  description = "TF_DESCR"
  layer = "Layer3"
  admin_state  = "up"
}

 

Before I applied configuration I prepared two interfaces:

- Eth1/10 - I left it as it is by default, so no configuration has been added. By default interfaces is administratively up

 

nxos# sh int eth1/10 | g admin
admin state is up, Dedicated Interface
    admin fec state is auto, oper fec state is off
nxos# sh run int eth1/10

!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:51:03 2024
!Time: Thu Aug 29 10:51:47 2024

version 10.3(3) Bios:version

interface Ethernet1/10
nxos#

 

- Eth1/11 - I left it as it is by default, but I typed "shutdown" command to disable it.

 

nxos# sh int eth1/11 | g admin
admin state is down, Dedicated Interface
    admin fec state is auto, oper fec state is off
nxos# sh run int eth1/11

!Command: show running-config interface Ethernet1/11
!Running configuration last done at: Thu Aug 29 10:51:03 2024
!Time: Thu Aug 29 10:51:57 2024

version 10.3(3) Bios:version

interface Ethernet1/11
  shutdown

 

I enabled terraform debug logging by setting env var "TF_LOG" to "DEBUG". I applied configuration with Terraform. TF Output stated it's all applied, but once i got to the command line - no "no shutdown" command was present:

 

nxos# sh run int eth1/10

!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:52:21 2024
!Time: Thu Aug 29 10:52:30 2024

version 10.3(3) Bios:version

interface Ethernet1/10
  description TF_DESCR
  no switchport

 

I applied changes again and this time - Terraform discovered there are differences with admin state. I applied that and this time command showed up in the config:

 

nxos# sh run int eth1/10

!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:56:04 2024
!Time: Thu Aug 29 10:56:30 2024

version 10.3(3) Bios:version

interface Ethernet1/10
  description TF_DESCR
  no switchport
  no shutdown

 

Same happened to Eth1/11 when I tested it, so initial state before applying changes via Terraform doesn't matter in this case.

Due to the fact I enabled Terraform logging, I was able to see what API calls were sent to Nexus.

 

2024-08-30T09:39:58.347+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:58 [DEBUG] HTTP Request: POST, https://sbx-nxos-mgmt.cisco.com/api/aaaLogin.json
2024-08-30T09:39:59.368+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:59 [DEBUG] Exit from Do method
2024-08-30T09:39:59.368+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:59 [DEBUG] HTTP Request: POST, https://sbx-nxos-mgmt.cisco.com/api/mo/sys/intf/phys-[eth1/10].json, {{"l1PhysIf":{"attributes":{"id":"eth1/10","FECMode":"auto","accessVlan":"vlan-1","adminSt":"up","autoNeg":"on","bw":"0","delay":"1","descr":"TF_DESCR","duplex":"auto","layer":"Layer3","linkLog":"default","linkDebounce":"100","linkDebounceLinkUp":"0","medium":"broadcast","mode":"access","mtu":"1500","nativeVlan":"vlan-1","speed":"auto","speedGroup":"auto","trunkVlans":"1-4094","uniDirectionalEthernet":"disable","userCfgdFlags":""}}}}

 

I recovered Eth1/10 to its initial state before I run terraform, but this time I simply did two REST API calls directly to NXOS to isolate the issue and see if this is related to Terraform or not.

First call - Retrieving a token:

TOKEN=$(curl -s -k -X POST https://sbx-nxos-mgmt.cisco.com/api/aaaLogin.json --data-raw '{ "aaaUser" : { "attributes" : { "name" : "x", "pwd" : "x" } } }' | jq .imdata[0].aaaLogin.attributes.token -r)

Second - modifying Eth1/10 using the payload I got from Terraform logs and by supplying token I got in the previous step:

curl -k --globoff -X POST 'https://sbx-nxos-mgmt.cisco.com/api/mo/sys/intf/phys-[eth1/10].json' --header 'Content-Type: application/json' --header 'Cookie: APIC-cookie='$TOKEN --data-raw '{ "l1PhysIf": { "attributes": { "id": "eth1/10", "FECMode": "auto", "accessVlan": "vlan-1", "adminSt": "up", "autoNeg": "on", "bw": "0", "delay": "1", "descr": "VIA_CURL", "duplex": "auto", "layer": "Layer3", "linkLog": "default", "linkDebounce": "100", "linkDebounceLinkUp": "0", "medium": "broadcast", "mode": "access", "mtu": "1500", "nativeVlan": "vlan-1", "speed": "auto", "speedGroup": "auto", "trunkVlans": "1-4094", "uniDirectionalEthernet": "disable", "userCfgdFlags": "" } } }'

The result was the same when i did this with Terraform.

 

interface Ethernet1/10
  description VIA_CURL
  no switchport

 

But once I sent the same call again, then interface has been modified as expected, including "no shutdown" command:

 

interface Ethernet1/10
  description VIA_CURL
  no switchport
  no shutdown

 


To sum it up - unfortunately it seems to be an issue with NXAPI itself. This is something that should be raised to TAC for further investigation, because we need to send the same two requests to NXAPI to get desired result.


 

 

 

 

Mateusz Frak NetDevOps | DevNet | Automation DevNet Expert #20240068
Please mark this post as helpful if it solves your issue, to make this visible for other users, thank you!

View solution in original post

2 Replies 2

mfr-6
Spotlight
Spotlight

hi @krzysztofmaciejewskiit 

I went through that issue and I did a small lab. I've got two news - good and bad one.

The good one is that this is not the Terraform's issue (nor nxos provider). The bad one is - this is the issue with NXAPI that NXOS TF Provider uses under the hood.

I used Sandbox NXOS to troubleshoot that. See my Tf code below:

 

terraform {
  required_providers {
    nxos = {
      source  = "CiscoDevNet/nxos"
      version = "0.5.3"
    }
  }
}

provider "nxos" {
  username = "x"
  password = "x"
  url      = "https://sbx-nxos-mgmt.cisco.com"
}

resource "nxos_physical_interface" "e1_10_intf" {
  interface_id = "eth1/10"
  description = "TF_DESCR"
  layer = "Layer3"
  admin_state  = "up"
}

 

Before I applied configuration I prepared two interfaces:

- Eth1/10 - I left it as it is by default, so no configuration has been added. By default interfaces is administratively up

 

nxos# sh int eth1/10 | g admin
admin state is up, Dedicated Interface
    admin fec state is auto, oper fec state is off
nxos# sh run int eth1/10

!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:51:03 2024
!Time: Thu Aug 29 10:51:47 2024

version 10.3(3) Bios:version

interface Ethernet1/10
nxos#

 

- Eth1/11 - I left it as it is by default, but I typed "shutdown" command to disable it.

 

nxos# sh int eth1/11 | g admin
admin state is down, Dedicated Interface
    admin fec state is auto, oper fec state is off
nxos# sh run int eth1/11

!Command: show running-config interface Ethernet1/11
!Running configuration last done at: Thu Aug 29 10:51:03 2024
!Time: Thu Aug 29 10:51:57 2024

version 10.3(3) Bios:version

interface Ethernet1/11
  shutdown

 

I enabled terraform debug logging by setting env var "TF_LOG" to "DEBUG". I applied configuration with Terraform. TF Output stated it's all applied, but once i got to the command line - no "no shutdown" command was present:

 

nxos# sh run int eth1/10

!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:52:21 2024
!Time: Thu Aug 29 10:52:30 2024

version 10.3(3) Bios:version

interface Ethernet1/10
  description TF_DESCR
  no switchport

 

I applied changes again and this time - Terraform discovered there are differences with admin state. I applied that and this time command showed up in the config:

 

nxos# sh run int eth1/10

!Command: show running-config interface Ethernet1/10
!Running configuration last done at: Thu Aug 29 10:56:04 2024
!Time: Thu Aug 29 10:56:30 2024

version 10.3(3) Bios:version

interface Ethernet1/10
  description TF_DESCR
  no switchport
  no shutdown

 

Same happened to Eth1/11 when I tested it, so initial state before applying changes via Terraform doesn't matter in this case.

Due to the fact I enabled Terraform logging, I was able to see what API calls were sent to Nexus.

 

2024-08-30T09:39:58.347+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:58 [DEBUG] HTTP Request: POST, https://sbx-nxos-mgmt.cisco.com/api/aaaLogin.json
2024-08-30T09:39:59.368+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:59 [DEBUG] Exit from Do method
2024-08-30T09:39:59.368+0200 [DEBUG] provider.terraform-provider-nxos_v0.5.3.exe: 2024/08/30 09:39:59 [DEBUG] HTTP Request: POST, https://sbx-nxos-mgmt.cisco.com/api/mo/sys/intf/phys-[eth1/10].json, {{"l1PhysIf":{"attributes":{"id":"eth1/10","FECMode":"auto","accessVlan":"vlan-1","adminSt":"up","autoNeg":"on","bw":"0","delay":"1","descr":"TF_DESCR","duplex":"auto","layer":"Layer3","linkLog":"default","linkDebounce":"100","linkDebounceLinkUp":"0","medium":"broadcast","mode":"access","mtu":"1500","nativeVlan":"vlan-1","speed":"auto","speedGroup":"auto","trunkVlans":"1-4094","uniDirectionalEthernet":"disable","userCfgdFlags":""}}}}

 

I recovered Eth1/10 to its initial state before I run terraform, but this time I simply did two REST API calls directly to NXOS to isolate the issue and see if this is related to Terraform or not.

First call - Retrieving a token:

TOKEN=$(curl -s -k -X POST https://sbx-nxos-mgmt.cisco.com/api/aaaLogin.json --data-raw '{ "aaaUser" : { "attributes" : { "name" : "x", "pwd" : "x" } } }' | jq .imdata[0].aaaLogin.attributes.token -r)

Second - modifying Eth1/10 using the payload I got from Terraform logs and by supplying token I got in the previous step:

curl -k --globoff -X POST 'https://sbx-nxos-mgmt.cisco.com/api/mo/sys/intf/phys-[eth1/10].json' --header 'Content-Type: application/json' --header 'Cookie: APIC-cookie='$TOKEN --data-raw '{ "l1PhysIf": { "attributes": { "id": "eth1/10", "FECMode": "auto", "accessVlan": "vlan-1", "adminSt": "up", "autoNeg": "on", "bw": "0", "delay": "1", "descr": "VIA_CURL", "duplex": "auto", "layer": "Layer3", "linkLog": "default", "linkDebounce": "100", "linkDebounceLinkUp": "0", "medium": "broadcast", "mode": "access", "mtu": "1500", "nativeVlan": "vlan-1", "speed": "auto", "speedGroup": "auto", "trunkVlans": "1-4094", "uniDirectionalEthernet": "disable", "userCfgdFlags": "" } } }'

The result was the same when i did this with Terraform.

 

interface Ethernet1/10
  description VIA_CURL
  no switchport

 

But once I sent the same call again, then interface has been modified as expected, including "no shutdown" command:

 

interface Ethernet1/10
  description VIA_CURL
  no switchport
  no shutdown

 


To sum it up - unfortunately it seems to be an issue with NXAPI itself. This is something that should be raised to TAC for further investigation, because we need to send the same two requests to NXAPI to get desired result.


 

 

 

 

Mateusz Frak NetDevOps | DevNet | Automation DevNet Expert #20240068
Please mark this post as helpful if it solves your issue, to make this visible for other users, thank you!

Thanks for such a comprehensive and precise translation.
Great that you presented these tests.

I got around this problem in a different way. I simply create two resources with a different name that do “no shutdown” (the catch is that both must have the same content, i.e. when you set “no switchport”, “description”, “no shutdown” on one, even though in theory you are only interested in letting “no shutdown” go one more time, you must also repeat “no switchport”, “description” in the other identical resource).