MultiPod ACI infra-VLAN mismatch

[David Yang]

We have MultiPod implementation between two data centers. Two APIC's in Pod1 are managing all switches in both pods. When apic-3 was installed in Pod2, it was unable to join the cluster because of infra-VLAN mismatch. All 3 APIC's are configured infra-VLAN 3967 but apic-3 reported Pod2 leaf switch it connected with using 4094. All APIC's and switches are running same version 3.2(2l). apic-3 was configured with the same Fabric-id, TEP except Pod-id, Controller-id and Controller-name. Is there anything we have missed?

Topology:

apic1----Leaf521----Spine-----IPN-----Spine----Leaf426----apic3

Error message: Link between node 426 slot 1 port 1 and node 3 slot 2 port 1 is invalid. Infra vlan mismatch

****************POD1****************************

apic1# bash
admin@apic1:~> show lldptool in eth2-1
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
Chassis ID TLV
MAC: 40:01:7a:f8:76:06
Port ID TLV
Local: Eth1/47
Time to Live TLV
120
Port Description TLV
topology/pod-1/paths-521/pathep-[eth1/47]
System Name TLV
SFP521
System Description TLV
topology/pod-1/node-521
System Capabilities TLV
System capabilities: Bridge, Router
Enabled capabilities: Bridge, Router
Management Address TLV
MAC: 40:01:7a:f8:76:06
Ifindex: 83886080
Cisco 4-wire Power-via-MDI TLV
4-Pair PoE not supported
Spare pair Detection/Classification not required
PD Spare pair Desired State: Disabled
PSE Spare pair Operational State: Disabled
Cisco Port Mode TLV
0
Cisco Port State TLV
1
Cisco Serial Number TLV
FDO21272HFC
Cisco Model TLV
N9K-C93180YC-EX
Cisco Firmware Version TLV
n9000-13.0(2k)
Cisco Node Role TLV
1
Cisco Infra VLAN TLV
3967
Cisco Name TLV
SFP521
Cisco Fabric Name TLV
GBRX-1
Cisco Node IP TLV
IPv4:172.16.192.95
Cisco Node ID TLV
521
Cisco POD ID TLV
1
Cisco Appliance Vector TLV
Id: 2
IPv4: 172.16.0.2
UUID: b6e49a24-2c4e-11e8-bec3-f303cc3ccbbb
Id: 1
IPv4: 172.16.0.1
UUID: e393157c-1a84-11e8-b540-17fc89beb3d0
End of LLDPDU TLV

SFP521# vsh_lc
vsh_lc
module-1# show system internal eltmc info vlan brief
VLAN-Info
VlanId HW_VlanId Type Access_enc Access_enc Fabric_enc Fabric_enc BDVlan
Type Type
==================================================================================
5 9 BD_CTRL_VLAN 802.1q 3967 VXLAN 16777209 0 Unknown 0

****************POD2****************************
apic3# ifconfig
bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500
inet6 fe80::4201:7aff:fe35:174d prefixlen 64 scopeid 0x20<link>
ether 40:01:7a:35:17:4d txqueuelen 1000 (Ethernet)
RX packets 308 bytes 116424 (113.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5124 bytes 382493 (373.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

bond1: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500
inet6 fe80::1a80:90ff:fe7e:6e04 prefixlen 64 scopeid 0x20<link>
ether 18:80:90:7e:6e:04 txqueuelen 1000 (Ethernet)
RX packets 12368 bytes 928393 (906.6 KiB)
RX errors 0 dropped 5 overruns 0 frame 0
TX packets 762 bytes 191820 (187.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

bond0.3967: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1496
inet 172.16.0.3 netmask 255.255.255.255 broadcast 172.16.0.3
inet6 fe80::4201:7aff:fe35:174d prefixlen 64 scopeid 0x20<link>
ether 40:01:7a:35:17:4d txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4800 bytes 201840 (197.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

apic3# bash
admin@apic3:~> show lldptool in eth2-1
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
Chassis ID TLV
MAC: 40:01:7a:f8:75:16
Port ID TLV
Local: Eth1/1
Time to Live TLV
120
Port Description TLV
topology/pod-2/paths-426/pathep-[eth1/1]
System Name TLV
SFP426
System Description TLV
topology/pod-2/node-426
System Capabilities TLV
System capabilities: Bridge, Router
Enabled capabilities: Bridge, Router
Management Address TLV
MAC: 40:01:7a:f8:75:16
Ifindex: 83886080
Cisco 4-wire Power-via-MDI TLV
4-Pair PoE not supported
Spare pair Detection/Classification not required
PD Spare pair Desired State: Disabled
PSE Spare pair Operational State: Disabled
Cisco Port State TLV
1
Cisco Port Mode TLV
0
Cisco Serial Number TLV
FDO212805A6
Cisco Model TLV
N9K-C93180YC-EX
Cisco Firmware Version TLV
n9000-13.0(2k)
Cisco Node Role TLV
1
Cisco Infra VLAN TLV
4094
Cisco Name TLV
SFP426
Cisco Fabric Name TLV
GBRX-1
Cisco Node IP TLV
IPv4:172.17.56.88
Cisco Node ID TLV
426
Cisco POD ID TLV
2
Cisco Appliance Vector TLV
Id: 1
IPv4: 172.16.0.1
UUID: e393157c-1a84-11e8-b540-17fc89beb3d0
Id: 2
IPv4: 172.16.0.2
UUID: b6e49a24-2c4e-11e8-bec3-f303cc3ccbbb
End of LLDPDU TLV

SFP426# show lldp nei
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
apic3 Eth1/1 120 eth2-1
Spine402 Eth1/49 120 BR Eth1/7
Spine402 Eth1/50 120 BR Eth1/8
Spine401 Eth1/53 120 BR Eth1/7
Spine401 Eth1/54 120 BR Eth1/8
Total entries displayed: 5
 

SFP426# vsh_lc
vsh_lc
module-1# show system internal eltmc info vlan brief
VLAN-Info
VlanId HW_VlanId Type Access_enc Access_enc Fabric_enc Fabric_enc BDVlan
Type Type
==================================================================================
module-1# exit
SFP426# show vlan extended

VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------

VLAN Type Vlan-mode Encap
---- ----- ---------- -------------------------------

Thanks,

David

[Nuno Marques]

Hi David,

start with a clean wipe in leaf 426, probably with was being used before.

If you have FEX attached, disconnect them, perform the wipe and reconnect after apic 3 joins the cluster.

Regards.

Nuno

[David Yang]

Thank you Nuno.

Leaf-426 and leaf-428 (another leaf connecting apic-3) were wiped, decommissioned and re-registered to the fabric. Apic-3 was also wiped and reconfigured but we saw the same issue. What command can we use to verify infra-VLAN on the leaf?

Thanks,

David

[Robert Burns]

You're using the right commands which is to look at the LLDP the APIC3 is receiving from that leaf. I don't suspect you wiped Leaf-426 correctly, as it still shows VLAN -4094 per the output above. Get onto the console, and issue "setup-clean-config.sh" then "reload" once the wipe is complete. It "should" discover with the correct 3967 VLAN after that.

Robert

[David Yang]

Thank you Robert.

I don't have physical access to the switch so I did the wipe "setup-clean-config.sh" "reload" by attaching to the switch from the working APIC-1. This was done a few times already. Does the leaf store infra-VLAN in its config? if yes, how can we check this info locally?

Thanks,

David

[David Yang · Accepted solution]

For anyone who may have the same question on how to verify the infra-VLAN configuration locally on the switch, the command is "moquery -c lldpInst".

In our case, all 10 spine and leaf switches in a Pod had infra-VLAN 4094. The problem was resolved by erasing config and reload on all 10 switches at the same time.

Thanks,
David

[ChaimTaub7725]

where in ACI is it specified that pod 2 should be a different infra vlan?