ACI leaf switches are not discovered anymore after a factory reset

[driss JABBAR]

Hello Community,

Our ACI starter-kit was working fine until i decided to do a factory reset and start the config again from the scratch.

My problem now is that our leaf switches are not discovered anymore by the apic did some check and i can confirm that my leaf has the apic as an LLDP neighbor but apic can't see any leaf as lldp neighbor.

My ACI version is 2.2(2i).

The reset has been executed on the apic as well as on all fabric switches 

any advice will realy be appreciated

Thanks

[Tomas de Leon]

Single APIC or 3 APICs?

• If 3 APICs, power off APIC2 & APIC3
• reload the leaf and spines
• If still issue, check & verify all cabling
• On the switches, run the command "set inservice"
• Verify that leaf1 can see APIC in "show lldp neighbors"

T.

[driss JABBAR]

Hello Thomas,

Thank you for your reply 

actually 

I have only one APIC.

i have reloaded the leaf and spines many times

the cabling is correct as everything was working fine before i reset the system

set inservice commande is executed but still not discoverd

the leaf can see the apic but the apic can"t see the leaf as lldp neighbor

[pradeepser]

What is leaf/spine firmware version?

[driss JABBAR]

the version is 2.2(2i) 

[micgarc2 · Accepted solution]

Driss,

wiringIssues : infra-vlan-mismatch

This can sometimes happen by connecting a leaf from one fabric to another therefore it having a different vlan and fabric domain. Basically the leaf keeps the same infra VLAN from the fabric it was originally apart of. If you remember doing this, try removing that leaf from the fabric to see if that issue goes away. Before you add an existing switch to a new fabric you need to clean the configuration properly and reload it. My guess is that your APIC and leaf have different infra VLANs. Please verify this.

To verify on the leaf:

leaf1#vsh_lc

module-1# show system internal eltmc info vlan brief | grep BD_CTRL_VLAN

      3        7    BD_CTRL_VLAN    802.1q      4093     VXLAN  16777209       0

To verify on the APIC:

rtp-pod3-apic1# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2-1
MII Status: up
MII Polling Interval (ms): 60
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2-1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 58:97:bd:e6:65:20
Slave queue ID: 0

Slave Interface: eth2-2 <----active interface
MII Status: up
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 58:97:bd:e6:65:21
Slave queue ID: 0

rtp-pod3-apic1# show lldptool in eth2-2
Error: Invalid argument 'lldptool in eth2-2 '. Please check syntax in command reference guide
rtp-pod3-apic1# bash
admin@rtp-pod3-apic1:~> show lldptool in eth2-2
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
Chassis ID TLV
----snippet---
Cisco Infra VLAN TLV
4093

Regards,

Michael G.

Thank you for participating in the Cisco Support Forum for ACI! If you have other questions related to this post, please let us know. If this response answers your questions, please mark this post "answered" and assign a rating to the response(s) provided. This will help notify other viewers that your question(s) is answered and this helps us provide better responses for this and future questions.

[driss JABBAR]

Hello Michael,

Thank you for your response

the leaf switch dont have any vlan on it as i reset it many times

module-1# show system internal eltmc info vlan summary

VLAN Summary
Control BD: 0
User BD Count: 0
Extrn BD Count: 0
FD Count: 0
FD VXLAN Count: 0
Total: 0
module-1#

module-1# show system internal eltmc info vlan brief | grep BD_CTRL_VLAN
module-1# show system internal eltmc info vlan brief
VLAN-Info
VlanId HW_VlanId Type Access_enc Access_enc Fabric_enc Fabric_enc BD
Vlan
Type Type

================================================================================
==
module-1#

but in the apic i can see that the vlan used is 4001 even if during my configuration i gave 4095, i think the factory reset in the apic is not working properly because the vlan 4001 was the infra_vlan during the first startup.

i will try to give my apic the vlan 4001 again and see what's will happen.

Best regards

[driss JABBAR]

It's working now, i had to give the the infa vlan 4001 that was used for the first installation.

it means that even if did the reset factory of the apic many times and i gave different infra vlan but the apic kept using the vlan 4001

it might be a bug.

[Joseph Young]

Hello,

What process did you take to wipe the switch? setup-clean-config.sh was the old way to wiping and in newer version (2.x and later I believe) you should use "acidiag touch clean" on the leaf (the same as you would on an apic). Can you confirm if you tried wiping the leaf that way as well?

Joe

[driss JABBAR]

I tried both way but i had the same result

[Tomas de Leon]

Take a look at the output from the following commands run on the APIC:

apic# bash
apic1:~> show lldptool in eth2-1
apic1:~> show lldptool in eth2-2
apic1:~> show lldptool out eth2-1
apic1:~> show lldptool out eth2-2

Look for what value is in the "Cisco Infra VLAN TLV"

T.

[Tomas de Leon · Accepted solution]

On the APIC CLI,  please run "acidiag restart dhcpd" and then reload the leaf attached to the APIC.  

On the CLI of the leaf & spines at the "(none)" prompt, run the following commands:

openssl asn1parse < /securedata/ssl/server.crt | grep PRINTABLESTRING
openssl asn1parse < /securedata/ssl/server.crt | grep UTF8STRING

Let me know what happens

Cheers!

T.

[driss JABBAR]

I have reloaded the dhcp service and rebooted the leaf with no result

here is the result of the commands

(none)# openssl asn1parse < /securedata/ssl/server.crt | grep PRINTABLESTRING
51:d=5 hl=2 l= 13 prim: PRINTABLESTRING :Cisco Systems
75:d=5 hl=2 l= 22 prim: PRINTABLESTRING :Cisco Manufacturing CA
142:d=5 hl=2 l= 34 prim: PRINTABLESTRING :PID:N9K-C93180YC-EX SN:FDO210721GX
187:d=5 hl=2 l= 11 prim: PRINTABLESTRING :FDO210721xx
(none)# openssl asn1parse < /securedata/ssl/server.crt | grep UTF8STRING
(none)#

[Tomas de Leon]

Ok the CERTs are good.

Please run the "Date" command on the APIC and the Leaf attache to the APIC.  The date & time need to be in sync (meaning relatively close, like 60 minutes).

If there is a big difference, set the date on the leaf to match the  APIC.  Then you will need to SAVE running clock to system clock.  And then reboot the leaf.

For Example:

admin@leaf1# date --set="30 JUN 2016 08:39:30"
Thu Jun 30 08:39:30 UTC 2016

admin@leaf1# setup-hwclock.sh
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done
Software clock (date) - Thu Jun 30 08:42:02 UTC 2016
Hardware clock (hwclock) - Fri Jun 30 12:42:18 2017 -0.984876 seconds

admin@leaf1# setup-hwclock.sh --systohc
In progress
In progress
In progress
In progress
Done
Software clock (date) - Thu Jun 30 08:43:02 UTC 2016
Hardware clock (hwclock) - Thu Jun 30 08:43:03 2016 -0.500399 seco

[driss JABBAR]

Hello again Thomas,

actually the clock is in synch

i see something strange when i was looking on the summary of the interface that connect my leaf to the apic,here is the output

(none)# cd /mit/sys/lldp/inst/if-\[eth1--1\]/
(none)# cat summary
# LLDP Interface
id : eth1/1
adminRxSt : enabled
adminSt : enabled
adminTxSt : enabled
childAction :
descr :
dn : sys/lldp/inst/if-[eth1/1]
lcOwn : local
mac : 70:DB:98:32:C7:XX
modTs : 2017-07-06T07:18:19.068+00:00
monPolDn : uni/fabric/monfab-default
name :
operRxSt : enabled
operTxSt : enabled
portDesc :
portMode : normal
portVlan : unspecified
rn : if-[eth1/1]
status :
sysDesc :
wiringIssues : infra-vlan-mismatch