ACI leaf doesn't send dhcp discover

[config]

Hi team,

I can't to add my first leaf to the apic.  When i enter the tcpdump commad for the kpm_inb interface , i get nothing.

(none)# tcpdump -ni kpm_inb port 67 or 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on kpm_inb, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

When i check the errors on the switch using moquery -c faultInfo command, i get the following error "F0341: fltDhcpClientIfIfOperStFailed" . I have already tried rebooting  the switch, but that didn't help.

Could you help me to find the reason?

@cisco ACI

[RedNectar]

Hi @config ,

Have you allocated the Leaf an ID yet? (Under Fabric > Inventory > Fabric Membership |> Nodes Pending Registration)

AFAIK the leaf will NOT send a DHCP request unless it has a Leaf ID

[config]

Hi @RedNectar 

I can see the node using lldp. But I don't have a node in Fabric Membership.

[RedNectar]

Hi @config ,

Is the leaf brand-new - i.e. never been used for ACI before? If the leaf is not showing up under Fabric > Inventory > Fabric Membership |> Nodes Pending Registration and has been used for ACI before then I suspect it hasn't been cleaned up properly.

The steps to reset an ACI Fabric are in an earlier post https://community.cisco.com/t5/application-centric-infrastructure/factory-reset-apic-and-nodes/m-p/3408496/highlight/true#M4879

If you have access to the leaf switch, it would be worth seeing if the leaf switch sees the APIC as a neighbour.

If the switch is brand new - you'll probably need to call TAC

And finally a hint:

---

When posting pictures inline -especially if it is a screenshot, you'll probably then want to click on the image and make the image large - like this.

This means you pictures are actually SEEN (a) in the email that gets sent to subscribers and (b) anyone who looks at this post in the future.

[config]

Hi @RedNectar 

- Yes, this switch has never been used for ACI before. 

The diagnostic command on the "show discoveryissues" shows the following:

(none)# show discoveryissues
================================================================================
Check 1 Platform Type
================================================================================
Test01 Retrieving Node Role PASSED
[Info] Current node role: LEAF
[Info] Please check CH09 DHCP status section for configured node role
================================================================================
Check 2 FPGA/BIOS in sync test
================================================================================
Test01 FPGA version check PASSED
[Info] No issues found for FPGA versions
Test02 BIOS version check PASSED
[Info] No issues found for BIOS versions
================================================================================
Check 3 HW Modules Check
================================================================================
Test01 Fans status check PASSED
[Info] All fans status is ok
Test02 Power Supply status check FAILED
[Warn] Operational state of sys/ch/psuslot-1/psu is: shut
[Info] Ignore this if it is a redundant power supply
Test03 Fan Tray status check PASSED
[Info] All FanTrays status is ok
Test04 Line Card status check PASSED
[Info] All LineCard status is ok
================================================================================
Check 4 Node Version
================================================================================
Test01 Check Current Version PASSED
[Info] Node current running version is : n9000-16.0(2h)
================================================================================
Check 5 System State
================================================================================
Test01 Check System State FAILED
[Warn] Top System State is : out-of-service
[Info] Node upgrade is in notscheduled state
================================================================================
Check 6 Updated LLDP Adjacencies
================================================================================
Port: eth1/47
Test02 Wiring Issues Check PASSED
[Info] No Wiring Issues detected
Test03 Port Types Check PASSED
[Info] No issues with port type, type is:leaf
Test04 Port Mode Check PASSED
[Info] No issues with port mode, type is:trunk
Test02 Adjacency Check PASSED
[Info] Adjacency detected with APIC
================================================================================
Check 7 BootStrap Status
================================================================================
Test01 Check Bootstrap/L3Out config download FAILED
[Warn] BootStrap/L3OutConfig URL not found
[Info] Ignore this if this node is not an IPN attached device
================================================================================
Check 8 Infra VLAN Check
================================================================================
Test01 Check if infra VLAN is received PASSED
[Info] Infra VLAN received is : 3914
Test02 Check if infra VLAN is deployed PASSED
[Info] Infra VLAN deployed successfully
================================================================================
Check 9 DHCP Status
================================================================================
Test01 Check Node Id FAILED
[Error] Valid Node Id not received via DHCP response
Test02 Check Node Name FAILED
[Error] Valid Node name not revevied via DHCP
Test03 Check TEP IP FAILED
[Error] Valid TEP IP not revevied via DHCP
Test04 Check Configured Node Role FAILED
[Error] Valid Node Role not received via DHCP response
Test05 DHCP Msg Stats FAILED
[Info] Total DHCP discover sent by switch : 18

[Error] Cannot retrive DHCP offer stats
[Error] Cannot retrive DHCP request stats
[Error] Cannot retrive DHCP ACK stats
[Fatal-Error] Please check DHCP issues...Aborting command execution

The leaf dosen't send a DHCP Discover. 

(none)# tcpdump -ni kpm_inb port 67 or 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on kpm_inb, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

I did a factory reset for apic as written here https://community.cisco.com/t5/application-centric-infrastructure/factory-reset-apic-and-nodes/m-p/3408496/highlight/true#M4879 . But it didn't help. 

What else can I check to diagnose the dhcp problem?

[RedNectar]

Hi @config ,

Definitely time to call TAC.  But note a couple of things from your output.

Check 6 Updated LLDP Adjacencies
================================================================================
Port: eth1/47
Test02 Wiring Issues Check PASSED
[Info] No Wiring Issues detected
Test03 Port Types Check PASSED
[Info] No issues with port type, type is:leaf
Test04 Port Mode Check PASSED
[Info] No issues with port mode, type is:trunk
Test02 Adjacency Check PASSED
[Info] Adjacency detected with APIC

so it seems LLDP is working.  And if LLDP is working you should see the Leaf appear under Fabric > Inventory > Fabric Membership |> Nodes Pending Registration waiting for you to give it an ID - which it needs before it can get passed the DHCP bit.

[Robert Burns]

Have you checked the date/time on the Switch vs. that on APIC?  
Get the output of "date" from both devices and post it here.

Robert

[config]

Hi Robert Burns, 

I checked the date on switch vs apic. They have a difference of about 30 minutes. 

[config]

Hi team, @RedNectar , @Robert Burns 

I have checked the lldp adjacency. It is installed. Infra-VLAN 3914 was also received. But I still don't see the leaf on the apic controller.

(none)# show discoveryissues
================================================================================
Check 1 Platform Type
================================================================================
Test01 Retrieving Node Role PASSED
[Info] Current node role: LEAF
[Info] Please check CH09 DHCP status section for configured node role
================================================================================
Check 2 FPGA/BIOS in sync test
================================================================================
Test01 FPGA version check PASSED
[Info] No issues found for FPGA versions
Test02 BIOS version check PASSED
[Info] No issues found for BIOS versions
================================================================================
Check 3 HW Modules Check
================================================================================
Test01 Fans status check PASSED
[Info] All fans status is ok
Test02 Power Supply status check FAILED
[Warn] Operational state of sys/ch/psuslot-1/psu is: shut
[Info] Ignore this if it is a redundant power supply
Test03 Fan Tray status check PASSED
[Info] All FanTrays status is ok
Test04 Line Card status check PASSED
[Info] All LineCard status is ok
================================================================================
Check 4 Node Version
================================================================================
Test01 Check Current Version PASSED
[Info] Node current running version is : n9000-16.0(2h)
================================================================================
Check 5 System State
================================================================================
Test01 Check System State FAILED
[Warn] Top System State is : out-of-service
[Info] Node upgrade is in notscheduled state
================================================================================
Check 6 Updated LLDP Adjacencies
================================================================================
Port: eth1/47
Test02 Wiring Issues Check PASSED
[Info] No Wiring Issues detected
Test03 Port Types Check PASSED
[Info] No issues with port type, type is:leaf
Test04 Port Mode Check PASSED
[Info] No issues with port mode, type is:trunk
Test02 Adjacency Check PASSED
[Info] Adjacency detected with APIC
================================================================================
Check 7 BootStrap Status
================================================================================
Test01 Check Bootstrap/L3Out config download FAILED
[Warn] BootStrap/L3OutConfig URL not found
[Info] Ignore this if this node is not an IPN attached device
================================================================================
Check 8 Infra VLAN Check
================================================================================
Test01 Check if infra VLAN is received PASSED
[Info] Infra VLAN received is : 3914
Test02 Check if infra VLAN is deployed PASSED
[Info] Infra VLAN deployed successfully
================================================================================
Check 9 DHCP Status
================================================================================
Test01 Check Node Id FAILED
[Error] Valid Node Id not received via DHCP response
Test02 Check Node Name FAILED
[Error] Valid Node name not revevied via DHCP
Test03 Check TEP IP FAILED
[Error] Valid TEP IP not revevied via DHCP
Test04 Check Configured Node Role FAILED
[Error] Valid Node Role not received via DHCP response
Test05 DHCP Msg Stats FAILED
[Info] Total DHCP discover sent by switch : 16

[Error] Cannot retrive DHCP offer stats
[Error] Cannot retrive DHCP request stats
[Error] Cannot retrive DHCP ACK stats
[Fatal-Error] Please check DHCP issues...Aborting command execution

(none)# show lldp neighbors
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
apic1 Eth1/47 120 eth2-1
Total entries displayed: 1

(none)# show vlan encap-id 3914

VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
7 infra:default active Eth1/47

VLAN Type Vlan-mode
---- ----- ----------
7 enet CE

[Robert Burns]

What is the exact model of the Leaf you're using?  
From the APIC CLI paste the output of "moquery -c dhcpPool"

Robert

[config]

Hi @Robert Burns ,

I am using a n9k-c93180yc-fx switch model with image 16.0.(2h) and virtual APIC 6.0(2h) deployed on a Cisco UCS C220M5 SFF. The output of the "moquery -c dhcpPool" commands is as follows:

apic1# moquery -c dhcpPool
No Mos found
apic1#

Only the APIC is connected to the switch, nothing else.

[Robert Burns]

Ok.  We should have started with these details.  Virtual APIC involves a bunch of different considerations than physical APIC (which is what both RedNectar and myself likely assumed you were using.  Some additional questions:
1. Provide a screenshot of the APICs Port Groups connected on vMware (showing VLAN tag etc)
2. Provide a screenshot of the CDP / LLDP settings for the port group APIC is connected to on the vSwitch/vDS.
3. Provde a screenshot of the vAPIC VM settings showing CPU/Memory/Disk configuration

Robert 

[config]

Hi @Robert Burns , 

I've attached screenshots. 

1. I use vSwitch2 with VLAN 0 between vAPIC and leaf.

I have a CDP configuration on vmnic2.

It is hardware configuration of vAPIC

[config · Accepted solution]

Hi @Robert Burns 

I fixed it. I replaced VLAN 0 with VLAN 4095 and everything worked. If I understand correctly, CDP packet forwrding requires a trunk (VLAN 4095), and LLDP packet forwrding requires a bridge (VLAN 0). Am I right?

[Robert Burns]

Yes, the link between APIC & Leaf needs to allow multiple VLANs.  You either do this by creating a trunk port group, allowing VLAN 0 (untagged), the Infra VLAN, the Inband Mgmt VLAN (if used).  Alternatley, if you set the port Group to '4095' that allows all VLANs.  I would harden this to just the VLANs you want to allow (mentioned previously).  

The port group dot1q tagging is what I suspected with CDP/LLDP working (rides untagged with VLAN0), but DHCP (which uses the Infra VLAN) was not.   This is mentioned in the Install guide: 

• Configure VLANs.

  Configure the New Distributed Port Group for virtual APICs. Enable VLAN trunking and set the VLAN range. Add the following VLANs to the VLAN trunk range:

  • Infra VLAN

  • Inband Managment VLAN (if Inband management is configured)

  • VLAN 0 (VLAN 0 is required for LLDP forwarding LLDP packets. LLDP packets sent from the leaf switch will be untagged)

Regards,
Robert