Re: ACI Multipod Fabric Discovery

Ziad El Achkouty · ‎11-27-2018

Hello,

I have configured 2 IPNs and POD2 on ACI with tep pool 10.1.0.0/16

OSPF between spine and IPN is up

I can see the pod2 spine in the discovered fabric however, its status is stuck at discovering and no infa IP is assigned to it.

RedNectar · ‎11-27-2018

Hi Zaid,

Forgive me for stating the obvious, but when you say "I have configured 2 IPNs" - does that mean you have configured the DHCP relay configuration as well?

At this stage, that Pod2 spine should be sending DCHP Discover packets, so if you have DHCP relay configured, I'd start looking at debuggin that to be sure the DHCP packets can get back to the APICs in Pod1.

I hope this helps

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

Ziad El Achkouty · ‎11-27-2018

Yes, dhcp relay is configured on both IPNs and dhcp requests are beeing forwarded to 10.0.0.1 which is the infra IP of apic1. However, no reply is received. (Moreover, ping from ipn2 to 10.0.0.1 is working)

RedNectar · ‎11-27-2018

I guess the next thing I would try would be to debug those DHCP requests/replies on the IPN router interfaces.

RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

udo.konstantin · ‎11-28-2018

Can you check that the DHCP requests are reached the IPN device ? Moreover debugging packets on IPN device will show detailed what happens with the packet. I assume there is any misconfiguration regarding DHCP.

Ziad El Achkouty · ‎11-28-2018

Here is the output of the debug DHCP. source is changing to the interface IP of IPN2 and destination is the infra ip of 10.0.0.1

2018 Nov 28 07:53:49.341373 dhcp_snoop: (PKT)dhcp_nw_get_packet_info : vlan = 1
2018 Nov 28 07:53:49.341386 dhcp_snoop: (PKT)From netstack info logical 0x1a000004 phy 0x1a000000
2018 Nov 28 07:53:49.341395 dhcp_snoop: (PKT)ether_type:0x800
2018 Nov 28 07:53:49.341403 dhcp_snoop: (PKT)dhcp_mtype: 0x1 of_index: 0x1a000004 vlan: 1 pvlan: 0 relay_enabled: 1
2018 Nov 28 07:53:49.341455 dhcp_snoop: (T)Enter Fn dhcp_relay_ip_get_local_addr: IfIndex 0x1a000004
2018 Nov 28 07:53:49.341488 dhcp_snoop: (PKT)Packet recieved on L3 interface
2018 Nov 28 07:53:49.341503 dhcp_snoop: (PKT)Its a DHCP_RELAY packet
2018 Nov 28 07:53:49.341511 dhcp_snoop: (PKT)Got packet type as rx_ctx_p->type: 0x1
2018 Nov 28 07:53:49.341519 dhcp_snoop: (PKT)Packet dhcp_mtype=0x1
2018 Nov 28 07:53:49.341583 dhcp_snoop: (PKT)Initial if_index from netstack Ethernet1/1.4
2018 Nov 28 07:53:49.341603 dhcp_snoop: (PKT)Changed if_index as calculated from giaddr Ethernet1/1.4
2018 Nov 28 07:53:49.341619 dhcp_snoop: (EV)Executing [mcecm_api_is_pc_mcec]
2018 Nov 28 07:53:49.341631 dhcp_snoop: (EV)input: if_index = [0x1a000004]
2018 Nov 28 07:53:49.341642 dhcp_snoop: (EV)Executing [mcecm_sdb_is_pc_mcec]
2018 Nov 28 07:53:49.341656 dhcp_snoop: (EV)if_map == NULL: if_index = [0x1a000004]
2018 Nov 28 07:53:49.341668 dhcp_snoop: (EV)output:0
2018 Nov 28 07:53:49.341675 dhcp_snoop: (PKT)From netstack, case2 info logical 0x1a000004 phy 0x1a000000
2018 Nov 28 07:53:49.341693 dhcp_snoop: (PKT)RECIEVED packet type DHCP RELAY
2018 Nov 28 07:53:49.341708 dhcp_snoop: (PKT) DHCPDISCOVER on Intf Ethernet1/1.4(42), phy Ethernet1/1(5), vlan 1, vni 0, pvlan 0, vdc 0, vrf fabric-mpod
2018 Nov 28 07:53:49.341720 dhcp_snoop: (PKT)Packet: Len 378, l2 HDR 14, DHCP data 336, hdr flag 0x36
2018 Nov 28 07:53:49.341735 dhcp_snoop: (PKT)L2: DMAC ff ff ff ff ff ff , SMAC 78 0c f0 a2 7a ec , PROTO 0x8
2018 Nov 28 07:53:49.341750 dhcp_snoop: (PKT)IP: DIP 255.255.255.255, SIP 0.0.0.0, tl 364, protocol 17, ttl 255
2018 Nov 28 07:53:49.341762 dhcp_snoop: (PKT)UDP: sport 68, dport 67, len 344
2018 Nov 28 07:53:49.341778 dhcp_snoop: (PKT)DHCP HDR:op 1, htype 1, hlen 6, hops 0, xid 0x570f65c9, secs 0, flags 0x8000, ciaddr 0.0.0.0, yiaddr 0.0.0.0, siaddr 0.0.0.0, giaddr 0.0.0.0, chaddr 78 0c f0 a2 7b 0a , sname , file
2018 Nov 28 07:53:49.341836 dhcp_snoop: (PKT)DHCP Options: list 43,51,53,60-61 [ Code 53, len 1, val 01 Code 61, len 19, val 46 44 4f 32 32 31 38 32 51 43 57 2d 65 74 68 31 2f 34 38 Code 43, len 56, val 01 01 03 02 01 01 06 01 01 03 02 00 01 0a 04 00 00 00 01 0c 0a 65 74 68 31 2f 34 38 2e 34 38 0b 0b
2018 Nov 28 07:53:49.341848 dhcp_snoop: (PKT)Handle pkt: logical 0x1a000004 phy 0x1a000000 rcvd_sisf_intf:0x0 pkt_type:1
2018 Nov 28 07:53:49.341859 dhcp_snoop: (PKT)Made copy of pkt len 378
2018 Nov 28 07:53:49.341869 dhcp_snoop: (PKT)ether_type:0x800
2018 Nov 28 07:53:49.341877 dhcp_snoop: (PKT)Enter Fn dhcp_relay_pkt_handler
2018 Nov 28 07:53:49.341885 dhcp_snoop: (PKT)Enter Fn dhcp_relay_validate_pkt
2018 Nov 28 07:53:49.341903 dhcp_snoop: (EV)Executing [mcecm_api_is_pc_mct]
2018 Nov 28 07:53:49.341917 dhcp_snoop: (EV)input: if_index = [0x1a000000], use_cache = 0
2018 Nov 28 07:53:49.341930 dhcp_snoop: (EV)Executing [mcecm_sdb_is_pc_mct]
2018 Nov 28 07:53:49.341943 dhcp_snoop: (EV)if_map == NULL: if_index = [0x1a000000]
2018 Nov 28 07:53:49.341955 dhcp_snoop: (EV)output:0
2018 Nov 28 07:53:49.341964 dhcp_snoop: (PKT)Made copy of pkt len 378
2018 Nov 28 07:53:49.341974 dhcp_snoop: (PKT)ether_type:0x800
2018 Nov 28 07:53:49.341987 dhcp_snoop: (T)Enter Fn dhcp_relay_select_giaddr: Client Mac Address 78 0c f0 a2 7b 0a
2018 Nov 28 07:53:49.341998 dhcp_snoop: (T)Enter Fn dhcp_relay_send2server
2018 Nov 28 07:53:49.342056 dhcp_snoop: (PKT)source-intf is Ethernet1/1.4 vrf fabric-mpod
2018 Nov 28 07:53:49.342094 dhcp_snoop: im_get_extension_index(242): Reading iod 0x2a, from ext <267,0> (volatile:/dev/shm/im_sdb_extension_267_0)
2018 Nov 28 07:53:49.342111 dhcp_snoop: im_get_extension_index(258): sdb_read successful - returned pss_datum 0x10ca4e74
2018 Nov 28 07:53:49.342121 dhcp_snoop: im_get_extension_index(265): sdb_read succesful - returned pss_datum, size4, ptr 0x10cbd5bc (val 0x2c65956d)
2018 Nov 28 07:53:49.342133 dhcp_snoop: (PKT)Got src intf address 172.31.254.42
2018 Nov 28 07:53:49.342143 dhcp_snoop: (PKT)Client and Server are in the same VRF
2018 Nov 28 07:53:49.342150 dhcp_snoop: (T)Enter Fn dhcp_relay_ip_get_local_addr: IfIndex 0x1a000004
2018 Nov 28 07:53:49.342162 dhcp_snoop: (PKT)Relay Option 82 is disabled
2018 Nov 28 07:53:49.342171 dhcp_snoop: (PKT)RELAYING to server 10.0.0.1
2018 Nov 28 07:53:49.342188 dhcp_snoop: (PKT)IP: DIP 10.0.0.1, SIP 172.31.254.42, tl 27649, protocol 17, ttl 255
2018 Nov 28 07:53:49.342197 dhcp_snoop: (PKT)UDP: sport 68, dport 67, len 344
2018 Nov 28 07:53:49.342212 dhcp_snoop: (PKT)DHCP HDR:op 1, htype 1, hlen 6, hops 1, xid 0x570f65c9, secs 0, flags 0x8000, ciaddr 0.0.0.0, yiaddr 0.0.0.0, siaddr 0.0.0.0, giaddr 172.31.254.42, chaddr 78 0c f0 a2 7b 0a , sname , file
2018 Nov 28 07:53:49.342278 dhcp_snoop: (PKT)DHCP Options: list 43,51,53,60-61 [ Code 53, len 1, val 01 Code 61, len 19, val 46 44 4f 32 32 31 38 32 51 43 57 2d 65 74 68 31 2f 34 38 Code 43, len 56, val 01 01 03 02 01 01 06 01 01 03 02 00 01 0a 04 00 00 00 01 0c 0a 65 74 68 31 2f 34 38 2e 34 38 0b 0b
2018 Nov 28 07:53:49.342313 dhcp_snoop: (PKT)Made copy of pkt len 378
2018 Nov 28 07:53:49.342324 dhcp_snoop: (PKT)ether_type:0x800
2018 Nov 28 07:53:51.241218 dhcp_snoop: (T)Enter Fn dhcp_handle_rate_counter_timeout:
2018 Nov 28 07:53:51.241266 dhcp_snoop: (T)calculating rx rate 5=(82-57)/299
2018 Nov 28 07:53:51.241282 dhcp_snoop: (T)calculating tx rate 5=(82-57)/299
2018 Nov 28 07:53:51.241294 dhcp_snoop: (T)calculating drop rate 0=(0-0)/299
2018 Nov 28 07:53:51.241314 dhcp_snoop: (T)calculating rx rate 0=(0-0)/299
2018 Nov 28 07:53:51.241327 dhcp_snoop: (T)calculating tx rate 0=(0-0)/299
2018 Nov 28 07:53:51.241340 dhcp_snoop: (T)calculating drop rate 0=(0-0)/299
2018 Nov 28 07:53:51.241363 dhcp_snoop: (T)Received Msg with opcode 6483 and id 276445
2018 Nov 28 07:53:51.241460 dhcp_snoop: (T)PPF process message status 1
2018 Nov 28 07:53:51.241471 dhcp_snoop: (T)Message was processed by library
2018 Nov 28 07:53:51.406673 dhcp_snoop: (T)dhcp_snoop_process_timeout: Heart beat timer is expired

RedNectar · ‎11-28-2018

So does your APIC have a route back to 172.31.254.42?

2018 Nov 28 07:53:49.342212 dhcp_snoop: (PKT)DHCP HDR:op 1, htype 1, hlen 6, hops 1, xid 0x570f65c9, secs 0, flags 0x8000, ciaddr 0.0.0.0, yiaddr 0.0.0.0, siaddr 0.0.0.0, giaddr 172.31.254.42, chaddr 78 0c f0 a2 7b 0a , sname , file

A simple network diagram showing 172.31.254.42 might help too. Presumably

APIC [10.0.0.1] -- Leaf --- [10.0.x.x]Spine[172.31.x.x]---[172.31.x.x]IPNRtr1[172.31.254.x]---[172.31.254.42]IPNRtr2[172.31.x.x]---[RemoteSpine]

RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

Ziad El Achkouty · ‎11-28-2018

ping 10.0.0.1 source 172.31.254.42 is working, and 172.31.254.40/30 is adverised through ospf to ACI.

Below is the network diagram:

Pod1-Spine1 (172.31.254.1/30) --- (172.31.254.2/30) IPN1 (172.31.254.10/30) --- (172.31.254.9/30) IPN2 (172.31.254.42/30) --- (172.31.254.41/30) Pod2-Spine1

More output:

apic1# moquery -c dhcpExtIf

Total Objects shown: 2

# dhcp.ExtIf

ifId : eth1/48

childAction :

dn : client-[SN]/if-[eth1/48]

ip : 172.31.254.1/30

lcOwn : local

modTs : 2018-11-28T08:04:34.956+00:00

name :

nameAlias :

relayIp : 0.0.0.0

rn : if-[eth1/48]

status :

subIfId : unspecified

# dhcp.ExtIf

ifId : eth1/48

childAction :

dn : client-[SN]/if-[eth1/48]

ip : 0.0.0.0

lcOwn : local

modTs : 2018-11-28T11:44:21.446+00:00

name :

nameAlias :

relayIp : 172.31.254.42

rn : if-[eth1/48]

status :

subIfId : eth1/48.48

apic1# grep ISC /var/log/dme/log/dhcpd.bin.log | egrep "SN"

11:46:57.486+00:00||dhcp||INFO||||ISC dhcpd: svc_write_client: client uid = SN-eth1/48 ||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||45 bico 47.534

10287||18-11-28

11:46:57.486+00:00||dhcp||INFO||||ISC dhcpd: svc_write_client: serial number = SN ||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||45

10287||18-11-28

11:47:01.539+00:00||dhcp||INFO||||ISC dhcpd: Received host decl = SN { uid "SN"; dynamic; option host-name "Pod2-Spine1"; fixed-address 10.1.136.64; option cisco.node-role 3; option cisco.spine-level 1; pool subclass "pod2" 00:2; option cisco.fabric-id 1; option cisco.pod-id 2; option cisco.node-id 201; filename "fwrepo/boot/node-SN"; } ||../svc/dhcpd/src/gen/ifc/beh/imp/./DhcpdSvc.cc||45 bico 57.534

RedNectar · ‎11-28-2018

OK. We've established that

the DHCP request is getting to the APIC
The APIC is allocating an address and is ready to send it to 172.31.254.42

Back to my last question. Does the APIC have a route to 172.31.254.42? (netstat -rn on the APIC)

RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

Ziad El Achkouty · ‎11-28-2018

There is no route on the APIC to this subnet.

However, there was no configuration required in the cisco documentation.

How can i advertise this subnet to the APIC? I have already included it in the fabric external routing profile.

Nuno Marques · ‎11-28-2018

Hi,

can you check if you have a working and connected leaf in pod2?

The Spine will be stuck in discovery if no leaf is attached in pod 2.

Regards.

Nuno Marques

Ziad El Achkouty · ‎11-28-2018

yes, i have a leaf that is connected to the spine in pod2

SupportAC · ‎07-04-2019

I have the same problem, do you solved?

udo.konstantin · ‎07-04-2019

Yes, Problem was solved.
APIC instance in the cluster first discovers only the leaf switch to which it is directly connected. This means you must register the first leaf Switch. After the leaf switch is registered with the APIC, the APIC discovers all spine switches that are directly connected to the leaf switch.
Then you can register all the other switches with:
Fabric > Inventory
—> Fabric Membership

Don‘t forget to „update“ after you assign the ID to the Switch.

Hope this helps.

BR
Udo

e-chuah · ‎12-02-2020

Hi Ziad,

I have very similar symptoms as yours. THe APIC receives DHCP broadcast from from Pod2 spine, but some how it is not sending out the DHCP offer packet.

Did u managed to solve this issue ?

Thanks

Eng Wee