Hi again @WarrenT-CO ,
OK, I guess I need to know what your l3Out config is - whether you are using the internal EPG method (method 1 in my bog) or the L3out method (method 3 in that blog).
My lab is using the L3out approach (because you need to set up that way for NDI and for multi-site), and the APIC is able to ping the leaves, (BUT NOT the spine - and that's always been the case)
However, if I was troubleshooting this, I'd start with this logic
- Check the routing tables on the APIC
- Check the routing tables the leaves
- Check the IP addresses of the leaves
- Check the ARP cache on the APIC
- Check the ARP cache on the APIC the leaves
- Check the MAC address tables on the leaves
In my case, my mgmt VLAN is vlan 203, which is canonical VLAN 10 on leaf 2201 and canonical VLAN 13 on leaf 2202 (see previous reply) and my mgmt subnet is 10.10.3.0/24. Here we go:
Check the routing tables on the APIC
apic1# bash
admin@apic1:~> netstat -rn | grep 10\.10\.3\.
0.0.0.0 10.10.3.1 0.0.0.0 UG 0 0 0 bond0.203
10.10.0.0 10.10.3.1 255.255.0.0 UG 0 0 0 bond0.203
10.10.1.31 10.10.3.1 255.255.255.255 UGH 0 0 0 bond0.203
10.10.3.0 0.0.0.0 255.255.255.0 U 0 0 0 bond0.203
10.10.3.1 0.0.0.0 255.255.255.255 UH 0 0 0 bond0.203
10.10.3.1 is the L3Out IP address, and the mgmt subnet (10.10.3.0/24) is local to the mgmt vlan (203)
Check the routing tables the leaves
apic1# fabric 2201-2202 show ip route 10.10.3.0/24 vrf mgmt:inb
----------------------------------------------------------------
Node 2201 (Leaf2201)
----------------------------------------------------------------
IP Route Table for VRF "mgmt:inb"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
10.10.3.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.2.24.64%overlay-1, [1/0], 1d12h, static
recursive next hop: 10.2.24.64/32%overlay-1
----------------------------------------------------------------
Node 2202 (Leaf2202)
----------------------------------------------------------------
IP Route Table for VRF "mgmt:inb"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
10.10.3.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.2.24.64%overlay-1, [1/0], 1d12h, static
recursive next hop: 10.2.24.64/32%overlay-1
As expected the route exists locally on each leaf, and the next-hop for anything not on the local subnet is the spine proxy
Check the IP addresses of the leaves
apic1# fabric 2201-2202 show ip interface brief vrf mgmt:inb
----------------------------------------------------------------
Node 2201 (Leaf2201)
----------------------------------------------------------------
IP Interface Status for VRF "mgmt:inb"(5)
Interface Address Interface Status
vlan9 10.10.3.5/24 protocol-up/link-up/admin-up
----------------------------------------------------------------
Node 2202 (Leaf2202)
----------------------------------------------------------------
IP Interface Status for VRF "mgmt:inb"(23)
Interface Address Interface Status
vlan12 10.10.3.1/24 protocol-up/link-up/admin-up
Hmmm. Leaf2202's inb address is actually 10.10.3.6 - 10.10.3.1 is the IP address assigned to the inb Bridge Domain - so I expected it to show up, but I also expected 10.10.3.6 to show up too.
Let's look closer:
apic1# fabric 2202 show ip interface vlan12
----------------------------------------------------------------
Node 2202 (Leaf2202)
----------------------------------------------------------------
IP Interface Status for VRF "mgmt:inb"
vlan12, Interface status: protocol-up/link-up/admin-up, iod: 73, mode: pervasive
IP address: 10.10.3.6, IP subnet: 10.10.3.0/24 secondary
IP address: 10.10.3.1, IP subnet: 10.10.3.0/24
IP broadcast address: 255.255.255.255
IP primary address route-preference: 0, tag: 0
Ah. That explains how the IP addresses are assigned! And gives me a thought about the output of show ip interface brief for leaf 2201 - I'm going to check vlan9 on leaf2201
apic1# fabric 2201 show ip interface vlan9
----------------------------------------------------------------
Node 2202 (Leaf2202)
----------------------------------------------------------------
IP Interface Status for VRF "mgmt:inb"
vlan9, Interface status: protocol-up/link-up/admin-up, iod: 70, mode: pervasive
IP address: 10.10.3.5, IP subnet: 10.10.3.0/24
IP address: 10.10.3.1, IP subnet: 10.10.3.0/24 secondary
IP broadcast address: 255.255.255.255
IP primary address route-preference: 0, tag: 0
Of course - the BD IP address is on BOTH switches, but curiously shows up as a secondary IP address on leaf2201 (FWIW I later experimented with making the 10.10.3.1/24 IP address Primary in the BD and output as shown immediately above changed to swap the primary and secondary addresses.
Check the ARP cache on the APIC
Start by loading the ARP cache with some pings
admin@apic1:~> ping 10.10.3.5 -c 2 #Leaf2201 inb IP address
PING 10.10.3.5 (10.10.3.5) 56(84) bytes of data.
64 bytes from 10.10.3.5: icmp_seq=1 ttl=64 time=0.199 ms
64 bytes from 10.10.3.5: icmp_seq=2 ttl=64 time=0.250 ms
--- 10.10.3.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 0.199/0.224/0.250/0.029 ms
admin@apic1:~> ping 10.10.3.6 -c 2 #Leaf2202 inb IP address
PING 10.10.3.6 (10.10.3.6) 56(84) bytes of data.
64 bytes from 10.10.3.6: icmp_seq=1 ttl=63 time=0.221 ms
64 bytes from 10.10.3.6: icmp_seq=2 ttl=63 time=0.267 ms
--- 10.10.3.6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.221/0.244/0.267/0.023 ms
admin@apic1:~> ping 10.10.3.1 -c 2 #Leaf2202 L#Out router IP address
PING 10.10.3.1 (10.10.3.1) 56(84) bytes of data.
64 bytes from 10.10.3.1: icmp_seq=1 ttl=64 time=0.210 ms
64 bytes from 10.10.3.1: icmp_seq=2 ttl=64 time=0.203 ms
--- 10.10.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1058ms
rtt min/avg/max/mdev = 0.203/0.206/0.210/0.014 ms
Curious - the TTL for the replies from 10.10.3.6 (Leaf2202 inb IP address) was 63, indicating the the ICMP reply had passed through a router! Not sure what's going on there, might have something to do with the secondary address thing discovered above. Let's look at the ARP table
apic1# bash
admin@apic1:~> arp -ani bond0.203
? (10.10.3.6) at 00:22:bd:f8:19:fe [ether] on bond0.203
? (10.10.3.1) at 00:22:bd:f8:19:fe [ether] on bond0.203
? (10.10.3.8) at <incomplete> on bond0.203
? (10.10.3.5) at 00:22:bd:f8:19:fe [ether] on bond0.203
Well. That's going to be a problem! Multiple IP addresses sharing the same MAC address - which may explain that TTL of 63 above! Of course we already know that 10.10.3.1 and 10.10.3.6 share an interface, so they'd be expected to have the same MAC.
But this could well be why your APIC is having trouble pinging!
Now the interesting thing is that the MAC address for 10.10.3.1 is defined in the inb Bridge Domain - and so I tried changing the MAC address for 10.10.3.1 to 00:22:bd:f8:19:fd and NOW the ARP cache looks like:
admin@apic1:~> arp -ani bond0.203
? (10.10.3.6) at 00:22:bd:f8:19:fd [ether] on bond0.203
? (10.10.3.1) at 00:22:bd:f8:19:fd [ether] on bond0.203
? (10.10.3.8) at <incomplete> on bond0.203
? (10.10.3.5) at 00:22:bd:f8:19:fd [ether] on bond0.203
ALL entries changed - the reason for this might become clearer in the next couple of steps
Check the ARP cache on the leaves
apic1# fabric 2201,2202 show ip arp vrf mgmt:inb
----------------------------------------------------------------
Node 2201 (Leaf2201)
----------------------------------------------------------------
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface
IP ARP Table for context mgmt:inb
Total number of entries: 1
Address Age MAC Address Interface
10.10.5.1 00:17:11 0022.bdf8.19ff vlan3
----------------------------------------------------------------
Node 2202 (Leaf2202)
----------------------------------------------------------------
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface
IP ARP Table for context mgmt:inb
Total number of entries: 0
Address Age MAC Address Interface
WELL. That was useless! The 10.10.5.1 IP address is the L3Out address! Nothing showing for the 10.10.3.0/24 subnet!
So that didn't help much. (Sorry)
Check the MAC address tables on the leaves
Before I do this, I don't think is going to help much either, because we already know that the MAC address is repeated for each leaf, but what the heck. Let's look anyway. I know the APIC is attached via eth1/1 and the spine via eth1/51, so I'll look only at those interfaces.
apic1# fabric 2201,2202 show mac address-table interface eth1/1
----------------------------------------------------------------
Node 2201 (Leaf2201)
----------------------------------------------------------------
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 10 84b8.02f5.efd1 dynamic - F F eth1/1
* 8 84b8.02f5.efd1 dynamic - F F eth1/1
----------------------------------------------------------------
Node 2202 (Leaf2202)
----------------------------------------------------------------
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
apic1# fabric 2201,2202 show mac address-table interface eth1/51
----------------------------------------------------------------
Node 2201 (Leaf2201)
----------------------------------------------------------------
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
----------------------------------------------------------------
Node 2202 (Leaf2202)
----------------------------------------------------------------
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
As expected - pretty useless!
Where to from here?
I guess the burning question is
Why do you want to be able to ping these IP addresses?
I suspect the answer is "To give me warm fuzzy feelings" - which I count as a legitimate reason.
But from the point of view of what the inband management is used for, you really only care that external devices can access the inband management, so if that works OK, then maybe you should be happy.
Given the fact that ACI uses the same MAC address across multiple interfaces in the same broadcast domain, I think I've finally got the picture. It's worth a separate reply, so I'll post this as it is to show my working, and start a new more succinct reply.
RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.
