APIC : how to add routes on the APIC controllers ?

[Mathieu M]

Hello all,

I would like the APICs to talk to some VMs via in band. The VM have their own EPGs placed in a user Tenant, the corresponding BD/VRF/L3Out are in the common tenant. the network is 2.2.2.0/24.

I was able to make APIC inband management IP address (in the inb subnet 1.1.1.0/24), ping the F5 VM (shared VRF, export contract, subnet in EPGs, etc...) but only when I specify the inband interface. (the toogle preference between inband and ooband is set to ooband because this is were the rest of the world and tools are located) .

On the APICs, if I don't specify the inband interface in the ping, the ping to the VM 2.2.2.2 goes to the ooband... How I can a route on the APICs to reach the 2.2.2.0/24 subnet via inband ?

NB : on the leaf switches, the VRF mgmt:inb does have the routes to the VM subnet... but there are not present on the APICs.

Thank you in advance for any help,

Sincerely yours, Mathieu.

[joezersk]

Hello Mathieu:

APIC is set up to use the following forwarding logic when it comes to inband and oob traffic. 

1. Packets that come in an interface, go out that same interface

2. Packets sourced from the APIC, destined to a directly connected network, go out the directly connected interface

3. Packets sourced from the APIC, destined to a remote network, prefer In-band, followed by Out-of-band

To point #3, this is something you can change and in your description it appears you have.  Nothing wrong with preferring OOB, but it helps to know how APIC interprets that.  In your case, your pings have to specify the source interface of inband because your settings prefer OOB first to reach remote networks.

[Mathieu M]

Hello

Thank you. but point 3 is a bit frustrating.

The goal is to make HTTPs API calls to the VM (F5 management interface) in the Fabric from the APIC (for the F5 ACI Service Center App).

Behind the OOB, I have several subnets.

In the User tenants, I may have severel subnets as well.

If I switched to "prefered inband", the non-connected subnets in the oob will be unreachable, right ?

Would it be possible that the routing table of the inb VRF be pushed on the APICs ?

Sincerely yours, Mathieu.

[joezersk]

I feel your pain.  There is a way to get the best of both (i.e being able to reach inband and oob networks at the same time).  You would prefer inband in your APIC connection settings and then create an L3Out from tn-mgmt in vrf:inb to connect to some external device that has reachability to your OOB networks.  The issue here is the same as with any server/host.  You can only have ONE preferred default gateway (which I am sure you know already hence your logical desire to simply add routes to the host).  It's not that OOB would be unreachable without this L3 out, but it would only be able to reach IPs on the directly connected network of the OOB interface. 

For posterity, here is what my APIC looks like (from shell):

admin@apic-ams:~> route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.136.1 0.0.0.0 UG 8 0 0 bond0.11
default 10.50.129.254 0.0.0.0 UG 16 0 0 oobmgmt

bond0.11 is my inband interface and oobmgmt is self-explanatory.  Note the APIC does have both routes...but with different metrics reflecting your preference of OOB or Inband per APIC connection settings policy. There was a feature request to be able to add routes years ago but it never went anywhere.  It might be that we needed to root to set them, but we don't expose root to the customer.

[Mathieu M]

Thank you very much.

For security reason, we don't want to expose the management of the APICs through L3Out (only through the secured OOB Network) ...

We may have to allow trafic from oob interface APIC, to the VM interface, through our firewalls (and the rest of the DC infrastructure)

Do you know how this feature could be inserted in the backlog ? It could "just" to load on the APICs the content of the routing table of the inb VRF ;-)

Thank you for all your explanations.

Sincerely yours, Mathieu.

[Mathieu M · Accepted solution]

To add and complete find a somewhat solution, we managed to add an interface to the VM, that will be attached to an EPG link to the inb BD / subnet.

the inb BD / Subnet don't have to be routed outside of the Fabric.