How to connect vCenter to ACI Out of Band Management network

[eahmed007]

Dear All ,

Please can you tell me how can I connect Vcenter which is residing one of VM in UCS blade system .

I going to connect each and every components (spine, Leaf, APIC controller ) with OOB Management Switch and put different IP block a part from TEP address .

However , I am little bit confuse to connect Vcenter as it does not have management port separately .

So please assist me how can I connect ACI with Vcenter which is VM in UCS blade  using Fabric Interconnect for management purpose only .

What would be the initial appose to connect Vcenter using OOB management using Fabric interconnect with static tagging.

I am looking forward to your valuable comments .

With regards

Erfan

[eahmed007]

Dear All ,

Please reply as I am waiting for your valuable comments .

With regards

Erfan

[RedNectar · Accepted solution]

[Update: Please scroll down to see my two more detailed responses to this question including diagrams and mini-tutorial. RedNectar] 

Hi Erfan,

Please can you suggest me where I need to create Application Profile and EPG .Whether I need to create in Mgmt Tenant or common tenant or user define tenant .As per your mail , you said that in some Tenant .

You could use any Tenant - I would use the mgmt Tenant - BUT you could use the common tenant, or even a user defined tenant.

Do I need to create Interface policy group separately for FI-A , FI-B and MGMT-Switch. and associate Interface selector with Interface policy group under Interface Profile .

Yes, every VPC needs its own Interface Policy Group.

Please assist me form where I choose MAC-Pinning-PC option .

You would have to create a Port Channel Interface Policy that specified MAC Pinning.  Have you read my tutorials?  The process is explained in tutorial 4 on the Access Policy Chain

While creating Leaf Profile , do i need to associate interface profile with leaf selector 101 and 102 .But you have choose 101 & 101 leaf selector .Please advice on this .

Oooooops. Yes - it should be 101 and 102.  I'll fix it sometime :)

I hope this helps

RedNectar
aka Chris Welsh

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

[eahmed007]

Hi Welsh ,

Thanks for your mail .I will configure according to your suggestion and let you know the update .very soon.

I am looking forward to your assistance in coming days .

With regards

Erfan

[eahmed007]

Hi Welsh ,

Hope that you are doing fine .

I have some queries on connectivity from Leaf 101 and Leaf 102 with Mgmt Switch .

Please advise whether I need to connect Leaf 101 and Leaf 102 using M-port with Mgmt Switch or any other port of Leaf 101 and Leaf 102 .

In my scenario , I have connected leaf 101 and Leaf 102 with Mgmt Switch for OOB management purpose .Should I create static Path in ACI using M-Port of leaf 101 and 102  or using other port available port .

Should I attach any other port from both Leaf 101 and Leaf 102 to Mgmt Switch as per your design to configure static path .

I am eagerly waiting for your valuable comments.

With regards

Erfan

[RedNectar]

[Update: This answer was greatly expaned later, so continue to scroll down to see the mini-tutorial I wrote. RedNectar]

Hi Erfan,

I have added the following to my original answer, and updated the pictures with the double reference to leaf101:

There is perhaps one more remaining issue.  The OOB Mgmt interfaces of the Leaf and Spine switches. How should they be connected?

The answer to that is that you you don't need to connect the OOB Mgmt interfaces of the ACI switches to the Mgmt Switch, but you certainly can if you wish.  The reason you don't need to, is because you can get to the management interface of these devices via the APIC.  And the reason that it is best practice and my recommendation to indeed connect these interfaces to the Mgmt switch is that should you ever loose all connectivity with the APICs, you may wish to access these switches via OOB, such as shown here:

Now once the cabling is done, you still have one more task. You will need to configure the OOB interfaces of the Leaf (and Spine) switches through the mgmt Tenant.

Tenants > Tenant mgmt > Node Management Addresses > Static Node Management Addresses >+ Create Static Node Management Addresses

Node Range: 101 – 102
Config: Out-Of-Band Addresses
Out-Of-Band Mangement EPG: default
Out-Of-Band IPV4 Address: start.ip.pool.address/mask
Out-Of-Band IPV4 Gateway: ip.pool.default.gateway

Note that the IP address that you give will be treated as the first address of a pool of addresses.   The first device in the range will be given the first available IP address, the second device the second address and so on.  If for instance you had set the range to 101-255, and the IP Address to 192.168.1.101/24 and you had say 8 leaf switches with IDs 101-106, and four two spines switches with IDs of 201 and 202, they leaf switches would get IPs of 192.168.1.101/24 to 192.168.1.106, then IP address 192.168.1.107 would be allocated to device ID 107 etc until device ID 201 was found, in which case it would get an IP address of 192.168.1.201, and the device with ID 201 would get 192.168.1.202.

I hope this helps

RedNectar
aka Chris Welsh

---

[eahmed007]

Hi Welsh ,

Thanks for your nice explanation and sharing the step by step process .I will configure the connectivity and let you know the update .

Its been great to get help form you and looking forward to your assistance in coming days.

With regards

Erfan

[eahmed007]

Hi Welsh ,Hope that you are doing fine .

According to my scenario , I am facing one more challenge which is ; I am connecting FI-A and FI-B with Leaf 103(N9K-C9396PX) and Leaf 104(N9K-C9396PX) using 10G connectivity and Management Switch would be connected with 1G copper with Leaf 101(N9K-C93128TX) and Leaf 102(N9K-C93128TX) .Considering the above connectivity,  can you provide me the access policy chain to configure .

Your valuable comments is highly appreciated  .

With regards

Erfan

[eahmed007]

Hi Welsh ,

According to my scenario , I am using Four Leaf Switch for this connectivity . Two Leaf 103(N9K-C9396PX) and Leaf 104(N9K-C9396PX) are connecting with FI-A and FI-B and this switch doesn't have 1G connectivity for Mgmt Switch .So I am using Two .101(N9K-C93128TX) and Leaf 102(N9K-C93128TX for MGmt Switch connectivity .

I am need you suggestion on this case and what would be the access policy chain considering the above topology .

I am looking forward to your reply soon.

With regards

Erfan

[RedNectar · Accepted solution]

Hi Erfan,

[Firstly my apologies for the slow reply, I've had this half written for a week, but work got in the way]

This is an interesting question, and one requires a good knowledge of UCS and VMWare. I'm going to try and avoid writing a huge VMWare/UCS tutorial, so I'll stick to concepts.
But first, lets assume that a VLAN exists which represents your management network. I'll call it the Mgmt VLAN. I'm going to also assume that there is a management switch somewhere where the APIC OOB interfaces and the UCSM Mgmt interfaces connect. And finally, I'll assume that the management switch is connected upstream of the UCS Fabric Interconnects. Assuming the FIs are connected to the Leaf Switches, then it makes sense to assume the management switch has a connection to a Leaf Switch too.

A picture always helps.

Now, lets deal with the ESXi hosts in the UCS Chassis.

The ESXi hosts will have at least two Physical NICs. In UCS Manager you will create a service profile for this blade, probably using a Service Profile Template, and that template will probably use a VNIC template.

Make sure you include the Mgmt VLAN as one of the VLANs in the VNIC Template.

Using the KVM facility in UCS Manager, you'll install ESXi then the vCenter Appliance on your blade server - and in the process set up one of the NICs (say vmnic0) as an uplink port for virtual switch vSwitch0 and assign a management port where you'll apply the IP address that you want to be able to communicate with both the APIC and the vCenter Appliance once it is installed - presumably this is already done.

And I'll also assume that the vCenter Appliance (VCA) is installed on one of these ESXi hosts with a connection to vSwitch0.  Time to update the picture.

As you can see, there are VPCs configured between Leaf101/102 and the FIs, and another to the Mgmt switch.  I'll make the assumption thatan Access Policy Chain has been configured as far as an AEP for the VPCs - something like the picture below.

In ACI you'll need to create a new VLAN Pool, or add the VLAN ID of the Mgmt-VLAN to an existing VLAN Pool.  Let's say it's VLAN 1000

Fabric > Access Policies > Pools > VLAN >+ Create VLAN Pool

Name: oob-VLAN.Pool
Allocation Mode: Static Allocation
(+) Encap Blocks:
Range: VLAN 1000 – VLAN 1000

Then of course you'll need a Physical Domain to link the VLAN Pool with - unless you are using an existing domain.

Fabric > Access Policies > Physical and External Domains > Physical Domains >+ Create Physical Domain

Name: oob-PhysDom
Associated Attachable Entity Profile: MyDC-AEP
VLAN Pool: oob-VLAN.Pool

And now the Access Policy Chain looks like

In ACI, you will create a new EPG, possibly in the mgmt Tenant.  Mind you, my advice would be to use inband Management on the APIC (using one of the three methods I've described elsewhere) and then that special EPG, but you have specified out-of-band, so I'll continue and assume you have created an Application Profile and an EPG called OOB.Mgmt-EPG in some Tenant.

All you need to do now is link map the VPC interfaces to the FIs and the Mgmt Switch to the OOB.Mgmt-EPG - assuming your EPG is in the mgmt Tenant in an Application Profile called OOB-AP:

Tenants > Tenant mgmt > Application Profiles > OOB-AP > OOB.Mgmt-EPG > Domains >+ Add Physical Domain Association

Physical Domain Profile: oob-PhysDom

Then still working with the OOB.Mgmt-EPG

... > OOB.Mgmt-EPG > Static Ports >+ Deploy Static EPG on PC, VPC, or Interface

Path Type: Virtual Port Channel
Path: FI.A-VPCIPG
Port Encap (...): VLAN 1000

Repeat for the VPC to FI-B

... > OOB.Mgmt-EPG > Static Ports >+ Deploy Static EPG on PC, VPC, or Interface

Path Type: Virtual Port Channel
Path: FI.B-VPCIPG
Port Encap (...): VLAN 1000

And of course the VPC to the Mgmt Switch

... > OOB.Mgmt-EPG > Static Ports >+ Deploy Static EPG on PC, VPC, or Interface

Path Type: Virtual Port Channel
Path: MgmtSw-VPCIPG
Port Encap (...): VLAN 1000

And now your management station should be able to access vCenter as well as the APIC and UCSM.

Note: When you configure a Virtual Machine Management Domain in the APIC to integrate with the vCenter, you'll need the VLANs that the VM will use in a VLAN Pool that is linked to your vCenter VMM, and you'll need to link that VMM to the same AEP that already is connected to the VPCs to FI-A and FI-B.  Something like the picture below:

There is perhaps one more remaining issue.  The OOB Mgmt interfaces of the Leaf and Spine switches. How should they be connected?

The answer to that is that you you don't need to connect the OOB Mgmt interfaces of the ACI switches to the Mgmt Switch, but you certainly can if you wish.  The reason you don't need to, is because you can get to the management interface of these devices via the APIC.  And the reason that it is best practice and my recommendation to indeed connect these interfaces to the Mgmt switch is that should you ever loose all connectivity with the APICs, you may wish to access these switches via OOB, such as shown here:

Now once the cabling is done, you still have one more task. You will need to configure the OOB interfaces of the Leaf (and Spine) switches through the mgmt Tenant.

Tenants > Tenant mgmt > Node Management Addresses > Static Node Management Addresses >+ Create Static Node Management Addresses

Node Range: 101 – 102
Config: Out-Of-Band Addresses
Out-Of-Band Mangement EPG: default
Out-Of-Band IPV4 Address: start.ip.pool.address/mask
Out-Of-Band IPV4 Gateway: ip.pool.default.gateway

Note that the IP address that you give will be treated as the first address of a pool of addresses.   The first device in the range will be given the first available IP address, the second device the second address and so on.  If for instance you had set the range to 101-255, and the IP Address to 192.168.1.101/24 and you had say 8 leaf switches with IDs 101-106, and four two spines switches with IDs of 201 and 202, they leaf switches would get IPs of 192.168.1.101/24 to 192.168.1.106, then IP address 192.168.1.107 would be allocated to device ID 107 etc until device ID 201 was found, in which case it would get an IP address of 192.168.1.201, and the device with ID 201 would get 192.168.1.202.

I hope this helps: Note - I have not been able to physically test this, so hope it works for you.  Let me know.

RedNectar
aka Chris Welsh

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

[eahmed007]

Hi Welsh ,

Thanks for your assistance and It's been great to have your nice explanation on my queries  and looking forward to your assistance in coming days as I am deploying ACI fabric .

Please can you suggest me where I need to create Application Profile and EPG .Whether I need to create in Mgmt Tenant or common tenant or user define tenant .As per your mail , you said that in some Tenant .

Do I need to create Interface policy group separately for FI-A , FI-B and MGMT-Switch. and associate Interface selector with Interface policy group under Interface Profile .Please assist me form where I choose MAC-Pinning-PC option .

While creating Leaf Profile , do i need to associate interface profile with leaf selector 101 and 102 .But you have choose 101 & 101 leaf selector .Please advice on this .

I am eagerly waiting for your valuable comments .

With regards

Erfan

[eahmed007]

Hi Welsh ,

I have configured successfully and its working fine as per your guidence .

With regards

Erfan

[RedNectar]

That's great news, and thanks for the feedback.

CW

[JatinderSharma0761]

Hi RedNectar,

I am assuming you need not to associate any layer3 bridge domain to oob-mgmt-EPG? So oob-mgmt-EPG will have bridge domain with no IP and unicast disabled?

Thanks

[RedNectar]

Hi @JatinderSharma0761 

This post is a mess! Too many branches. Too many threads.

But oob management on ACI needs an external gateway - there is NO OOB Bridge Domain in ACI, and the special oob-mgmt EPG (uni/tn-mgmt/mgmtp-default/oob-default) has no IP address, nor can you assign one.  

So there is no concept of unicast disabled either (perhaps you meant unicast routing?) - in either case, since there is no IP address, there is no unicast routing either.

The OOB management IP on the APIC essentailly lives outside ACI - on the linux box that runs the APIC application. And the APIC is NOT a router, unlike teh leaves where ACI BDs live.  To check the routing table of the APIC, you need to run the linux netstat -r command e.g.

apic1# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.16.11.1     0.0.0.0         UG        0 0          0 oobmgmt
0.0.0.0         10.119.1.1      0.0.0.0         UG        0 0          0 bond0.1911
10.1.0.0        10.1.0.30       255.255.0.0     UG        0 0          0 bond0.3961
10.1.0.30       0.0.0.0         255.255.255.255 UH        0 0          0 bond0.3961
10.119.1.0      0.0.0.0         255.255.255.0   U         0 0          0 bond0.1911
10.119.1.1      0.0.0.0         255.255.255.255 UH        0 0          0 bond0.1911
169.254.1.0     0.0.0.0         255.255.255.0   U         0 0          0 teplo-1
169.254.254.0   0.0.0.0         255.255.255.0   U         0 0          0 lxcbr0
172.16.11.0     0.0.0.0         255.255.255.0   U         0 0          0 oobmgmt
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0

HTH