cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
2243
Views
0
Helpful
7
Replies
m1xed0s
Contributor

Confusion on Nexus Dashboard...

Some background: I have a ACI Multi-Site (one brownfield with a greenfield DC) setup soon and the appliances purchased for the MSO is three Nexus Dashboard (ND)... As far as I know, there are three interfaces of each ND appliance need to be connected: CIMC, DATA and MANAGEMENT. Currently plan is to only run MSO on the ND cluster. Day-2 app might be added in the future.

 

Confusion #1: Which interface (DATA or Management) does the ND use to communicate with APIC OOB for the MSO? Cisco ND guide can not be more confusing on this...Like what does this statement even mean "For Multi-Site Orchestrator application, the data network can have in-band and/or out-of-band IP reachability depending on your deployment scenario."???? I do know MSO need to communicate with APIC OOB...

 

Confusion #2: The guide does state "The two interfaces can be in the same or different subnets." for ND...But will there be downsides if I put the ND DATA and MANAGEMENT on the same subnet? Like for future additional Day-2 app? 

 

Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions


@m1xed0s wrote:

if “Nexus Dashboard nodes to Cisco APIC nodes communication” includes MSO, then ND is using the DATA to communicate for MSO. But as far as I know, MSO needs to talk to APIC OOB...so is it still the same when running MSO on ND?


You're confusing the ND Data Network (Source) & APIC Management Interfaces  (Destination).  These provide mutually exclusive functions.  We never said that MSO > APIC had to use OOB.  It can use either.  The point to remember here is that MSO hosted on ND is different than MSO OVA.   For ND, you onboard sites to ND (rather than adding site to MSO as was the case with the OVA deployments).  After a site has been onboarded to ND, you can enable services (MSO, Nexus Insights etc) on those site.  Whichever APIC interfaces you used to onboard the ACI Site to ND, that's how those services will communicate with the APICs. 
What I want to stress is that in order to utilize Day2 Ops App (Nexus Insights or NAE), your only onboarding option for the site is via Inband due to the need to receive telemetry traffic from the Switches (explained above).  With ND there's no option to have MSO connect to APIC via oob and have Day 2 Apps connect to APIC via inb - it's one or the other across all services.
Simple summary:

Option1:  ND to host only MSO
APIC Onboarding Interface = OOB or INB
Option2: ND to host only Day 2 Apps
APIC Onboarding Interface = INB
Options3: ND to host Day 2 Apps & MSO
APIC Onboarding Interface = INB

Regardless which APIC interface was used to onboard the site, this communication will ALWAYS source from the ND's Data Network Interface.
Robert 

View solution in original post

7 REPLIES 7
Robert Burns
Cisco Employee

Confusion #1: Which interface (DATA or Management) does the ND use to communicate with APIC OOB for the MSO? Cisco ND guide can not be more confusing on this...Like what does this statement even mean "For Multi-Site Orchestrator application, the data network can have in-band and/or out-of-band IP reachability depending on your deployment scenario."???? I do know MSO need to communicate with APIC OOB...

All this means is that the ND Data Network (that MSO uses to reach APICs) just needs reachability to either the APIC's Inband or Out of Band IP.  This could be clearer, I'll get this updated. For the Day2 Apps (Insights/NAE) these have a hard requirement to reach the fabric device (APICs and Switches) using only the Inband IPs.   My advice would be this: If you plan on deploying MSO + Day2 Apps, on the same ND cluster, then plan to use only the Inband IPs of the fabrics.  ND_Data_Network_Interface >>> Inband_IPs_of_APICs_and_Switches.  If you were only ever deploying MSO on Nexus Dashboard (bit of a waste), then you could get away with ND_Data_Network_Interface >>> OOB_IPs_of_APICs.

 

Confusion #2: The guide does state "The two interfaces can be in the same or different subnets." for ND...But will there be downsides if I put the ND DATA and MANAGEMENT on the same subnet? Like for future additional Day-2 app? 

Just like anything else, the Management traffic of a system should be separated from the Data traffic.  Can ND interfaces (Data & Management) belong to the same subnet - sure.  Is it a good idea - No.  When you start using Day2 Apps (Nexus Insights, NAE etc), you're going to be pushing a considerable amount of telemetry flows to the ND.  This traffic will always utilize the Data Network interface of the ND.  Do you really want your Management traffic contending with this additional congestion, or would you prefer it to be separated/isolated from it?

Hope this helps.

Robert


@Robert Burns wrote:

Confusion #1: Which interface (DATA or Management) does the ND use to communicate with APIC OOB for the MSO? Cisco ND guide can not be more confusing on this...Like what does this statement even mean "For Multi-Site Orchestrator application, the data network can have in-band and/or out-of-band IP reachability depending on your deployment scenario."???? I do know MSO need to communicate with APIC OOB...

All this means is that the ND Data Network (that MSO uses to reach APICs) just needs reachability to either the APIC's Inband or Out of Band IP.  This could be clearer, I'll get this updated. For the Day2 Apps (Insights/NAE) these have a hard requirement to reach the fabric device (APICs and Switches) using only the Inband IPs.   My advice would be this: If you plan on deploying MSO + Day2 Apps, on the same ND cluster, then plan to use only the Inband IPs of the fabrics.  ND_Data_Network_Interface >>> Inband_IPs_of_APICs_and_Switches.  If you were only ever deploying MSO on Nexus Dashboard (bit of a waste), then you could get away with ND_Data_Network_Interface >>> OOB_IPs_of_APICs.


So the ND DATA interface is actually where MSO communicates with APIC? If so, that's cool. But I thought the MSO can only communicate with the OOB of the APIC controllers for multi-site...Is it because of the ND, MSO can use the In-Band of APIC now? OR I missed feature improvement here? Also regarding the comment of "bit of a waste" of only running MSO on ND, we were initially planning to just get 3 UCS C servers with vSphere but were told MSO will only be available on ND going forward...

 


Confusion #2: The guide does state "The two interfaces can be in the same or different subnets." for ND...But will there be downsides if I put the ND DATA and MANAGEMENT on the same subnet? Like for future additional Day-2 app? 

Just like anything else, the Management traffic of a system should be separated from the Data traffic.  Can ND interfaces (Data & Management) belong to the same subnet - sure.  Is it a good idea - No.  When you start using Day2 Apps (Nexus Insights, NAE etc), you're going to be pushing a considerable amount of telemetry flows to the ND.  This traffic will always utilize the Data Network interface of the ND.  Do you really want your Management traffic contending with this additional congestion, or would you prefer it to be separated/isolated from it?


All fair points, I guess if I had to deploy the cluster with DATA and Management on the same subnet per ND, would it be possible to re-configure IP address to different network without rebuilding the ND cluster? If so, how much effort/impact to re-assign IP address to different network for either the DATA or Management? and also which one would be easier to change?

Maybe this one helps:

  • Data Network is used for:

    • Nexus Dashboard node clustering

    • Application to application communication

    • Nexus Dashboard nodes to Cisco APIC nodes communication

      For example, the network traffic for Day-2 Operations applications such as NAE.

  • Management Network is used for:

    • Accessing the Nexus Dashboard GUI

    • Accessing the Nexus Dashboard CLI via SSH

    • DNS and NTP communication

    • Nexus Dashboard firmware upload

    • Cisco DC App Center (AppStore)

      If you want to use the Nexus Dashboard App Store to install applications, the https://dcappcenter.cisco.com page must be reachable via the Management Network

    • Intersight device connector

Ref: https://www.cisco.com/c/en/us/td/docs/dcn/nd/2x/deployment/cisco-nexus-dashboard-deployment-guide-2x/m-nd-deploy-overview.html

 

Stay safe,

Sergiu

Thanks, I did go through that section in the doc...which frankly gave me the 1st confusion in my post...

 

if “Nexus Dashboard nodes to Cisco APIC nodes communication” includes MSO, then ND is using the DATA to communicate for MSO. But as far as I know, MSO needs to talk to APIC OOB...so is it still the same when running MSO on ND?


@m1xed0s wrote:

if “Nexus Dashboard nodes to Cisco APIC nodes communication” includes MSO, then ND is using the DATA to communicate for MSO. But as far as I know, MSO needs to talk to APIC OOB...so is it still the same when running MSO on ND?


You're confusing the ND Data Network (Source) & APIC Management Interfaces  (Destination).  These provide mutually exclusive functions.  We never said that MSO > APIC had to use OOB.  It can use either.  The point to remember here is that MSO hosted on ND is different than MSO OVA.   For ND, you onboard sites to ND (rather than adding site to MSO as was the case with the OVA deployments).  After a site has been onboarded to ND, you can enable services (MSO, Nexus Insights etc) on those site.  Whichever APIC interfaces you used to onboard the ACI Site to ND, that's how those services will communicate with the APICs. 
What I want to stress is that in order to utilize Day2 Ops App (Nexus Insights or NAE), your only onboarding option for the site is via Inband due to the need to receive telemetry traffic from the Switches (explained above).  With ND there's no option to have MSO connect to APIC via oob and have Day 2 Apps connect to APIC via inb - it's one or the other across all services.
Simple summary:

Option1:  ND to host only MSO
APIC Onboarding Interface = OOB or INB
Option2: ND to host only Day 2 Apps
APIC Onboarding Interface = INB
Options3: ND to host Day 2 Apps & MSO
APIC Onboarding Interface = INB

Regardless which APIC interface was used to onboard the site, this communication will ALWAYS source from the ND's Data Network Interface.
Robert 

View solution in original post

From ACI Multi-site white paper here, there is the statement: “Cisco Multi-Site Orchestrator nodes must communicate with the Out-Of-Band (OOB) interface of the APIC nodes deployed in different sites.”.  Comparing to your statement: MSO can communicate with either OOB or INB of the APIC...

 

so which one is correct?

 

it is good to know regarding “What I want to stress is that in order to utilize Day2 Ops App (Nexus Insights or NAE), your only onboarding option for the site is via Inband due to the need to receive telemetry traffic from the Switches (explained above).  With ND there's no option to have MSO connect to APIC via oob and have Day 2 Apps connect to APIC via inb - it's one or the other across all services.”.

 

The information above is accurate.  The note in that whitepaper needs to be updated. It applied only when MSO was offered as the OVA form factor.  When hosting MSO on Nexus Dashboard for INB/OOB fabric reachability is supported.
Robert