Solved: How to add and configure POD2 switches multi-pod ACI

irenof · ‎10-01-2024

Hi all,

I am bringing up Multi-Pod ACI and I have problems with the "multi" part of the operation.

The final topology will have 3 pods with a 3-APIC cluster (one on each pod). (APIC 6.x release)

I am able to connect to the APIC 1 gui, I set up cluster size 3 anche added che CIMC of the other controllers. I can login into apic controller 2 and 3, but after I insert the password they are stuck on cluster bring up looping circle.

In Apic 1 I discovered all of the switches, I configured the IPN connection and I am able to see spines of POD 2 in the "pending registration" tab.

But now, what then? I have tried 2 ways, but nothing happens: the spine in pod 2 remains in "discovery" status forever.

These the ways I already tried:

1) registered the 2 spines of POD 2 in POD 1 (seed pod), since no other pods were present (with the idea to move them to pod 2 later, somehow). The spines remained in discovery mode. (so decommissioned it)

2) Started the Add Pod wizard, while I was discovering spines of PODs 2 (configured in pod 1) and selected them while creating the new pod (with its own TEP pools), then I decommissioned the two spines and discovered them again, but selecting pod 2 (now available), but nothing: still Discovering status.

So my question is? Ho to create a POD 2 and discovery and register the the correct switches?

Is there something wrong with the spines? (It is important to say that the swiches are recognized as leaf by the APIC).

UPDATE:
this is the discoveryissue of a spine in pod 2:

(none)# show discoveryissues

================================================================================

Check 1 Platform Type

================================================================================

Test01 Retrieving Node Role PASSED

[Info] Current node role: SPINE

[Info] Please check CH09 DHCP status section for configured node role

================================================================================

Check 2 FPGA/BIOS in sync test

================================================================================

Test01 FPGA version check PASSED

[Info] No issues found for FPGA versions

Test02 BIOS version check PASSED

[Info] No issues found for BIOS versions

================================================================================

Check 3 HW Modules Check

================================================================================

Test01 Fans status check PASSED

[Info] All fans status is ok

Test02 Power Supply status check PASSED

[Info] All PSUs status is ok

Test03 Fan Tray status check PASSED

[Info] All FanTrays status is ok

Test04 Line Card status check PASSED

[Info] All LineCard status is ok

================================================================================

Check 4 Node Version

================================================================================

Test01 Check Current Version PASSED

[Info] Node current running version is : n9000-15.2(8d)

================================================================================

Check 5 System State

================================================================================

Test01 Check System State FAILED

[Warn] Top System State is : out-of-service

[Info] Node upgrade is in notscheduled state

================================================================================

Check 6 Updated LLDP Adjacencies

================================================================================

Port: eth1/6

Test02 Wiring Issues Check PASSED

[Info] No Wiring Issues detected

Test03 Port Types Check PASSED

[Info] No issues with port type, type is:fab

Test04 Port Mode Check PASSED

[Info] No issues with port mode, type is:routed

Test02 Adjacency Check PASSED

[Info] Adjacency detected with leaf

Port: eth1/4

Test02 Wiring Issues Check PASSED

[Info] No Wiring Issues detected

Test03 Port Types Check PASSED

[Info] No issues with port type, type is:fab

Test04 Port Mode Check PASSED

[Info] No issues with port mode, type is:routed

Test02 Adjacency Check PASSED

[Info] Adjacency detected with leaf

Port: eth1/2

Test02 Wiring Issues Check PASSED

[Info] No Wiring Issues detected

Test03 Port Types Check PASSED

[Info] No issues with port type, type is:fab

Test04 Port Mode Check PASSED

[Info] No issues with port mode, type is:routed

Test02 Adjacency Check PASSED

[Info] Adjacency detected with leaf

Port: eth1/3

Test02 Wiring Issues Check PASSED

[Info] No Wiring Issues detected

Test03 Port Types Check PASSED

[Info] No issues with port type, type is:fab

Test04 Port Mode Check PASSED

[Info] No issues with port mode, type is:routed

Test02 Adjacency Check PASSED

[Info] Adjacency detected with leaf

Port: eth1/1

Test02 Wiring Issues Check PASSED

[Info] No Wiring Issues detected

Test03 Port Types Check PASSED

[Info] No issues with port type, type is:fab

Test04 Port Mode Check PASSED

[Info] No issues with port mode, type is:routed

Test02 Adjacency Check PASSED

[Info] Adjacency detected with leaf

Port: eth1/63

Test02 Wiring Issues Check PASSED

[Info] No Wiring Issues detected

Test03 Port Types Check PASSED

[Info] No issues with port type, type is:fab

Test04 Port Mode Check PASSED

[Info] No issues with port mode, type is:routed

Test02 Adjacency Check PASSED

[Info] Mpod spine to spine adjacency detected

Port: eth1/64

Test02 Wiring Issues Check PASSED

[Info] No Wiring Issues detected

Test03 Port Types Check PASSED

[Info] No issues with port type, type is:fab

Test04 Port Mode Check PASSED

[Info] No issues with port mode, type is:routed

Test02 Adjacency Check PASSED

[Info] Mpod spine to spine adjacency detected

Port: eth1/5

Test02 Wiring Issues Check PASSED

[Info] No Wiring Issues detected

Test03 Port Types Check PASSED

[Info] No issues with port type, type is:fab

Test04 Port Mode Check PASSED

[Info] No issues with port mode, type is:routed

Test02 Adjacency Check PASSED

[Info] Adjacency detected with leaf

================================================================================

Check 7 BootStrap Status

================================================================================

Test01 Check Bootstrap/L3Out config download FAILED

[Warn] BootStrap/L3OutConfig URL not found

[Info] Ignore this if this node is not an IPN attached device

================================================================================

Check 8 Infra VLAN Check

================================================================================

Test01 Check if infra VLAN is received PASSED

[Info] Infra VLAN received is : 3914

Test02 Check if infra VLAN is deployed SKIPPED

[Info] No ctrlr adjacency detected. Skipping infra VLAN deployed check

================================================================================

Check 9 DHCP Status

================================================================================

Test01 Check Node Id FAILED

[Error] Valid Node Id not received via DHCP response

Test02 Check Node Name FAILED

[Error] Valid Node name not revevied via DHCP

Test03 Check TEP IP FAILED

[Error] Valid TEP IP not revevied via DHCP

Test04 Check Configured Node Role FAILED

[Error] Valid Node Role not received via DHCP response

Test05 DHCP Msg Stats FAILED

[Info] Total DHCP discover sent by switch : 1703

[Error] Cannot retrive DHCP offer stats

[Error] Cannot retrive DHCP request stats

[Error] Cannot retrive DHCP ACK stats

[Fatal-Error] Please check DHCP issues...Aborting command execution

I think that I found the problem... need to investigate

Thanks

irenof · ‎10-02-2024

Hi @ThinMan,

thank you for your interesting and fulfilling answer. The problem was the default role of the switch, as you mentioned. Even selecting role "spine" during the registration for my spines switches detected as "leaf" did not work and the registration remained in the state of discovery.

I was told that someone had fixed the default role by changing a bit in a file, do not know more and after a

>acidiag touch clean
>acidiag touch setup
>acidiag reboot

I was able to finally register the spine correctly.

For what concern my question about che steps to add a pod, my error was that during the "add wizard" I only saw the spines of pod 1 (already registered) from the drop-down menu, thinking I had to choose from there. But I could also type the "future" spine IDs of the POD 2 that I was adding (It was not necessary to al ready have the spines of pod 2 registered, obviously..).

Thank you

View solution in original post

ThinMan · ‎10-01-2024

Hello,

Let me try to understand and summarize your issues.

Issue 1 - APICs not joining the cluster via the CIMC network

1. Does the management network of APIC 1 have connectivity to the CIMC networks of the other APICs? I assume this is through the ACI OOB network?

2. Are you able to access the other APICs' GUIs via their CIMC IPs? Can you ping them?

3. If so, are you able to log in to the CIMC using the default credentials?

4. I assume you've already checked the cluster compatibility requirements?

This doesn’t seem like a critical issue. You can still bring up the entire multi-pod fabric using the single functional APIC, which is your APIC Node 1 in Pod 1. So, I would suggest focusing on getting the multi-pod fabric up and running first. See below.

Issue 2 - Multi-pod discovery issues

In ACI, nodes and devices are discovered in a cascading manner. Cisco 9K switches in ACI mode operate with two types of interfaces:

1. Access Interfaces
2. Fabric Interfaces

I’m not sure what specific switches you're using, but for example, if you have C93600CD-GX switches, they come in Leaf mode by default, and only ports 29-36 can be used for the discovery process. Once discovered in APIC, you can configure them as SPINEs during registration. For now, can you double-check the following for your SPINE switches in POD2 and POD3?

1. What is their default mode? Are they set as LEAF by default?

2. Which are their default fabric ports? Typically, the last 8 ports are fabric ports.

3. Are your IPN devices connected to the fabric interfaces on the spines?

I assume you’ve completed the rest of the Multipod IPN configuration as per the recommended guidelines?

Also, keep in mind that once your SPINE switches are discovered and ready for registration, make sure that at least one leaf node is connected to these SPINEs via the correct fabric interfaces on both ends.

There are still many unknowns, but hopefully, this will help get you started.

irenof · ‎10-02-2024

Hi @ThinMan,

thank you for your interesting and fulfilling answer. The problem was the default role of the switch, as you mentioned. Even selecting role "spine" during the registration for my spines switches detected as "leaf" did not work and the registration remained in the state of discovery.

I was told that someone had fixed the default role by changing a bit in a file, do not know more and after a

>acidiag touch clean
>acidiag touch setup
>acidiag reboot

I was able to finally register the spine correctly.

For what concern my question about che steps to add a pod, my error was that during the "add wizard" I only saw the spines of pod 1 (already registered) from the drop-down menu, thinking I had to choose from there. But I could also type the "future" spine IDs of the POD 2 that I was adding (It was not necessary to al ready have the spines of pod 2 registered, obviously..).

Thank you