Add second APIC controller to existing APIC cluster

tinhnho123 · ‎12-28-2015

Hello,

I currently have one APIC controller which online and running with 2 leafs and 2 spines. I'd like to add 2nd and 3rd APIC controller to the cluster but have difficulty to find the way to add it. Does anyone have any ideas how to add it?

The 2nd APIC controller is powered up and connected to 2 Leafs.

Thanks.

Tomas de Leon · ‎12-28-2015

I am making the assumption that when you created the cluster during the initial setup, you selected a Cluster size of "1". or "3"?

If the cluster size is "1", you will need to increase the cluster size to 3. Also, when running the setup for APIC 2 & 3, you must use the same parameters as APIC1 in regards to Fabric Name and Infra IP addressing scheme.

ssh to APIC 1
run the following commands:

moquery -c infraClusterPol

acidiag avread

Paste the results here
Also, verify the that APIC 2 connections have link on the leaf nodes in question.

T.

Tomas de Leon · ‎12-28-2015

YanL · ‎05-25-2017

One thing I found out the hard way is the firmware of the APIC you are trying to add needs to match the cluster you are trying to join.

Solution is to connect to the CIMC KVM and mount the firmware iso to install it.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz40545/?referring_site=bugquickviewredir

Yan

tinhnho123 · ‎12-30-2015

Hi Tomas,

I selected the cluster size of 3.

Here is the results those commands:

admin@My-APIC01:~> moquery -c infraClusterPol
Total Objects shown: 1

# infra.ClusterPol
childAction :
descr :
dn : uni/controller/clusterpol
lcOwn : local
modTs : 2015-12-17T14:03:59.654+00:00
name :
ownerKey :
ownerTag :
rn : clusterpol
size : 3
status :
uid : 0

admin@My-APIC01:~>

admin@My-APIC01:~>
admin@My-APIC01:~> acidiag avread
Local appliance ID=1 ADDRESS=10.0.0.1 TEP ADDRESS=10.0.0.0/16 CHASSIS_ID=ee4680a2-a4c6-11e5-b649-bbc802c54b19
Cluster of 1 lm(t):1(zeroTime) appliances (out of targeted 3 lm(t):1(2015-12-29T10:53:01.794+00:00)) with FABRIC_DOMAIN name=My ACI Fabric01 set to version=1.0(3f) lm(t):1(2015-12-17T14:05:05.338+00:00)
appliance id=1 last mutated at 1970-01-01T00:00:00.001+00:00 address=10.0.0.1 tep address=10.0.0.0/16 oob address=172.20.11.35/25 version=1.0(3f) lm(t):1(2015-12-29T10:53:01.480+00:00) chassisId=ee4680a2-a4c6-11e5-b649-bbc802c54b19 lm(t):1(2015-12-29T10:52:57.157+00:00) capabilities=0X1FFFFFFF lm(t):1(2015-12-29T10:52:57.157+00:00) rK=(stable,present,0X207373642D782434) lm(t):1(2015-12-29T10:53:01.556+00:00) aK=(stable,present,0X207373642D782434) lm(t):1(2015-12-29T10:53:01.556+00:00) commissioned=1 registered=1 active=yes(1970-01-01T00:00:00.001+00:00) health=(applnc:255 lm(t):1(2015-12-29T10:53:08.481+00:00) svc's)
clusterTime=<diff=-1 common=2015-12-30T07:07:20.099+00:00 local=2015-12-30T07:07:20.100+00:00 pF=<displForm=0 offsSt=0 offsVlu=0 lm(t):1(2015-12-17T14:05:05.476+00:00)>>
---------------------------------------------

admin@My-APIC01:~>

The APIC 2's physical links (SFP+ 10gb) are connected to 2 leafs and the green lights on these ports are lit.

I also ran the setup utility on the APIC 2, I have the same fabric name and ip address scheme (10.0.0.0/16) on APIC 2 but when I'm on APIC1, it still doesn't recognize the APIC 2 as a member of the cluster. Any ideas?

Thanks.

Tomas de Leon · ‎12-30-2015

Please SSH to the Leaf Nodes in which APIC2 is connected and perform the following:

show lldp neighbors

show interface eth1/1 (or the port connected to APIC2)

Also, HTTP to the CIMC of APIC2 and check the LLDP settings for the VIC1225 adapter for APIC2. LLDP should be DISABLED. If not disabled, please disable LLDP on the VIC1225 adapter.

tinhnho123 · ‎12-31-2015

Hi Tomas,

Here are the results of two commands from 2 leafs:

My-Leaf01# show lldp neighbors
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
My-APIC01 Eth1/1 120 eth2-1
APIC2 Eth1/2 120 eth2-1
My-Spine01 Eth1/53 120 BR Eth1/35
My-Spine02 Eth1/54 120 BR Eth1/36
Total entries displayed: 4

My-Leaf01# show interface e1/2
Ethernet1/2 is up (out-of-service)
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: ecbd.1d62.c453 (bia ecbd.1d62.c453)
MTU 9000 bytes, BW 10000000 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 00:27:35
Last clearing of "show interface" counters never
20 interface resets
30 seconds input rate 504 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 552 bps, 1 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 72 multicast packets 1458 broadcast packets
1530 input packets 106655 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 1475 input discard
0 Rx pause
TX
0 unicast packets 68 multicast packets 0 broadcast packets
68 output packets 23188 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

My-Leaf01#

=====================================

My-Leaf02# show lldp neighbors
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
My-APIC01 Eth1/1 120 eth2-2
APIC2 Eth1/2 120 eth2-2
My-Spine02 Eth1/53 120 BR Eth1/35
My-Spine01 Eth1/54 120 BR Eth1/36
Total entries displayed: 4

My-Leaf02# show interface e1/2
Ethernet1/2 is up (out-of-service)
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: ecbd.1dea.672b (bia ecbd.1dea.672b)
MTU 9000 bytes, BW 10000000 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 00:29:42
Last clearing of "show interface" counters never
21 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 48 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 59 multicast packets 0 broadcast packets
59 input packets 12685 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 72 multicast packets 0 broadcast packets
72 output packets 24552 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

My-Leaf02#

When I look at the VIC 1225 from CIMC, I dont see LLDP which you mentioned above. The firmware version is 2.2(1dS1). The attachment is the screen shot of VIC 1225 from CIMC of APIC 2.

Thanks.

Tomas de Leon · ‎12-31-2015

What version of the CIMC are you running?

You may want to upgrade the CIMC to 2.0(3i) using the Host Ugrade Utility and upgrade "ALL". This should upgrade all of the firmware which then should allow you to disable LLDP. Once this is done , the APIC & Leaf should communicate as expected.

T.

tinhnho123 · ‎01-06-2016

Hello,

I upgraded the UCS servers (APIC 1st and 2nd) using Host upgrade utility and upgrade 'All' as you suggested. The upgrade went well. Now I can see the LLDP is disabled under VIC Adapter on CIMC. But the 2nd APIC still can't join the cluster. The attachment is the screen shoot of Cisco VIC adapter from CIMC.

The results of these commands below still the same after the upgrade.

show lldp neighbors

show interface eth1/1 (or the port connected to APIC2)

Any ideas?

Thanks.

Tomas de Leon · ‎01-06-2016

On APIC2, please run the CLI Command:

apic2# acidiag verifyapic

openssl_check: certificate details
subject= CN=FCH1745V13S,serialNumber=PID:APIC-SERVER-L1 SN:FCH1745V13S
issuer= CN=Cisco Manufacturing CA,O=Cisco Systems
notBefore=Jun 24 14:34:08 2014 GMT
notAfter=Jun 24 14:44:08 2024 GMT
openssl_check: passed
ssh_check: passed
all_checks: passed

Also, run the "Date" command on APIC1, APIC2, and the Leaf that is connected to APICs:

For example:

apic1# date
Thu Jan 7 00:38:46 EST 2016

apic2# date
Thu Jan 7 00:39:40 EST 2016

leaf2# date
Thu Jan 7 05:39:20 UTC 2016

The dates need to be in sync for fabric discovery to work. This may require manually setting the clock until the APICs join the fabric and can use NTP.

Please paste the output here. Thanks