Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
4655
Views
5
Helpful
3
Replies

APIC Bond0 (Eth2-1 and Eth2-2) and OOB Interfaces down

Go to solution
TFashy
Level 1
Level 1

‎10-27-2017 12:29 AM - edited ‎03-01-2019 05:21 AM

Hi All,

 

I have a multipod fabric with the following APIC distributions

Pod1 - 2 APICs (APIC1 & 2 - both have joined the clustrer)

Pod2 - 1 APIC (APIC3 - not joined as Bond0 and OOB are down)

Pod3 - 2 APICs (APIC4 & 5 - waiting to join cluster)

 

Pod2's 1 APIC Bond0 (Eth2-1 and Eth2-2) and OOB Interfaces are down. I have reset the APIC a few times and after the setup script, it comes up with this error

 

Bond0 ErrorBond0 Error

Then it comes up with more errors -

Error02.jpg

 

The acidiag avread shows that APIC3 is stopping APICs 4 & 5 from joining the cluster as shown below

 

APIC-1# acidiag avread
Local appliance ID=1 ADDRESS=10.0.0.1 TEP ADDRESS=10.0.0.0/16 CHASSIS_ID=98bb5ec6-a895-11e7-8f27-77fa02efb848
Cluster of 2 lm(t):1(2017-10-04T07:24:00.371+00:00) appliances (out of targeted 5 lm(t):2(2017-10-18T03:16:49.894+00:00)) with FABRIC_DOMAIN name=IPN set to version=2.1(1h) lm(t):2(2017-10-04T07:24:31.067+00:00); discoveryMode=PERMISSIVE lm(t):0(1970-01-01T00:00:00.002+00:00)
        appliance id=1  address=10.0.0.1 lm(t):1(2017-10-04T06:14:43.830+00:00) tep address=10.0.0.0/16 lm(t):1(2017-10-04T06:14:43.830+00:00) oob address=10.100.2.3/24 lm(t):1(2017-10-04T06:14:48.285+00:00) version=2.1(1h) lm(t):1(2017-10-18T03:16:51.352+00:00) chassisId=98bb5ec6-{...deleleted} lm(t):1(2017-10-18T03:16:51.352+00:00) capabilities=0XFFFFFFFF--0X2020--0X3 lm(t):1(2017-10-04T07:24:05.561+00:00) rK=(stable,present,0X207373642D687373) lm(t):1(2017-10-04T06:14:48.286+00:00) aK=(stable,present,0X207373642D687373) lm(t):1(2017-10-04T06:14:48.286+00:00) cntrlSbst=(APPROVED, FCH2109V2CA) lm(t):1(zeroTime) podId=1 lm(t):1(2017-10-04T06:14:43.830+00:00) commissioned=YES lm(t):1(zeroTime) registered=YES lm(t):1(2017-10-04T06:14:43.830+00:00) active=YES(2017-10-04T06:14:43.830+00:00) health=(applnc:255 lm(t):1(2017-10-04T06:14:56.279+00:00) svc's)
        appliance id=2  address=10.0.0.2 lm(t):2(2017-10-18T03:16:40.964+00:00) tep address=10.0.0.0/16 lm(t):2(2017-10-04T07:23:58.321+00:00) oob address=10.100.2.4/24 lm(t):1(2017-10-25T05:43:27.685+00:00) version=2.1(1h) lm(t):2(2017-10-18T03:16:51.280+00:00) chassisId=e2234ddc-{...deleleted} lm(t):2(2017-10-18T03:16:51.280+00:00) capabilities=0XFFFFFFFF--0X2020--0X2 lm(t):2(2017-10-18T03:21:52.443+00:00) rK=(stable,present,0X207373642D687373) lm(t):1(2017-10-18T03:16:55.013+00:00) aK=(stable,present,0X207373642D687373) lm(t):1(2017-10-18T03:16:55.013+00:00) cntrlSbst=(APPROVED, FCH2109V267) lm(t):1(2017-10-04T07:24:05.176+00:00) podId=1 lm(t):2(2017-10-18T03:16:40.965+00:00) commissioned=YES lm(t):1(2017-10-04T07:24:00.371+00:00) registered=YES lm(t):1(2017-10-04T07:24:05.164+00:00) active=YES(2017-10-18T03:16:51.286+00:00) health=(applnc:255 lm(t):2(2017-10-18T03:17:03.781+00:00) svc's)
---------------------------------------------
*******Additional elements outside of cluster*******
        appliance id=4 address=10.0.0.4 lm(t):91(2017-10-18T03:17:01.371+00:00) chassisId=4e74ac16-{...deleleted} lm(t):91(2017-10-18T03:17:01.371+00:00) cntrlSbst=(DO_SOMETHING, FCH2109V24E) lm(t):91(2017-10-18T03:17:01.371+00:00)
        appliance id=5 address=10.0.0.5 lm(t):91(2017-10-18T03:17:01.367+00:00) chassisId=3f98dacc-{...deleleted} lm(t):91(2017-10-18T03:17:01.367+00:00) cntrlSbst=(DO_SOMETHING, FCH2109V25P) lm(t):91(2017-10-18T03:17:01.367+00:00)
---------------------------------------------
clusterTime=<diff=5835 common=2017-10-25T06:40:05.663+00:00 local=2017-10-25T06:39:59.828+00:00 pF=<displForm=0 offsSt=0 offsVlu=0 lm(t):2(2017-10-04T07:24:23.512+00:00)>>



Application Policy Infrastructure Controller
Last login: Wed Oct 25 22:45:12 2017
********************************************************************************
     Fabric discovery in progress, show commands are not fully functional
     Logout and Login after discovery to continue to use show commands.
********************************************************************************
APIC-4#
APIC-4#
APIC-4#
APIC-4# acidiag avread
Local appliance ID=4 ADDRESS=10.0.0.4 TEP ADDRESS=10.0.0.0/16 CHASSIS_ID=4e74ac16-b2e3-11e7-92f1-35fc5c35f772
Cluster of 4 lm(t):4(zeroTime) appliances (out of targeted 5 lm(t):4(zeroTime)) with FABRIC_DOMAIN name=IPN set to version=2.1(1h) lm(t):4(zeroTime); discoveryMode=PERMISSIVE lm(t):0(zeroTime)
        appliance id=1  address=10.0.0.1 lm(t):4(2017-10-17T02:32:29.609+00:00) tep address=0.0.0.0 lm(t):0(zeroTime) oob address=0.0.0.0 lm(t):0(zeroTime) version= lm(t):0(zeroTime) chassisId=98bb5ec6-{...deleleted} lm(t):4(2017-10-17T02:32:29.609+00:00) capabilities=0XFFFFFFF--0X2020--0 lm(t):0(zeroTime) rK=(stable,absent,0) lm(t):0(zeroTime) aK=(stable,absent,0) lm(t):0(zeroTime) cntrlSbst=(UNDEFINED, ) lm(t):0(zeroTime) podId=0 lm(t):0(zeroTime) commissioned=YES lm(t):4(zeroTime) registered=YES lm(t):4(2017-10-17T02:32:29.571+00:00) active=NO(zeroTime) health=(applnc:1 lm(t):0(zeroTime))
        appliance id=2  address=10.0.0.2 lm(t):4(2017-10-17T02:32:29.608+00:00) tep address=0.0.0.0 lm(t):0(zeroTime) oob address=0.0.0.0 lm(t):0(zeroTime) version= lm(t):0(zeroTime) chassisId=e2234ddc-{...deleleted} lm(t):4(2017-10-17T02:32:29.608+00:00) capabilities=0XFFFFFFF--0X2020--0 lm(t):0(zeroTime) rK=(stable,absent,0) lm(t):0(zeroTime) aK=(stable,absent,0) lm(t):0(zeroTime) cntrlSbst=(UNDEFINED, ) lm(t):0(zeroTime) podId=0 lm(t):0(zeroTime) commissioned=YES lm(t):4(zeroTime) registered=YES lm(t):4(2017-10-17T02:32:29.571+00:00) active=NO(zeroTime) health=(applnc:1 lm(t):0(zeroTime))
        appliance id=3  address=0.0.0.0 lm(t):0(zeroTime) tep address=0.0.0.0 lm(t):0(zeroTime) oob address=0.0.0.0 lm(t):0(zeroTime) version= lm(t):0(zeroTime) chassisId= lm(t):0(zeroTime) capabilities=0XFFFFFFF--0X2020--0 lm(t):0(zeroTime) rK=(stable,absent,0) lm(t):0(zeroTime) aK=(stable,absent,0) lm(t):0(zeroTime) cntrlSbst=(UNDEFINED, ) lm(t):0(zeroTime) podId=0 lm(t):0(zeroTime) commissioned=YES lm(t):4(zeroTime) registered=NO lm(t):0(zeroTime) active=NO(zeroTime) health=(applnc:1 lm(t):0(zeroTime))
        appliance id=4  address=10.0.0.4 lm(t):4(2017-10-17T02:32:24.796+00:00) tep address=10.0.0.0/16 lm(t):4(2017-10-17T02:32:24.796+00:00) oob address=10.200.1.4/24 lm(t):4(2017-10-17T02:32:27.794+00:00) version=2.1(1h) lm(t):4(2017-10-17T02:32:28.089+00:00) chassisId=4e74ac16-{...deleleted} lm(t):4(2017-10-17T02:32:28.089+00:00) capabilities=0XFFFFFFFF--0X2020--0X8 lm(t):4(2017-10-17T02:32:28.089+00:00) rK=(stable,present,0X207373642D687373) lm(t):4(2017-10-17T02:32:27.799+00:00) aK=(stable,present,0X207373642D687373) lm(t):4(2017-10-17T02:32:27.799+00:00) cntrlSbst=(APPROVED, FCH2109V24E) lm(t):4(zeroTime) podId=3 lm(t):4(2017-10-17T02:32:24.796+00:00) commissioned=YES lm(t):4(zeroTime) registered=YES lm(t):4(2017-10-17T02:32:24.796+00:00) active=YES(2017-10-17T02:32:24.796+00:00) health=(applnc:112 lm(t):4(2017-10-17T02:33:07.459+00:00) svc's[3]:1 lm(t):4(2017-10-17T02:32:27.794+00:00)[6]:1 lm(t):4(2017-10-17T02:32:27.794+00:00)[9]:1 lm(t):4(2017-10-17T02:32:27.794+00:00)[10]:1 lm(t):4(2017-10-17T02:32:27.794+00:00)[11]:1 lm(t):4(2017-10-17T02:32:27.794+00:00)[14]:1 lm(t):4(2017-10-17T02:32:27.794+00:00)[16]:1 lm(t):4(2017-10-17T02:32:27.794+00:00)[22]:1 lm(t):4(2017-10-17T02:32:27.794+00:00)[23]:1 lm(t):4(2017-10-17T02:32:27.794+00:00))
---------------------------------------------
clusterTime=<diff=0 common=2017-10-25T22:50:18.424+00:00 local=2017-10-25T22:50:18.424+00:00 pF=<displForm=1 offsSt=0 offsVlu=0 lm(t):0(zeroTime)>>
---------------------------------------------

APIC-4#

Labels:
Preview file
7 KB
0 Helpful
1 Accepted Solution

Accepted Solutions

Go to solution
TFashy
Level 1
Level 1

‎11-01-2017 12:49 AM

The Issue is now resolved as it was a hardware problem. it seems the VIC PCI card was not sitting tightly enough. It was discovered looking at the CIMC --> Inventory --> PCI Adapter tab that the VIC card was not visible.

Another way of troubleshooting at this is SSH to the OOB (or CIMC) and run the follwoing commands

1  - run bash

2 - lspci | grep "Ethernet\|VIC" <-- this list all the physical Ethernet ports

3 - ifconfig -a

View solution in original post

3 Replies 3

Go to solution
Rick1776
Level 5
Level 5

‎10-28-2017 09:58 AM

That's really weird, it looks like APIC 4 is trying to use ID=2 which APIC 1 is also using. What does the output for the discovery look like on the APIC manger?

Also check out this link that looks like it might solve your issue....

https://supportforums.cisco.com/t5/application-centric/multipod-apic-discovery/td-p/3087816
0 Helpful

Go to solution
TFashy
Level 1
Level 1

‎11-01-2017 12:49 AM

The Issue is now resolved as it was a hardware problem. it seems the VIC PCI card was not sitting tightly enough. It was discovered looking at the CIMC --> Inventory --> PCI Adapter tab that the VIC card was not visible.

Another way of troubleshooting at this is SSH to the OOB (or CIMC) and run the follwoing commands

1  - run bash

2 - lspci | grep "Ethernet\|VIC" <-- this list all the physical Ethernet ports

3 - ifconfig -a

Go to solution
Rick1776
Level 5
Level 5
In response to TFashy

‎11-01-2017 08:54 AM

Crazy, it comes down to the physical connection again. Thanks for sharing.
0 Helpful
Customers Also Viewed These Support Documents

Review Cisco Networking for a $25 gift card

Save 25% on Day-2 Operations Add-On License