ACI Path selection is empty for EPG static ports, L3out vpc path type

[most_ahdy]

I have an issue when creating L3out with interface type of virtual port channel, the path box is empty, although I created interface policy group of type vpc.

The same issue when adding static port of type vpc for application EPG

[RedNectar]

Hi @most_ahdy ,

Check that your VPC Interface Policy Group is linked to an AAEP that is linked to your L3 Domain that is tied to your L3Out.

The easiest way to to this is to navigate to Fabric > Access Policies > Physical and External Domains > L3 Domains > YourL3Domain and then click the Show Usage button. From there, you show see Nodes using this policy - and you should see both nodes that make up your VPC, and if you click on ONE of those nodes, you should see the interface numbers of your VPC.

If you DON'T see the correct ports, you have a problem with your Ascess Policy Chain

[most_ahdy]

Hi RedNectar,

   Thanks for your help, the VPC Interface Policy Group is already linked to an AAEP that is linked to a L3 Domain that is tied to a L3Out, as below, and I can see both vpc nodes and the interfaces used in vpc, but still the path box is not including any thing.

is there is a way I can write it in the box, instead of selecting the interface policy group?

[Sergiu.Daniluk]

Hi @most_ahdy 

What you can do to verify that the vpc is actually configured, is to connect on Leaf-1 and Leaf-2 and use these commands:

show vpc
show port-channel summary 

If your port-channel is formed and up, and associated to a VPC, then you have a GUI issue.

If you don't see the port-channel formed, the one thing left to be verified is the vpc domain config (Fabric > Access Policies  > Policies >  Switch > Virtual Port Channel Default)

Stay safe,

Sergiu

[most_ahdy]

Hi @Sergiu.Daniluk 

So the port-channel must be up with the other side in order to see it in the path box, right?

vpc domain config (Fabric > Access Policies  > Policies >  Switch > Virtual Port Channel Default)  is ok

If the 

[Sergiu.Daniluk]

I don't think it needs to be up, but it should be at least `configured`.

What do you see in the commands output?

[most_ahdy]

@Sergiu.Daniluk 

The vpc is ok as below

show vpc extended:

# show vpc extended
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 101
Peer status : peer adjacency formed ok
vPC keep-alive status : Disabled
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary, operational secondary
Number of vPCs configured : 2
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)
Operational Layer3 Peer : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 up -

vPC status
-------------------------------------------------------------------------------- -
id Port Status Consistency Reason Active vlans Bndl Grp Name
-- ---- ------ ----------- ------ -------------------- --------------- -

345 Po5 down* success success - Core-PG

7 Po7 down* success success - VxrailTierPG

[RedNectar]

Hi @most_ahdy ,

Good morning. Seems there's been a bit more discussion overnight.  While @Sergiu.Daniluk seems to be concentrating on the physical side of things, I'm still not 100% convinced that it is not an Access Policy Chain issue, especially since you say you are having problems with EPG assignment as well.

---

Random thought: You haven't upgraded your system to APIC v6.0 have you? We tried that, strange things happened (we lost sight of FEX ports in ND) so we rolled back.

---

Back to TS. from the APIC, can you run this command: (it's a reallllllly wide output, so make sure you set your terminal width to at least 176 chars before running the command - and as well as pasting the output, attach it as a text file as well [=I'm really p...d off that Cisco has removed the ability to paste pre-formatted code with horizontal scroll])

apic1# show vpc map

This will prove that your VPC Interface Policy Group has been applied to both Leaf 101 and Leaf 103 

Again from the APIC

apic1# fabric 101,103 show port-channel summary

This will prove that your VPC is up on both leaves - you are looking for the (P) after the interface names.

However these commands are just the beginning - and really just re-enforcing @Sergiu.Daniluk's line of thinking.

To follow the FULL ACCESS POLICY CHAIN, you need the GUI

1. Locate the Leaf Profile for leaf 101 and 103 [I HOPE there is a single leaf profile for both switches]
2. In the work pane, double-click the listed Associated Interface Selector Profile (which will actually be a Leaf Profile - there is NO SUCH OBJECT IN ACI CALLED Interface Selector Profile) [I HOPE there is a single Interface Profile listed]
3. When the Interface Profile opens, locate the Interface Selector that represents your port-channel, and double-click it. This will take you to the Access Port Selector page (don't you just love Cisco's naming consistency?)
4. In the Access Port Selector page, locate the Policy Group, and click the external link button 
   
5. This will open a pop-up for your VPC Interface Policy group. On this screen, verify:
   1. The Link Aggregation Type is indeed Virtual Port Channel (VPC)
   2. The Port Channel Policy is a policy that sets LACP to Active (use the external link button to check)
      [Actually, this step has already been verified by the CLI commands, but I'm leaving it here for future readers]
   3. There is indeed a value in the Attached Entity Profile (even though there is NO SUCH OBJECT IN ACI CALLED Attached Entity Profile - instead what you SHOULD see there is an Attachable Access Entity Profile)
6. Click on the external-link button next to the Attachable Access Entity Profile name. This will open a pop-up page for the  Attachable Access Entity Profile 
7. In the Attachable Access Entity Profile page, locate your L3 Domain in the list of Domains for this AAEP.  You can identify that it is a L3 dDomain because it will have (L3) after the name. Double-click on the L3 Domain name. This will open the page for your L3 Domain Profile
8. In the L3 Domain Profile page, locate the VLAN Pool and click the external-link icon next to it. This will open a pop-up page showing your VLAN Pool values
9. In the VLAN Pool page, validate that
   1. The Allocation Method is Static (although with v5.2 onwards it doesn't matter)
   2. The VLANs you wish to use are listed.

If all of the above works as expected, then I'm stumped.

[Edit: 7:38am] Is this related to your issue you had connecting a VPC to an ESXi?  If so, in my step 5.2 above (The Port Channel Policy is a policy that sets LACP to Active) should probably be 
The Port Channel Policy is a policy that sets the policy to MAC Pinning

[/Edit]

[Sergiu.Daniluk · Accepted solution]

@RedNectar: the reason for looking directly at the leaf level (if the vpc/port-channel is configured and up on leafs) is really to prove that access policies are configured or not correctly. If something is not configured correctly, and there are problems with the logical model, then the vpc/po will not be pushed to concrete model.

Meanwhile, I confirmed that port-channel can be down, and still be seen when deploying static ports in EPG.

So definitely there is something wrong in the config/logical model.

My approach in tshoot would be:

1. verify config (as listed above by @RedNectar )

2. Look for faults (System -> Faults) - this usually indicates if there are things I missed in config, or if something else is wrong

3. Moquery checks on APIC level:

moquery -c fabricPathEp -x 'query-target-filter=eq(fabricPathEp.lagT,"link")'
moquery -c fabricPathEp -x 'query-target-filter=and(eq(fabricPathEp.lagT,"node"),wcard(fabricPathEp.dn,"^topology/pod-[\d]*/protpaths-"))'

+ first command checks for port-channels

+ second command checks for vpcs

 NOTE: you can have configuration ok, and still not have the objects created. One situations when this can happen - leaf is inactive or down.

4. Checks on Leaf level, if everything ok at APIC level. Checks should be done on BOTH leafs.

+ here are the commands listed in previous post.

@most_ahdy  can you share the output of the above commands +  the cmds listed by Chris + if any faults are seen on APIC related to leafs/interfaces/po/vpc.

Cheers,

Sergiu

[most_ahdy]

Hi @Sergiu.Daniluk 

Thank you for your information, regrading CLI command, I can not provide the output this week, so later I'll share it

if any faults are seen on APIC related to leafs/interfaces/po/vpc.    No I cant see any faults, as in each step covered by @RedNectar  I check the faults tab and I can not see any faults,

My be it something related to this bug

https://bst.cisco.com/bugsearch/bug/CSCvn15506

I wonder if I can enter the vpc policy group manually instead of selecting it

[Sergiu.Daniluk]

Related to bug, one of the condition is to use "quick start" wizards to configure vpcs. Did you configured them with the wizard or manually? Based on the policies/profiles names, I would say manually, but you can confirm it. And if is like this, bug can be ignored. It's not that one.

[most_ahdy]

I used both manual and wizard, and in both I can not select the vpc interface policy group

[most_ahdy]

@Sergiu.Daniluk 

if any faults are seen on APIC related to leafs/interfaces/po/vpc.

Yes this the issue there was a fault to the interface selector, I enter wrong interfaces IDs

Thanks for your support

[most_ahdy]

@RedNectar Thanks for your detailed information.

I haven't upgraded to APIC v6.0. my version is 5.1(4c)

For CLI output I would not be able to send them this week, I'll send them when available.

1. Locate the Leaf Profile for leaf 101 and 103 [I HOPE there is a single leaf profile for both switches --> yes

2. In the work pane, double-click the listed Associated Interface Selector Profile (which will actually be a Leaf Profile - there is NO SUCH OBJECT IN ACI CALLED Interface Selector Profile) [I HOPE there is a single Interface Profile listed] there is more than one interface profile listed

3. When the Interface Profile opens, locate the Interface Selector that represents your port-channel, and double-click it. This will take you to the Access Port Selector page (don't you just love Cisco's naming consistency?) No

4. In the Access Port Selector page, locate the Policy Group, and click the external link button

This will open a pop-up for your VPC Interface Policy group. On this screen, verify:

1. The Link Aggregation Type is indeed Virtual Port Channel (VPC): yes ok
2. The Port Channel Policy is a policy that sets LACP to Active (use the external link button to check) yes ok

Click on the external-link button next to the Attachable Access Entity Profile name. This will open a pop-up page for the  Attachable Access Entity Profile 

In the Attachable Access Entity Profilepage, locate your L3 Domain in the list of Domainsfor this AAEP.  You can identify that it is a L3 dDomain because it will have (L3)after the name. Double-click on the L3 Domainname. This will open the page for yourL3 Domain Profile   Ok

In the L3 Domain Profile page, locate the VLAN Pool and click the external-link icon next to it. This will open a pop-up page showing your VLAN Pool values

1. In the VLAN Pool page, validate that
   1. The Allocation Method is Static (although with v5.2 onwards it doesn't matter) OK
   2. The VLANs you wish to use are listed. OK

      Is this related to your issue you had connecting a VPC to an ESXi?  If so, in my step 5.2 above (The Port Channel Policy is a policy that sets LACP to Active) should probably be 
      The Port Channel Policy is a policy that sets the policy to MAC Pinning     No, it is VPC to Cat9500 core switch

[RedNectar · Accepted solution]

Hi @most_ahdy ,

OK. A couple of things.

Firstly, kudos for such a detailed answer! (BTW - answering your question inspired me to write this). I just LOVE picture - you've told me so many things with your screendumps!

1. I did notice that there were more than one Interface Profile in Step 1- However, the descriptions seem to indicate that someone knew what they were doing, so no problem there
2. I see there is an error showing in step 2 - it would be good to see what that is
   
3. There is also a fault showing in step 3 - but it is likely to be the same fault as Step 2
   
4. I've noticed something in you Domain configuration which I'll I'll discuss later
   

So thanks for going through that process. Let's discuss

Bug CSCvn15506

That bug DOES look suspiciously like your problem.  I think the following will help determine if the Wizard was to blame:

From the APIC, issue the following commands: - note that that is TWO underscore characters before "ui" in the 2nd command

apic1# cd /mit/uni/infra
apic1# find *__ui*
zsh: no matches found: *__ui*

IF ANY objects are found, we'll go down the path of deleting them - hopefully you'll get the same output as above

Security Domains

I noticed that the L3 Domain was assigned to a Security Domain. So I have a question

Were you logged in as THE admin user when the drop-down didn't appear?

If not, can you please try again logged in as admin

The reason being is that IF you create a security domain, say Tenant1 Security Domain, and then assign that security domain to a user, say User1 and to a tenant, say Tenant_1 (to avoid confusion with the security domain name), and then you log in as User1 then you are seeing EXACTLY WHAT I'D EXPECT you to see when editing Tenant_1

I know that is a poor explanation, and if it does turn out to be the cause of your problem, I'll write a blog post about the Incredible Inadequacies of the ACI RBAC System