04-05-2023 09:33 AM
Solved! Go to Solution.
04-19-2023 01:15 PM - edited 04-19-2023 01:24 PM
UCSM side is programmed fine. Disjoint L2 & pin groups are accurate. The problem is from ACI side. I don't know 100% why yet, but ACI will receive discovery info from all UCS FI interfaces, and this will tell the fabric that the FIs are a looseNode (lsnode). I'm thinking that since VMM is tied to some of the interfaces already, its also trying to bind the VMM to the physdom Leafs as well - since the same "looseNode" is also see on those interfaces. I don't often see two separate AEPs & domains from the same FIs, so I can't confirm if this is the reason. Usually for this, I'd setup a single AEP for all UCS interfaces, linked to two domains (one phys and one VMM). The AEP is really representative of a single networking enviornment - in which UCSM is just that - one environment (its masquerading as a single host with mulitple MACs behind it). I'd have to lab this up to confirm, but I don't have a UCSM + ACI envornment with multiple uplink sets to test this. I'll keep digging around and see if this might explain things.
Robert
04-19-2023 01:46 PM
Hi @dnoc43 & @Robert Burns ,
Does this sound like an explanation?
https://quickview.cloudapps.cisco.com/quickview/bug/CSCvt02685 (or
https://bst.cisco.com/bugsearch/bug/CSCvt02685 for complete description)
Note the following:
Workaround:
1) Acked Faults | Hide Acked Faults | Hide Delegated Faults
2) Ignore Faults
3) Use testapi to delete the F0467 faults. In the lab, we see the faults coming back after the APIC reload.
4) Only enable Discovery protocol in the VMM Facing Ports towards the lsnode.
I'm not 100% sure and I'm too lazy to look back, but I had thought that you'd already done 4) above
04-05-2023 02:05 PM - edited 04-05-2023 02:06 PM
Hi @dnoc43 ,
Firstly - THANK YOU for the diagrams and the clear explanation of your problem. That's why I marked your Q as Helpful
My bet is that
Now changing either one of these should fix your problem, but let me suggest that the BETTER way is to create a separate AAEP for the management and vMotion functions - and keep the VMs isolated to their own AAEP and VLAN pool.
04-05-2023 03:53 PM
Thanks for the reply Chris. I double checked AAEPs on physical domain and vmm domain. Both are different AAEPs. I also checked the IPGs for both VPCs and they are associated with correct corresponding AAEP. My resolution immediacy was pre-provisioned. I updated them all to on-demand. That didn't seem to resolve the issue.
04-06-2023 12:14 AM - edited 04-08-2023 04:48 AM
Hi @dnoc43 ,
OK. So go to the EPG, then to [Operational] > [Configured Access Policies] or, if you have the Policy Viewer plugin installed, that may be a better option. [Edit: 2023.04.08 - updated diagram so both diagrams match]
From one of these two views, you should be able to see the problem. I hope!
04-06-2023 10:35 AM
Forgot to tag you on response.
04-06-2023 07:47 AM
Very odd! When I attach the phy dom to the EPG it shows every IPG I have created even though no static ports are assigned. Is that normal behavior?
I've collaped the phy domain AAEP to show full policy
04-07-2023 02:10 PM - edited 04-08-2023 05:23 AM
Hi @dnoc43 ,
OK. I've had a bit more of a look - tried to lab it but I can't hook up an ESXi in a VPC ATM, so just added VMs to the design I had before. BUT...
Very odd! When I attach the phy dom to the EPG it shows every IPG I have created even though no static ports are assigned. Is that normal behavior?
YES - this is normal behaviour IF you have mapped the AMDC-AAEP up to the TEST2_EPG for a particular VLAN (this effectively assigns EVERY IPG linked to the AAEP to the EPG)
On the other hand, linking two AAEPs to the same Phys Dom is NOT normal behaviour - at least NOT best practice, although it is technically possible to do so.
I reckon that if you can collapse your two AAEPs to a single AAEP, or if that's not possible, create a new PhysDom (with the same VLAN pool if necessary) and add that PhysDom association to the TEST2_EPG and then make sure there is one PhysDom per AAEP, then your conundrum from above may fix itself.
Let me know how it goes.
04-12-2023 05:26 PM
If I do a endpoint lookup on a known mac address of VM on VMM this is what I see. It's still being learned from 162/172?? Since this is a test environment my next step was to disable the vmotion/storage/mgmt VPC ports on 162/172. Then did the endpoint search again. It's still there??
04-12-2023 06:59 PM - edited 04-12-2023 07:01 PM
Hi @dnoc43 ,
I'm beginning to wonder if the problem MAY be with the bits of the diagram that you didn't include. Based on your naming, I'm assuming your physical setup looks more like this in reality (no wonder you didn't put all the detail in!)
The thing is that your VPCs are NOT connected to ESXi hosts, but to FIs
Now, within the ESXi hosts, the VMM vSwitches will send traffic either to FI-A or FI-B (based on source virtual port which means the same MAC always goes to the same FI)
And UNLESS YOU ARE USING DISJOINT VLANS on the FIs, the FIs will load balance on all uplink ports, which in your case is the two VPCs! And that would explain perfectly why you are seeing traffic from a non-expected VLAN on one or both of the VPCs
So I think your next troubleshooting step would be to look at UCS Manager, and check to see how your uplinks are configured
In UCS Manager, thats Equipment > Fabric Interconnects > Fabric Interconnect A (or B) then click the LAN Uplinks Manager from the General tab.
04-13-2023 09:49 AM
Yes, sorry you're 100% correct. That's exactly how it's wired up. Only difference is we have pinning setup in UCS to pin specific VLANs to port-channels. My concern is why is the APIC still pointing to a VPC IPG that's shut down.
If I look at LEAF162 I see only a remote EP pointing to a tunnel interface destined for LEAF173
04-13-2023 03:19 PM
Hi @dnoc43 ,
You say
we have pinning setup in UCS to pin specific VLANs to port-channels.
that is also known as Disjoint VLANs in UCS speak. But given the behaviour that you are seeing, I'd be checking carefully the UCS configuration to see that the UCS pinning/disjoint VLANs are set up correctly
From the ACI Leaf side of the story, the way VPC switch pairs work is that any MAC address learned on one switch is immediately learned by the other, because the learning switch sends a message to its partner switch - but it should NOT appear as being at the end of a tunnel like your image above.
But back to the UCS Pinning!
Side-Thought | Have you checked that the MAC address is in fact unique? Because if the same MAC existed on both ESXi1 and ESXi2, that could explain this behaviour. |
Starting on one of the ESXi hosts (it doesn't matter which one - not withstanding my side-thought above), I'll assume the MAC address 00:50:56:a0:56:31 belongs to a VM - it certainly LOOKS like a VM MAC.
The VM sends a frame which will arrive at the VMM vSwitch on one of the ESXi hosts on a virtual port.
Side-Thought #2 | Have you checked that the VMM vSwitch is configured for Route based on the originating virtual port |
Assuming that load-balancing on the vSwitch has been configured for Route based on the originating virtual port, the vSwitch encapsulates it with an 802.1Q tag of 2518 and should ALWAYS send that frame towards the same FI. To continue the model, let's assume it is FI-A
FI-A gets the frame, and based on the VLAN grouping for the pinning of VLAN 2518, will always send it towards Leaf163 OR Leaf173. This time, the load balancing will be based on LACP (Fabric Interconnects don't give you a choice) - but that shouldn't matter. Remember both Leaf163 AND Leaf173 share knowledge of that MAC address - or SHOULD share that knowledge.
BUT THIS IS NOT WHAT YOU SEE
Somehow, MAC address 00:50:56:a0:56:31 has turned up on Leaf162 or Leaf172 (which were NOT in your original picture) so it seems there is yet another set of uplinks from the FIs to ACI
Again - this leads me to say - CHECK THE MAC PINNING ON THE FIs. Somehow one of the FIs MUST have sent a packet up the Leaf162/172 VPC link with a source MAC of 00:50:56:a0:56:31 and an 802.1Q tag of 2518.
04-13-2023 01:32 PM
When I add a VMM domain to an EPG. How are the EPG members dynamically learned? As soon as I add the VMM all my leaf switches are added to the dynamic EPG members. This might help point me in the correct direction.
04-13-2023 03:20 PM - edited 04-14-2023 02:50 AM
Hi @dnoc43 ,
When I add a VMM domain to an EPG. How are the EPG members dynamically learned? As soon as I add the VMM all my leaf switches are added to the dynamic EPG members. This might help point me in the correct direction.
04-14-2023 02:24 PM
Chris thanks so much for the video. It did help with the learning source. Which brings me to today's question.
I've noticed as soon as I attach a vm to the dvs I see this in the Client Endpoint in EPG. Notice the 2 vmm incorrect paths and the correct learned path once it sees a packet. The two incorrect paths are port-channels connected to the same FI. I guess ACI is assuming these are also valid paths to the vmware hosts? Is there a way to prevent this learning? The two incorrect paths don't go into "learning" because it never receives a packet on those interfaces.
Below is the layout for one of the two FIs. Both LEAF1s should not be there.
04-17-2023 02:31 PM
Hi @dnoc43 ,
I have to say I'm getting a bit confused. Maybe it's just the naming you are using. From your original posts there were the following VPCS pair of leaf switches
Now, from FI-A's point of view there is
I'm having a conceptual problem resolving the two naming systems.
Now. The bits that worries me most are:
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide