BD L2ext Packet Loss

[Maurlai]

Hi, 

   I need to ask someone have the behavior I describe below:

I'm stretching Layer 2 with 2 EPGs and when I put both EPG into the same Bridge Domain, I see a ping packet loss to their default gateway.

Default gateway is a device out of Cisco ACI Fabric.

Scenario:

EPG-A: Tenant|VRF1|BD1|EPG-A

EPG-B: Tenant|VRF1|BD1|EPG-B

Inside the EPG-A I have a static path interfaces and VMM integration with vlan 100 (with virtual machine 100 named VM100)

Inside the EPG-B I have a static path interfaces and VMM integration with vlan 200 (with virtual machine 200 named VM200)

Bridge Domain BD1 have ARP flooding, unicast routing, no subnet entries: I see all mac-address of all virtual machine VM100 and VM200.

Both VMs suffering packet ping loss to their IP default gateway.

If I create a new Bridge Domain (BD2) and I move 1 EPG under this new one BD2, I have no packet loss no more!!!

In other words, this NOT works:

EPG-A: Tenant|VRF1|BD1|EPG-A

EPG-B: Tenant|VRF1|BD1|EPG-B

And this WORKS:

EPG-A: Tenant|VRF1|BD1|EPG-A

EPG-B: Tenant|VRF1|BD2|EPG-B

Any suggestions?

[micgarc2 · Accepted solution]

Best practice for migrating legacy networks (network centric approach) is one VLAN/per BD/per EPG.  Since a BD is our layer 2 boundary (broadcast/flood domain).  This way each Legacy VLAN has a unique Bridge Domain.

-Michael G.

[Jayesh Singh]

 Hi Maurlai,

This happens when your gateways for Vlan 100 and 200 are being learnt via same MAC address. I am assuming you have the same port-channel or trunk port through which both gateways are learnt.

BD is your L2 domain.

Since your BD is same, gateways are flapping between both EPGs, regardless of multiple encaps used because they are learnt with same mac.

To use same BD with multiple VLANs and GW outside ACI, Fabric need to learn both GWs with unique mac addresses. If that is not possible then you need to have separate BDs for separate VLANs.

Because of this it works fine with different BD, but causes performance issue while you have both VLANs in same BD.

Regards,

Jayesh

Don't forget to rate the post if that solves your query.

[Maurlai]

Hi Jayesh,
I appreciate what you explain because it's clarify me the behavior.
My datacenter schema is exactly what you have described.

Thank you.

[M@xim0]

Dear Jayesh,

I was in a situation exactly like the one reported, but after reading your messages I didi a new test. Now I have two different MACs, one for every DG.
I no longer have flapping problems, but now I lose randomly about 10% of the ping for some VMs located in the BD on which the two vlan, thanks to two EPG, are "stretched".

Following your really clear explanation, I was not expecting this. Do you have any opininon?

Thanks,

Max

[Jayesh Singh]

Hi Max,

Thats interesting.

So my understanding of your test scenario is as follows:

1. There are 2 EPGs with same BD

2. Both EPGs have VMM domain attached

3. Static port binding in EPG 1 has Leaf-X,port Y and EPG 2 has Leaf-X,port Z. This is just to confirm that your L2 stretch from ACI to GW device is done from separate physical ports/VPC.

4. You are trying to ping GW from VMs and seeing packet loss. As you said packet loss is observed on some VMs, so for some VMs is it working fine? Also, is it only VMs in one EPG having performance issue or both?

Can you please quickly confirm if my understanding is correct and answer to my few queries above.

Regards,

Jayesh

[M@xim0]

1. There are 2 EPGs with same BD [1.Max] Yes. It is configured as below:

2. Both EPGs have VMM domain attached [2.Max] and they have also Physical Domain for L2Stretching.

3. Static port binding in EPG 1 has Leaf-X,port Y and EPG 2 has Leaf-X,port Z. This is just to confirm that your L2 stretch from ACI to GW device is done from separate physical ports/VPC.
[3.Max] No, I have the same static ports binding on both EPG, same physical ports same VPC.

The two DGs have two IPs on two subnets with two MAC addresses.

4. You are trying to ping GW from VMs and seeing packet loss. As you said packet loss is observed on some VMs, so for some VMs is it working fine? Also, is it only VMs in one EPG having performance issue or both?
[4.Max] On the single VM on the first vlan there is no packet lost from my PC to VM and from VM to DG, instead on different VMs on second vlan I have different packets lost, a sort of isolation for 20/30 seconds.

[Jayesh Singh]

Thanks Max for all details!

Looks like there is problem in point number 3.

If you read my first post it reads,

"This happens when your gateways for Vlan 100 and 200 are being learnt via same MAC address. I am assuming you have the same port-channel or trunk port through which both gateways are learnt."

Connection to both GWs are stretched via your traditional switch and you have the same VPC from ACI to this switch. So the GWs are learnt in ACI via same VPC mac address. GWs are learnt as endpoints but are not connected directly to ACI so mac address learnt in ACI of both gateways are of external switch side portchannel's mac.

Thats where the problem is, same as in case of Maurali...

If feasible try to have separate physical VPC or trunk connectivity to external switch and allow respective vlans on them. Let me know how it works!

Regards,

Jayesh

[M@xim0]

Dear Jayesh,

Thanks for your replay and your time. Unfortunately, I do not have the chance to test with a different connection. My investigation started in order to have different vlans, not only two, in the order of one hundred with related EPG, but in a single BD.

Going a little bit deeper in your explanation, I tried to check MAC address learned on ACI and found the two MAC addresses of the two GWs as learned, and not associated with the VPC MAC address.

10.1.46.3 Vlan 46:

 

10.1.23.3 Vlan18:

Honestly I don't understand why mac address learnt in ACI of both gateways are of external switch side portchannel's mac.

Probably I'm wrong, the GW MAC addresses arrive at the ACI via the same VPC link, but this doesn't mean that their MAC address will be the VPC link MAC. VPC is simple L2, so there are no encaps or decaps for MAC address.

Br,

Max

[Jayesh Singh]

Hi Max,

Thanks for clarifying. There are 2 different MACs learnt in ACI for your respective GWs.

I have come across a scenario where 2 different IPs were learnt via same mac address which kept flapping between 2 EPGs.

However, your scenario is not the same. 

Digging in a lot about L2 extension in ACI, my understanding about the subject is that when we connect ACI with traditional network it has to be done in network centric approach.

This ensures at both the side L2 domain is symmetric and avoids any leak. Performance issue might be because of suboptimal L2 paths as we have single L2 domain(BD) on ACI side and separate on traditional network side.

Hence, it is also the recommended way.

I am still looking for a better document on L2 extension which would give us better insights on this topic.

Regards,

Jayesh