Intermittent communication between EPG's without a contract

[CTRLS Data Centers Ltd]

Hi folks,

We are facing a wiered issue, EPG's with no contract are intermittently communicating without a contract between them. 

Example:

We've 2 EPG's (A and B) under same VRF, there is no contract between EPG-A and EPG-B for tcp 443, but there is a contract on that L3Out. When we are trying to communicate to EPG-B over tcp 443 from EPG-A,  intermittently its communicating. But if we apply a contract between these 2 EPG's, stable communication is happening. If ACI works on Whitelisting model, then there should not be a communication anytime till we apply a contract. We are unable to explain this to our Client on why this intermittent communication is happening without a Contract.

Need help on this as we are facing challenges in understanding this wiered behavior.

Regards,

Ganesh

[Robert Burns]

Are you using vzAny on the user tenant and/or within Common tenant?

Robert

[Mayank Nauni]

Hi Mr. Burns,

Thanks for the pointers. I am also facing a similar issue but in my case the EPGs are communicating without any contracts ( one BD, one subnet and both end hosts ( VMM domain) are in the same subnet). I am not using vzAny in this specific scenario.

I would appreciate any further pointers for the same.

Regards,

Mayank

The detailed scenario is :-
1. Tenant XYZ
2. One VRF
3. One BD with one subnet 192.168.100.X/24
4. Both EPG are in the same subnet ( WEB-APP and DB; WEB APP is 192.168.100.100 & DB is 192.168.100.110, GW is BD for both)
5. Policy is "enforced" on VRF and I am not using vzAny
6. Both EPGs are in a VMM domain and using VMware VDS ( both VMs have been assigned the port-group which are generated by ACI-VMM integration), vlan pool is static with only vlan 1 on it
7. There is a L3 out running OSPF with a FortiGate firewall and I have used default contracts for it ( between L3OUt and Web-EPG)

[Marcel Zehnder]

Hi Mayank

Could you explain point 6 more detailed? I think you should use one vlan per EPG in your pool attached to the VMM-domain. So in your case at least two vlans.

I think because you use only one vlan the VDS switches the traffic directly, but this is just a guess. Could you login via SSH to your APIC and post the output from the following command:

moquery -c hvsExtPol | egrep "dn|Encap"

Marcel

[Mayank Nauni]

HI Marcel,

Thanks for responding in fact I had a similar doubt as well that having only one vlan may be causing this problem but wasn't quite sure about it. Please find the required output below:-

admin@apic1:~> moquery -c hvsExtPol | egrep "dn|Encap"
dn               : comp/prov-VMware/ctrlr-[VCenter_ACI-Lab]-ACI_Lab/sw-dvs-22/extpol-dvportgroup-23
endEncap         : vlan-4094
startEncap       : vlan-1
dn               : comp/prov-VMware/ctrlr-[VCenter_ACI-Lab]-ACI_Lab/sw-dvs-22/extpol-dvportgroup-24
endEncap         : vlan-666
startEncap       : vlan-666
dn               : comp/prov-VMware/ctrlr-[VCenter_ACI-Lab]-ACI_Lab/sw-dvs-22/extpol-dvportgroup-47
endEncap         : vlan-1
startEncap       : vlan-1
dn               : comp/prov-VMware/ctrlr-[VCenter_ACI-Lab]-ACI_Lab/sw-dvs-22/extpol-dvportgroup-48
endEncap         : vlan-1
startEncap       : vlan-1
admin@apic1:~>

[Marcel Zehnder]

Are these two the affected port groups?

dn               : comp/prov-VMware/ctrlr-[VCenter_ACI-Lab]-ACI_Lab/sw-dvs-22/extpol-dvportgroup-47
endEncap         : vlan-1
startEncap       : vlan-1
dn               : comp/prov-VMware/ctrlr-[VCenter_ACI-Lab]-ACI_Lab/sw-dvs-22/extpol-dvportgroup-48
endEncap         : vlan-1
startEncap       : vlan-1

You can check this if you include the name in the grep: moquery -c hvsExtPol | egrep "dn|Encap|name"

I would recommend two things: 

1. Use a dynamic pool for vmm

2. Make your pool bigger 

After that you should delete the vmm domain from the EPGs and reassign it. Then the two port groups will use different vlans which should solve your issue.

[Mayank Nauni]

Awesome, thanks a lot Marcel. I have been stupid enough not to realize that VDS would be locally switching the frames between port-groups in the same vlan.

[Marcel Zehnder]

:-) glad I could help. Have a nice day

[Mayank Nauni]

I did what you had recommended, and it works absolutely fine now. Thanks a lot Marcel, "you are a genius" is an understatement for you :)

[mayank_nauni]

..

[CTRLS Data Centers Ltd]

Hi Robert,

We are using vzAny in the Tenant's for applying the Contracts under that VRF.

Regards,

Ganesh

[Robert Burns]

Ganesh,

See this KB article.  It should explain the behavior your're seeing with vzAny & your EPG's ability to communicate freely without a direct contract applied.

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Use_vzAny_to_AutomaticallyApplyCommunicationRules_toEPGs.html

Robert

[Marcel Zehnder]

Hi Ganesh

Could you post your contracts and subjects between EPG-A and EPG-B?

Do you use 0.0.0.0/0 as subnet on your L3Out?

Marcel

[CTRLS Data Centers Ltd]

Hi Marcel,

We are using 0.0.0.0/1 and 128.0.0.0/1 in the L3Out's.

EPG-A and EPG-B are part of same BD and are with 2 different VLAN's.

Contracts on EPG-A and EPG-B.

EPG-A: (Name: IS)

1. Contract name IS2PRF

2. Subject Sub_IS2PRF

3. Filter details - TCP 22, TCP 19000-19099

EPG-B: (Name: RPIN)

1. Contract name IS2RPIN

2. Subject Sub_IS2RPIN

3. Filter details  -  TCP 443

Even though there is no contract between these 2 EPG's, communication is happening intermittently, after applying the contract between these communication is normal.

Regards,

Ganesh