how to do inter-bd icmp?

[culprit]

Same vrf, another BD, another vlan, another epg, same vm-host, another vm-win-server, another subnet

there are not static route to each server, but there are to each server's gateway

how to solve?

[RedNectar]

Hi @culprit ,

Welcome to the community!

If you want ICMP to flow between two hosts on different EPGs and same VRF (does not matter if it is the same or different BD and does not matter if it is the same or different  vm-host if it is another subnet) then you need to:

1. Create a contract that uses the icmp filter found in the common tenant (see screenshot below)
2. have one of the two EPGs Provide that contract
3. have the other EPG Consume that contract
   • Because ICMP is stateless, it does not matter whether EPG1 provides the contract and EPG2 consumes it or the other way around. The IMPORTANT thing is that one of them provides the contract and the other consumes it.

Tip: You could also change steps 2 & 3 so that you navigate within your tenant to Networking > VRFs > YourVRF > EPG|ESG Collection for VRF and add the contract as BOTH a provided contract and a consumed contract - but this would allow ICMP communication between ALL EPGs linked to that VRF - which could be a good thing!

---

Screenshot- creating a contract using the icmp filter from the common tenant

[culprit]

Thanks for you kindness.

There is 'contract', but can not. i expect because there are not paths each way on route table. There are paths to each server's gateway, and can ping to each gateway from each server.(Ex. A can ping to B's gateway, The opposite is the same.)

I want to use ACI to only L2+L3 like BB+L2 switch combination on legacy.

[RedNectar]

Hi @culprit ,

I have assumed that the servers' gateway IP addresses are found in the BD for each server. Am I right?

If so, ACI should sort out the routes. The servers only need a route to their default gateway (as is normally the case). When server1 in EPG1 and in subnet1 connects to a leaf switch, the APIC looks at what contracts EPG1 has, and determines what routes are required to install on the switch to make the contracts work. If server2 in EPG2 and in subnet2 is also connected, then it will set up the necessary routing between the subnets as static routes.

So - when you say there are not paths in the route table, what commands are you using?  Below is the command you should use (from the APIC) but first, you'll need need the node or switch ID of the switches where server1 and server2 are attached, and the full name of the VRF (which will be in the form tenantName:vrfName )

So if:

• server1 is attached to switch ID 1201 (or the ESXi host that server1 is a guest on)
• server2 is attached to switch ID 1202 
• the tenant name is Tenant1
• the VRF name is Production_VRF
• then the command to use is:

apic1# fabric 1201,1202 show ip route vrf Tenant1:Production_VRF

you should see the route to both subnets on both leaf switches.

[culprit]

Good day.

I set server A : 172.25.92.75, server B : 172.25.93.75

on route table 

172.25.92.0/24 and 172.25.93.0/24 are set Next Hop VRF is overlay-1, and Next Hop is 10.0.24.66./32 and Route Type is Static

but 172.25.92.254/32 and 172.25.93.254/32 are set Next Hop VRF is tenantName:vrfName, and Next Hop is 172.25.92.254/32 and 172.25.93.254/32 each and Route Type is Local.

I think there must be 172.25.92.75/32 and 172.25.93.75/32 on route table cos 172.25.92.254/32 and 172.25.93.254/32 are taken ping.

Or I think it might be a situation where ping go to the overlay-1 path route and ping can't come back.

perhaps lost the way on spine?

sorry, i am newb

[RedNectar]

Hi @culprit 

Let me see if I can reconstruct your output (this would have been a lot easier for me if you had cut and pasted it - or if that's not possible, take a picture with your phone and pasted that)

apic1# fabric xxxx show ip route vrf tenantName:vrfName
172.25.92.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.0.24.66%overlay-1, [1/0], 16:56:38, static, tag 4294967294
172.25.92.254/32, ubest/mbest: 1/0, attached, pervasive
    *via 172.25.92.254, vlan37, [0/0], 16:56:38, local, local
172.25.93.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.0.24.66%overlay-1, [1/0], 15:49:01, static, tag 4294967294
172.25.93.254/32, ubest/mbest: 1/0, attached, pervasive
    *via 172.25.93.254, vlan97, [0/0], 15:49:01, local, local

Let's examine some of this:

---

172.25.92.0/24 and 172.25.93.0/24 are set Next Hop VRF is overlay-1, and Next Hop is 10.0.24.66./32 and Route Type is Static

That is EXACTLY what you would expect. What this says is "any traffic that arrives on this switch destined for 172.25.92.0/24 or 172.25.93.0/24, please send to the IPv4 Proxy (10.0.24.66%overlay-1) on the spine"

However, there is a step that the switch will take before doing that - and that is to look at the endpoint table in the switch 

apic1# fabric 1201,1202 show endpoint vrf Tenant1:Production_VRF

but back to the topic

---

but 172.25.92.254/32 and 172.25.93.254/32 are set Next Hop VRF is tenantName:vrfName, and Next Hop is 172.25.92.254/32 and 172.25.93.254/32 each and Route Type is Local.

Assuming 172.25.92.254/32 and 172.25.93.254/32 are the gateway IPs then again, what you'd expect is as I've shown above.

I have no idea what you mean by "'Next Hop VRF is tenantName:vrfName," - I would expect your output would be more like what I've put above, so I'll look at that instead

172.25.93.254/32, ubest/mbest: 1/0, attached, pervasive
   *via 172.25.93.254, vlan97, [0/0], 15:49:01, local, local

What this is saying is "any traffic that arrives destined for 172.25.93.254, send it to the internal VLAN 97 - which is where the actual IP address is configured locally ON THIS SWITCH" - the output from a different switch will most likely use a different internal VLAN ID for the same IP.

You can validate the IP address on that internal vlan using this command

apic1# fabric 1201,1202 show ip interface brief vrf Tenant1:Production_VRF

And finally,

---

I think there must be 172.25.92.75/32 and 172.25.93.75/32 on route table cos 172.25.92.254/32 and 172.25.93.254/32 are taken ping.

No, those IPs won't be in the route table, but in the endpoint table, which ACI looks at before the route table. As I mentioned above, you can check this with: 

apic1# fabric 1201,1202 show endpoint vrf Tenant1:Production_VRF

I hope this helps.

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.

---

[culprit]

Thanks for advise. Our company's policy deny uploading pic. sorry.

tenantName:vrfName means just blank  that can write tenant and vrf's name like your tenant,vrf's name Tenant1:Production_VRF.

Anyway The routing table you expected is almost accurate. Although the vlan number is a little different, I understand that this is a random value within ACI, so it's meaningless.

As your advice, i checked my endpoint table.

VLAN/DOMAIN	ENCAP VLAN	MAC ADDRESS/IP ADDRESS	MAC INFO/ IP INFO	INTERFACE
26	VLAN-2092	SERVER'S MAC ADDRESS	L	ETH1/6
tenantName:vrfName	VLAN-2092	172.25.92.75	L	ETH1/6
28	VLAN-2093	SERVER'S MAC ADDRESS	L	ETH1/6
tenantName:vrfName	VLAN-2093	172.25.93.75	L	ETH1/6

Contract topology is also well. 

172.25.92.75 -> 172.25.92.254 ok, 172.25.93.254 ok

172.25.93.75 -> 172.25.93.254 ok, 172.25.92.254 ok

i don't understand...

why...

[RedNectar · Accepted solution]

Hi @culprit ,

Just a thought that occurred to me: You said your servers are Windows servers I believe.

By default, Windows blocks ping.

Has the firewall on the Windows servers been modified to allow ICMP?

[culprit]

Thank you. I'm embarrassed.
The problem was in a trivial place, but I was able to learn a lot from it in the process of finding it.

Good day.

[RedNectar]

Hi @culprit ,

Don't be embarrassed - look at the bright side. Look how much you've learned about ACI!!!

So glad you got it sorted.

[culprit]

Thank you for your interest.

I'll go build again.

I'll be back with another issue.

See you again.