External IP is not reaching although default route is there towards FW

Marudhanandham · ‎01-17-2025

Hi,

we are facing issue for reaching a specific external IP , all other IPs from the same subnet can be reached.

we have noticed that trace is not reaching the Border leaf for that specific external IP, it reaches RR spine and get dropped. where as other IPs from the same subnet, trace is able to reach border leaf and get reached destination as well.

Verified that, for a specific destination external ip traffic is not reaching border leaf although there is a default router towards BL in source connected leaf switch

Could you please us to fix this issue

Thanks

Marudha

AshSe · ‎01-18-2025

Hello @Marudhanandham

The issue you're describing in your Cisco ACI setup, where traffic to a specific external IP is not reaching the border leaf (BL) but other IPs in the same subnet are working fine, could be caused by several factors. Below is a step-by-step troubleshooting guide to help you identify and resolve the issue:

1. Verify the Route on the Spine (RR Spine)

Since the trace is reaching the Route Reflector (RR) spine and getting dropped, check the routing table on the spine switch for the specific external IP.
Use the following command on the spine:
```
show ip route <specific-external-IP>
```
Ensure that the route for the specific external IP is present and points to the correct next hop (the border leaf).

If the route is missing:

Check the route advertisement from the border leaf to the spine. Ensure that the border leaf is advertising the route for the specific external IP.
Verify the external connectivity configuration (e.g., L3Out, BGP, OSPF, or static routes) on the border leaf.

2. Check the L3Out Configuration

Verify the L3Out configuration on the border leaf for the specific external IP.
Ensure that the subnet containing the specific external IP is included in the L3Out configuration.
If you're using BGP or OSPF, ensure that the specific external IP is being advertised by the external router and learned by the border leaf.

Commands to check:

On the border leaf:

show ip route <specific-external-IP>
show bgp l2vpn evpn <specific-external-IP> (if using BGP EVPN)

On the external router:
```
show ip bgp <specific-external-IP>
```

3. Check Contracts and Filters

Ensure that the EPGs (Endpoint Groups) involved in the traffic flow have the correct contracts applied.
Verify that the contract allows traffic to the specific external IP.
Check the filters associated with the contract to ensure that they are not blocking traffic to the specific external IP.

Commands to check:

On the APIC:
- Navigate to Tenants > [Your Tenant] > Contracts and verify the contract rules.
- Check the filters applied to the contract.

4. Check Policy-Based Redirect (PBR) or Service Graphs

If you're using Policy-Based Redirect (PBR) or service graphs, ensure that the traffic to the specific external IP is not being redirected incorrectly.
Verify the PBR rules and service graph configuration.

5. Check Endpoint Learning

Verify that the specific external IP is being learned correctly in the ACI fabric.
Use the following command on the APIC or leaf switch:
```
show endpoint ip <specific-external-IP>
```
If the endpoint is not learned, check the L3Out configuration and external router configuration.

6. Check for Blackhole Routes

Ensure that there are no blackhole routes configured for the specific external IP.
Check the routing table on the spine and border leaf for any null routes or incorrect next hops.

7. Check for Overlapping Subnets

Verify that there are no overlapping subnets in the ACI fabric that could cause traffic to the specific external IP to be misrouted.
Use the following command to check for overlapping subnets:
```
show ip route
```

8. Check MTU and Fragmentation Issues

Ensure that there are no MTU or fragmentation issues causing the traffic to the specific external IP to be dropped.
Use the following command to check for MTU issues:
```
ping <specific-external-IP> -f -l <packet-size>
```

9. Check Logs and Counters

Check the logs and counters on the spine and border leaf for any drops related to the specific external IP.
Use the following commands:
```
show logging
show interface counters
```

10. Perform Packet Capture

Perform a packet capture on the source leaf, spine, and border leaf to trace the traffic flow and identify where it is being dropped.

Use the following command to capture packets:

ethanalyzer local interface inband capture-filter "host <specific-external-IP>"

11. Verify Default Route on Source Leaf

Ensure that the source leaf has a default route pointing to the border leaf for external traffic.
Use the following command on the source leaf:
```
show ip route 0.0.0.0/0
```

12. Engage Cisco TAC

If the issue persists after performing the above steps, consider engaging Cisco TAC for further assistance. Provide them with the following information:
1. Configuration of the L3Out and contracts.
2. Routing tables from the source leaf, spine, and border leaf.
3. Packet captures and logs.

By following these steps, you should be able to identify and resolve the issue with traffic to the specific external IP. Let me know if you need further assistance!

Hope This Helps!!!

AshSe

Forum Tips:

Insert photos/images inline - don't attach.
Always mark helpful and correct answers, it helps others find what they need.
For a prompt reply, kindly tag @name. An email will be automatically sent to the member.