- Introduction
- ACI Topology
- Prerequisite
- Verify Boarder Leaf side
- Verify Server Leaf side
- Verify Contract
- SPAN/ELAM Analysis
- Appendix-1: How to confirm sclass(pcTag) for EPG / External EPG
- Appendix-2: How to confirm Nexthop Table
- Reference
Introduction
This document explains about L3Out basic troubleshooting.
Please refer to L3Out here.
ACI Topology
The topology is as shown below and the communication is from external EP (1.2.3.4/32) to Server-A placed under Leaf103/104 (Server Leaf). The external network is in Tenant: Cisco(vrf1), and Server-A belongs to Tenant: Datacenter(vrf2).
Prerequisite
- ACI Version: 4.1(1i) and Leaf devices are 9300 EX series switches.
- The configuration for Access Policy, Tenant, VRF, BD、External Routed Network are already done.
- Some configuration note:
(1) The following scope are checked under External Routed Networks > External Network Instance Profile > Subnets
This is required to share routes between vrfs and allow the communication.
- External Subnets for the External EPG
- Shared Route Control Subnet
- Shared Security Import Subnet
(2) Check "Shared between VRFs" Scope for Subnet under Server-A EPG.
Best Practice of shared service is to configure hared Subnet under provider EPG subnet.
For detail, refer to "Guidelines and Limitations for Shared Services".
(3) Contract
Create provided contract in Server-A EPG then export it to Cisco Tenant.
Then, apply the contract as consumed interface under External Routed Networks >External Network Instance Profile.
# Configuring Inter-Context Communication
Verify Boarder Leaf side
Firstly, check if the external route (1.2.3.4/32) is learnt on boarner leaf.
If the route is not learnt, it is possible that some L3Out configurations are wrong or neighbor session is not established, in this case OSPF neighbor.
| Boarder Leaf |
|---|
|
leaf3# show ip route 1.2.3.4 vrf Cisco:vrf1 -- snip --
1.2.3.4/32, ubest/mbest: 1/0 *via 192.168.2.100, vlan104, [110/5], 1d23h, ospf-default, intra |
How to check OSPF Neighbor
| Boarder Leaf |
|---|
|
leaf3# show ip ospf neighbors vrf Cisco:vrf1 OSPF Process ID default VRF Cisco:vrf1 Total number of neighbors: 2 Neighbor ID Pri State Up Time Address Interface 4.4.4.4 1 FULL/BDR 1d23h 192.168.2.201 Vlan104 1.2.3.4 1 FULL/DR 1d23h 192.168.2.100 Vlan104 |
Also, check the Server-A's subnet (100.125.255.0/24) is correctly imported on boarder leaf.
If the subnet is not on boarder lear, it is possible that "Shared between VRFs" is not checked in the above Prerequisite(2) section.
| Boarder Leaf |
|---|
|
leaf3# show ip route 100.125.255.0 vrf Cisco:vrf1 -- snip --
100.125.255.0/24, ubest/mbest: 1/0, attached, direct, pervasive *via 10.0.112.65%overlay-1, [1/0], 00:00:50, static, tag 4294967295 recursive next hop: 10.0.112.65/32%overlay-1 |
Verify Server Leaf side
Next, check if the external route (1.2.3.4/32) is learnt on server leaf.
If the route is not learnt, it is possible that contracts in Prerequisite(3) section might be wrong, or some required scopes are not checked as described in Prerequisite(1).
In the following example, the two TEP address 10.0.64.127 and 10.0.136.65 are shown. These are Leaf4 and Leaf3 which can be confirmed by "acidiag fnvread".
| Server Leaf |
|---|
|
leaf1# show ip route 1.2.3.4 vrf Datacenter:vrf2 -- snip --
1.2.3.4/32, ubest/mbest: 2/0 *via 10.0.64.127%overlay-1, [200/5], 1d23h, bgp-101, internal, tag 101 recursive next hop: 10.0.64.127/32%overlay-1 *via 10.0.136.65%overlay-1, [200/5], 1d23h, bgp-101, internal, tag 101 recursive next hop: 10.0.136.65/32%overlay-1
|
Relation of TEP and Node
| Server Leaf |
|---|
|
leaf1# acidiag fnvread ID Pod ID Name Serial Number IP Address Role State LastUpdMsgId ---------------------------------------------------------------------------------------------------- 103 1 leaf3 FDO21162J7H 10.0.136.65/32 leaf active 0 104 1 leaf4 FDO21162HY8 10.0.64.127/32 leaf active 0 -- snip -- |
Verify Contract
Let's check contract if the communication still fails even the routes are correctly learnt on boarder/server leaf.
Firstly, we need to check VRF VNID (scope).
From GUI, the segment Id under vrf policy is VRF VNID (scope).
From console, we can check it by "moquery -c l3Ctx".
| Boarder Leaf |
|---|
|
leaf3# moquery -c l3Ctx | egrep "\#|encap|ctxPKey|scope" -- snip -- # l3.Ctx encap : vxlan-3047430 ctxPKey : uni/tn-Cisco/ctx-vrf1 scope : 3047430 |
| Server Leaf |
|---|
|
leaf1# moquery -c l3Ctx | egrep "\#|encap|ctxPKey|scope" -- snip -- # l3.Ctx encap : vxlan-2850819 ctxPKey : uni/tn-Datacenter/ctx-vrf2 scope : 2850819 |
Next, check if the contract is correctly applied using VRF VNID (scope).
# The procedure is same on boarder and service leaf. The following example is on boarder leaf.
Applied rules can be confirmed by "show zoning-rule scope <VRF VNID> "
| Boarder Leaf |
|---|
|
leaf3# show zoning-rule scope 3047430 +---------+--------+--------+----------+---------------+---------+---------+------------+----------+-----------------------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+---------------+---------+---------+------------+----------+-----------------------+ | 4122 | 0 | 0 | implicit | uni-dir | enabled | 3047430 | | deny,log | any_any_any(21) | | 4123 | 0 | 0 | implarp | uni-dir | enabled | 3047430 | | permit | any_any_filter(17) | | 4124 | 0 | 15 | implicit | uni-dir | enabled | 3047430 | | deny,log | any_vrf_any_deny(22) | | 5637 | 84 | 0 | implicit | uni-dir | enabled | 3047430 | | deny,log | shsrc_any_any_deny(12)| | 4252 | 5518 | 84 | 306 | bi-dir | enabled | 3047430 | server_ssh | permit | fully_qual(7) | | 4247 | 84 | 5518 | 307 |uni-dir-ignore | enabled | 3047430 | server_ssh | permit | fully_qual(7) | +---------+--------+--------+----------+---------------+---------+---------+------------+----------+-----------------------+ |
In this case, DstEPG 5518 is External EPG and SrcEPG 84 is Server-A (100.125.255.10/24). If a zoning-rule is not created like this, configuration of contract would be something wrong.
EPG's sclass(pcTag) can be confirmed by GUI or CLI. Refer to Appendix-1 below.
From the above rules, the communication matching to FiterID 306 and 307 are permitted. (5518 --> 84 / 84 --> 5518)
ssh (tcp22) is permitted in this case, Reverse Filter Ports (bi-dir) is checked in contract subject.
In the next step, check "policy-mgr stats" and if counters for matched Rule ID is incremented.
Rule ID 4252 should be incremented in this case.
If this counter is not incremented, it is possible that Server-A's subnet is not imported on boarder leaf or contract issue is there.
Also, if receiving traffic which is not permitted, implicit deny is incremented. Rule 4122 is in this case.
| Boarder Leaf |
|---|
|
leaf3# show system internal policy-mgr stats | grep 3047430 Rule (4122) DN (sys/actrl/scope-3047430/rule-3047430-s-any-d-any-f-implicit) Ingress: 0, Egress: 0, Pkts: 11 RevPkts: 0 Rule (4123) DN (sys/actrl/scope-3047430/rule-3047430-s-any-d-any-f-implarp) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4124) DN (sys/actrl/scope-3047430/rule-3047430-s-any-d-15-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4247) DN (sys/actrl/scope-3047430/rule-3047430-s-84-d-5518-f-307) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4252) DN (sys/actrl/scope-3047430/rule-3047430-s-5518-d-84-f-306) Ingress: 0, Egress: 0, Pkts: 32 RevPkts: 0 Rule (5637) DN (sys/actrl/scope-3047430/rule-3047430-s-84-d-any-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 |
SPAN/ELAM Analysis
SPAN/ELAM is good option if the communication still fails even the routes and contract are correct.
ACI SPAN
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/cli/nx/cfg/b_APIC_NXOS_CLI_User_Guide/b_APIC_NXOS_CLI_User_Guide_chapter_010000.html
ACI ELAM Assistant
https://learningnetwork.cisco.com/docs/DOC-34985
Appendix-1: How to confirm sclass(pcTag) for EPG / External EPG
From GUI:
＜External EPG＞
Tenants > Tenant name > Operational > Resource IDs > External Networks Instance Profile (Class ID)
or
Tenants > External Routed Networks > Networks > Networks Instance Profile (pcTag)
＜EPG＞
Tenant > Tenant name > AP > EPG > policy > General > pcTag(sclass)
From CLI:
＜External EPG＞ vsh -c "show system internal policy-mgr prefix"
| Boarder Leaf |
|---|
|
leaf3# vsh -c "show system internal policy-mgr prefix" | egrep "Class|1.2.3.4" Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete 3047430 12 0xc Up Cisco:vrf1 1.2.3.4/32 5518 True True False |
# In 3.1 or earlier releases, vsh_lc -c "show system internal aclqos prefix"
＜EPG＞ show system internal epm endpoint ip 100.125.255.10
| Server Leaf |
|---|
|
leaf1# show system internal epm endpoint ip 100.125.255.10
MAC : 0050.5690.40e3 ::: Num IPs : 1 IP# 0 : 100.125.255.10 ::: IP# 0 flags : ::: l3-sw-hit: No Vlan id : 32 ::: Vlan vnid : 30065 ::: VRF name : Datacenter:vrf2 BD vnid : 16285706 ::: VRF vnid : 2850819 Phy If : 0x1a02c000 ::: Tunnel If : 0 Interface : Ethernet1/45 Flags : 0x80005c04 ::: sclass : 84 ::: Ref count : 5 |
Appendix-2: How to confirm Nexthop Table
Regarding nexthop entry in route table, we can check the consistency of hardware table as following step.
It is good option to check this table in some cases such as failing to receive packet or forwarding to wrong destination.
# The method is applicable to only EX/FX series
1) vsh_lc -c "show platform internal hal l3 routes" and find "NB HW Idx" for target external route.
In this case, NB HW Idx is "802e" for 1.2.3.4/32. NB (NextBaseType) は A = Adjacency.
# If the target route is ECMP, NB (NextBaseType) is E (= Ecmp). Refer to later section.
2) Next, vsh_lc -c "show plat int hal l3 nexthops" and find "HIT IDX (H)" matches to "NB HW Idx" confirmed in above 1).
This result shows, IP address / MAC address for nexthop.
ECMP case:
1) vsh_lc -c "show platform internal hal l3 routes"
2) N (Next Base ID) shows "E" is ECMP entry.
3) vsh_lc -c "show platform internal hal l3 ecmp ecmp-id " <NB-ID>
In this case, 0x24.
|
Server Leaf |
|---|
|
leaf1# vsh_lc -c "show platform internal hal l3 ecmp ecmp-id 0x24" | grep Nhop Mbr[0] LID:0 NhopId:30315 L2ptr:0x8049 Mbr[1] LID:0 NhopId:30317 L2ptr:0x804b |
4) From these steps, we can see multiple NhopId.
5) vsh_lc -c "show platform internal hal l3 nexthop nexthop-id <XXXXX>"
| Server Leaf |
|---|
|
leaf1# vsh_lc -c "show platform internal hal l3 nexthop nexthop-id 30315" | grep IPv4 IPv4 Address : 10.0.136.65 leaf1# leaf1# vsh_lc -c "show platform internal hal l3 nexthop nexthop-id 30317" | grep IPv4 IPv4 Address : 10.0.64.127 |
Reference
Cisco APIC Layer 3 Networking Configuration Guide, Release 4.0(1)
