Introduction   This document describes the steps to setup and configure ACI multi-site fabric from the scratch.    The ACI Multi-Site feature introduced in version 3.0  allows you to interconnect separate Cisco ACI Application Policy Infrastructure Controller (APIC) cluster domains (fabrics), each site that represents a different availability zone. This helps ensure multi-tenant Layer 2 and Layer 3 network connectivity across sites, and it also extends the policy domain end-to-end across fabrics.  You can create policies in the   Multi-Site   GUI and push them to all integrated sites or selected sites. Alternatively, you can import tenants and their policies from a single site and deploy them on other sites.   Contributed by Linda Wang, Cisco TAC Engineer   Prerequisites   Requirements    1. Setup MultiSite Controller (MSC) by following the Cisco ACI Multi-Site Installation Guide below.   https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/aci_multi-site/sw/1x/installation/b_Cisco_ACI_Multi-Site_Installation_Guide/b_Cisco_ACI_Multi-Site_Installation_Guide_chapter_010.html    2.  ACI fabrics have been fully discovered in two or more sites.     3.  Make sure the APIC clusters deployed in separte sites have the Out of Band (OOB) management connectivity to the MSC nodes.    Components Used The hardware and software used in this example configuration is listed in the table.   Site A Hardware Device Logical Name N9K-C9504 w/ N9K-X9732C-EX spine109 N9K-C93180YC-EX leaf101 N9K-C93180YC-EX leaf102 N9K-C9372PX-E leaf103 APIC-SERVER-M2 apic1   Site B Hardware   Device Logical Name N9K-C9504 w/ N9K-X9732C-EX spine209 N9K-C93180YC-EX leaf201 N9K-C93180YC-EX leaf202 N9K-C9372PX-E leaf203 APIC-SERVER-M2 apic2   IP Network(IPN) N9K-C93180YC-EX   Hardware     Version APIC Version 3.1(2m) MSC Version:   1.2(2b) IPN NXOS: version 7.0(3)I4(8a)      Note: The Cross-site namespace normalization is performed by the connecting spine switches, this would require the 2nd generation or newer Cisco Nexus 9000 Series switches with "EX" or "FX" on the end of the product name. Alternatively Nexus 9364C is supported in 1.1(x) and later ACI Multi-Site release.    For more details on hardware requirements and compatibility information, refer to ACI Multi-Site Hardware Requirements Guide .   Configure:   Logical network diagram     Configurations:   This document mainly focuses on ACI and MSC side configuration for Multi-Site deployment and the IPN switch configuration details are not fully covered.  However, a few important configurations from IPN switch are listed below for reference purpose.    IPN Switch configuration   The following are the configuration used in the IPN device connected to the ACI Spines. vrf context intersite description VRF for Multi-Site lab feature ospf router ospf intersite vrf intersite //Towards to Spine109 in Site-A // Towards to Spine209 in Site-B  interface Ethernet1/49 speed 100000 mtu 9216 no negotiate auto no shutdown interface Ethernet1/49.4 mtu 9150 encapsulation dot1q 4 vrf member intersite ip address 172.16.1.34/27 ip ospf network point-to-point ip router ospf intersite area 0.0.0.1 no shutdown interface Ethernet1/50 speed 100000 mtu 9216 no negotiate auto no shutdown interface Ethernet1/50.4 mtu 9150 encapsulation dot1q 4 vrf member intersite ip address 172.16.2.34/27 ip ospf network point-to-point ip router ospf intersite area 0.0.0.1 no shutdown     Note: MTU of MP-BGP EVPN control-plane communication between spine nodes in different sites: By default, the spine nodes generate 9000-byte packets for exchanging endpoint routing information. If that default value is not modified, the ISN must support an MTU size of at least 9100 bytes. The default value can be tuned by modifying the corresponding system settings in each APIC domain.    In this example, we are using the default Control Plane MTU size (9000 bytes) on the Spine nodes.       Required configuration from APIC   Step 1: Configure iBGP AS and Route Reflector for each site from APIC GUI     1. Login the site's APIC and configure  iBGP AS and Route Reflector for each site's APIC cluster (   APIC GUI--> System --> System Settings --> BGP Route Reflector) , this is the default BGP Route Route Reflector Policy which will be used for the fabric pod profile.      2. Configure the Fabric POD Profile for each Site's APIC cluster, Go to  APIC GUI--> Fabric --> Fabric Policies --> Pod Policies --> Policy Groups , click on the default Pod policy group, select the default "BGP Route Reflector Policy" from the dropdown list, shown as blow.    Step 2: Configure Spine Access Policies include External Routed Domains for each site from APIC GUI   Configure the Spine access policies for Spine uplink to IPN switch with an AEP( Access Entity Profile ) and L3 routed domain ( APIC GUI--> Fabric --> Access Policies), this part configuration steps will be omitted here, providing with the screenshots for reference.  1) create the switch profile     2) create the AAEP (Attachable Access Entity Profile), Layer 3 Routed domain and VLAN Pool under access policies, shown as below.     3) create the Spine access port policy Group, here associate the Attached Entity Profile "msite".   4) create the spine interface profile, associate the IPN facing spine access port to the interface policy group created in previous step.      Notes:   As for now, there is no need to configure L3Out of OSPF under infra tenant from APIC GUI. We will configure this part via MSC and push the configuration to each site directly later on.     Step 3: Configure the External Dataplane TEP per site from APIC GUI   Go to  APIC GUI-->  Infra --> Policies --> Protocol --> Fabric Ext Connection Policies , then create an Intrasite/Intersite Profile, see screenshot below.     Repeat the above steps to complete APIC side configuration for SiteB ACI fabric.      Multi-Site Controller Configuration   Step 1: Add each site one by one in MSC.   Connect and login to MSC GUI.   Click on the "ADD SITE" menu in order to register the Sites one by one in MSC,  you could also see the cluster status on the top right as highlighted.  Use one of the APIC's ip address and assign one unique site ID for each site. (1-127)    Step 2: Configure Infra policies per Site in MSC.   Login to MSC GUI, select "Sites" from the left-hand panel, then click on the "CONFIGURE INFRA" menu as shown in the screenshot below. Configure Fabric Infra General settings:  BGP peer type (full mesh - EBGP /route reflector - IBGP)         Once it's done, select one of the sites from the left panel.  Then you will see site info in the middle, there are three different configuration level by clicking on different area. If you can click on The Site , the Pod or the Spine, It will allow different settings on the config panel (right panel) Once we click on each site on the left panel, the site level configurations(Multi-Site Enable (On), Dataplane Multicast TEP, BGP ASN, BGP Community (e.g. extended:as2-nn4:2:22) , OSPF Area ID, OSPF Area Type (stub prevent tep pool advertising), External Route Domain .etc) will show on the right panel. Here we can configure or modify following. Specify Dataplane Multicast TEP (one lo per site) used for HREP BGP AS (matching AS from the site configured in apic) OSPF area type and area id External Routed Domain  In most cases, the attribute values would have already been retrieved automatically from APIC to MSC.    5. Next, let's click on the Pod area and go to the POD level specific policies. Here we need to specify the Data Plane Unicast TEP, as shown below. 6.  Then click on the Spine area and go to Spine specific infra settings, here we can configure followings (as shown in the screenshot below): For each interface from spine towards to IPN Switch, set IP address and mask, BGP peer on (yes), Control Plane TEP (Router Id), Spine is Reflector (On) 7. Repeat the steps above for other sites and complete the infra configuration in MSC.  8. Click on the "DEPLOY" button at the top,  and this will push the infra configuration to APICs in both sites.      Verify   Step 1: Verify infra configuration from APIC GUI on each APIC cluster Verify  the  Intrasite/Intersite Profile was configured under infra tenant on each apic cluster. Verify the infra L3Out (intersite), ospf, bgp was configured on each apic cluster (APIC GUI) Login the site's APIC and verify the  Intrasite/Intersite Profile  under Tenant  infra> Policies > Protocol > Fabric Ext Connection Policies , the Intersite profile will look like following when site is fully configured/managed by MSC.  Go to APIC GUI --> Tenant Infra --> Networking --> External Routed Networks, here the "intersite" L3Out profile should be created automatically under tenant infra in both sites.  And also, the L3Out logical node and interface profile configuration is correctly set in vlan 4.      Step 2: Verify OSPF/BGP session from Spine CLI on each APIC cluster Verify OSPF is up on spine and gets routes from IPN (Spine CLI) Verify BGP session is up to remote site (Spine CLI) Login to Spine CLI, verify the BGP l2vpn evpn and OSPF is up on each spine, and also verify the node-role for BGP is msite-speaker.    spine109# show ip ospf neighbors vrf overlay-1 OSPF Process ID default VRF overlay-1 Total number of neighbors: 1 Neighbor ID Pri State Up Time Address Interface 172.16.1.34 1 FULL/ - 04:13:07 172.16.1.34 Eth1/32.32 spine109# spine109# show bgp l2vpn evpn summary vrf overlay-1 BGP summary information for VRF overlay-1, address family L2VPN EVPN BGP router identifier 172.16.1.3, local AS number 100 BGP table version is 235, L2VPN EVPN config peers 1, capable peers 1 0 network entries and 0 paths using 0 bytes of memory BGP attribute entries [0/0], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 172.16.2.3 4 200 259 259 235 0 0 04:15:39 0 spine109# spine109# vsh -c 'show bgp internal node-role' Node role : : MSITE_SPEAKER spine209# show ip ospf neighbors vrf overlay-1 OSPF Process ID default VRF overlay-1 Total number of neighbors: 1 Neighbor ID Pri State Up Time Address Interface 172.16.1.34 1 FULL/ - 04:20:36 172.16.2.34 Eth1/32.32 spine209# spine209# show bgp l2vpn evpn summary vrf overlay-1 BGP summary information for VRF overlay-1, address family L2VPN EVPN BGP router identifier 172.16.2.3, local AS number 200 BGP table version is 270, L2VPN EVPN config peers 1, capable peers 1 0 network entries and 0 paths using 0 bytes of memory BGP attribute entries [0/0], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 172.16.1.3 4 100 264 264 270 0 0 04:20:40 0 spine209# spine209# vsh -c 'show bgp internal node-role' Node role : : MSITE_SPEAKER   Login to Spine CLI to check and verify Overlay-1 interfaces.   ETEP (Multipod Dataplane TEP)  The Dataplane Tunnel Endpoint address used to route traffic between multiple Pods within the single ACI fabric.   DCI-UCAST (Intersite Dataplane unicast ETEP (anycast per site)  This anycast dataplane ETEP address is unique per site, it's assigned to all the spines connected to the IPN/ISN device and used to receive L2/L3 unicast traffic.    DCI-MCAST-HREP (Intersite   Dataplane   multicast TEP)  This anycast ETEP address is  assigned to all the spines connected to the IPN/ISN device and used to receive L2 BUM (Broadcast, Unknown unicast and Multicast) traffic.   MSCP-ETEP (Multi-Site Control-plane ETEP)  This is the control plane ETEP address, which is also known as BGP Router ID on each spine for MP-BGP EVPN.      spine109# show ip int vrf overlay-1 <snip> lo17, Interface status: protocol-up/link-up/admin-up, iod: 83, mode: etep IP address: 172.16.1.4, IP subnet: 172.16.1.4/32 IP broadcast address: 255.255.255.255 IP primary address route-preference: 1, tag: 0 lo18, Interface status: protocol-up/link-up/admin-up, iod: 84, mode: dci-ucast IP address: 172.16.1.1, IP subnet: 172.16.1.1/32 IP broadcast address: 255.255.255.255 IP primary address route-preference: 1, tag: 0 lo19, Interface status: protocol-up/link-up/admin-up, iod: 85, mode: dci-mcast-hrep IP address: 172.16.1.2, IP subnet: 172.16.1.2/32 IP broadcast address: 255.255.255.255 IP primary address route-preference: 1, tag: 0 lo20, Interface status: protocol-up/link-up/admin-up, iod: 87, mode: mscp-etep IP address: 172.16.1.3, IP subnet: 172.16.1.3/32 IP broadcast address: 255.255.255.255 IP primary address route-preference: 1, tag: 0   spine209# show ip int vrf overlay-1 <snip> lo13, Interface status: protocol-up/link-up/admin-up, iod: 83, mode: etep IP address: 172.16.2.4, IP subnet: 172.16.2.4/32 IP broadcast address: 255.255.255.255 IP primary address route-preference: 1, tag: 0 lo14, Interface status: protocol-up/link-up/admin-up, iod: 84, mode: dci-ucast IP address: 172.16.2.1, IP subnet: 172.16.2.1/32 IP broadcast address: 255.255.255.255 IP primary address route-preference: 1, tag: 0 lo15, Interface status: protocol-up/link-up/admin-up, iod: 85, mode: dci-mcast-hrep IP address: 172.16.2.2, IP subnet: 172.16.2.2/32 IP broadcast address: 255.255.255.255 IP primary address route-preference: 1, tag: 0 lo16, Interface status: protocol-up/link-up/admin-up, iod: 87, mode: mscp-etep IP address: 172.16.2.3, IP subnet: 172.16.2.3/32 IP broadcast address: 255.255.255.255 IP primary address route-preference: 1, tag: 0   At the end, Make sure no faults is seen from MSC (multi-site controller)  Reference    Cisco ACI Multi-Site Architecture White Paper ... View more

This article aims to raise the awareness to wider audience about a very good tool-- Enhanced EP tracker application during ACI daily operational. It would help detect and remediate stale endpoints (along with monitoring of rapid moving endpoints).  You will find it extremely useful.    ACI-EnhancedEndpointTracker can be installed directly on the APIC as an ACI app or deployed as a standalone app.   You can install this via an OVA. The details are below: OVA: https://cisco.app.box.com/s/6us23gzr8nwplrmtjmpp5xaos1wywa22 Documentation: http://aci-enhancedendpointtracker.readthedocs.io/en/latest/install.html#standalone-application     ... View more

 Moquery is the command line cousin of Vizore, it's very helpful and efficient sometimes during the troubleshooting. This article aims to provide moquery cheat sheet to the users for some most common seen scenarios.   moquery usage   moquery --help usage: Command line cousin to visore [-h] [-i HOST] [-p PORT] [-d DN] [-c KLASS] [-f FILTER] [-a ATTRS] [-o OUTPUT] [-u USER] [-x [OPTIONS [OPTIONS ...]]] optional arguments: -h, --help show this help message and exit -i HOST, --host HOST Hostname or ip of apic -p PORT, --port PORT REST server port -d DN, --dn DN dn of the mo -c KLASS, --klass KLASS comma seperated class names to query -f FILTER, --filter FILTER property filter to accept/reject mos -a ATTRS, --attrs ATTRS type of attributes to display (config, all) -o OUTPUT, --output OUTPUT Display format (block, table, xml, json) -u USER, --user USER User name -x [OPTIONS [OPTIONS ...]], --options [OPTIONS [OPTIONS ...]] Extra options to the query   Basic moquery commands   BRIDGE DOMAIN moquery -c fvBD moquery -c fvBD -f "fv.BD.name==\"BDname\"" moquery -c fvSubnet (BD SVI) CONTEXT moquery -c fvCtx EPG moquery -c fvAEPg moquery -c fvAEPg -f 'fv.AEPg.pcTag=="xxxx"' ENDPOINT moquery -c fvCEp moquery -c fvCEp | grep x.x.x.x -A 10 -B 5 moquery -c fvCEp -f 'fv.CEp.name=="aa:bb:cc:dd:11:22:33:44"' moquery -c fvCEp -f 'fv.CEp.ip=="1.1.1.1"' moquery -c fvRsCEpToPathEp CONSUMED CONTRACT moquery -c vzBrCP PROVIDED CONTRACT moquery -c vzBrCP L3 OUT moquery -c l3extInstP moquery -c l3extDomP moquery -c actrl.PfxEntry moquery -c l3extSubnet FAULT moquery -c faultInst -f 'fault.Inst.code=="F0467"' moquery -c faultRecord -x order-by="faultRecord.created|desc" 'query-target-filter=wcard(faultRecord.created,"2017-12-1[2]")'>/home/admin/auditlog.txt TUNNEL INTERFACES moquery -c tunnelIf   APIC moquery  On APIC, please enter into 'bash' mode, then you can start with moquery VPC  <moquery -c fabricExplicitGEp> -  Find vPC Domain VIP(s)   lindawa@apic1:~> moquery -c fabricExplicitGEp Total Objects shown: 1 # fabric.ExplicitGEp name : leaf103-104 childAction : configQual : configSt : deployed dn : uni/fabric/protpol/expgep-leaf103-104 id : 2 lcOwn : local modTs : 2017-12-12T01:29:24.470+00:00 monPolDn : uni/fabric/monfab-default rn : expgep-leaf103-104 status : uid : 15374 virtualIp : 10.0.120.65/32  Which leaf nodes is this EPG(name known) deployed on in the fabric? lindawa@apic1:~> moquery -c fvLocale | grep dn | grep lindawa-web dn : uni/epp/fv-[uni/tn-lindawa/ap-lindawa-vmm/epg-lindawa-web]/node-101 dn : uni/epp/fv-[uni/tn-lindawa/ap-lindawa-vmm/epg-lindawa-web]/node-102 Is the EPG (keyword for filtering) using dynamic attachment and what leafs? lindawa@apic1:~> moquery -c fvDyPathAtt | grep dn | grep lindawa-web dn : uni/epp/fv-[uni/tn-lindawa/ap-lindawa-vmm/epg-lindawa-web]/node-101/dyatt-[topology/pod-1/protpaths-101-102/pathep-[lindawa-vpc-to-srv37]] dn : uni/epp/fv-[uni/tn-lindawa/ap-lindawa-vmm/epg-lindawa-web]/node-102/dyatt-[topology/pod-1/protpaths-101-102/pathep-[lindawa-vpc-to-srv37]] lindawa@apic1:~> What is the dynamic VLAN encap used for this EPG? lindawa@apic1:~> moquery -c fvIfConn | grep dn | grep lindawa-web dn : uni/epp/fv-[uni/tn-lindawa/ap-lindawa-vmm/epg-lindawa-web]/node-101/dyatt-[topology/pod-1/protpaths-101-102/pathep-[lindawa-vpc-to-srv37]]/conndef/conn-[vlan-826]-[0.0.0.0] dn : uni/epp/fv-[uni/tn-lindawa/ap-lindawa-vmm/epg-lindawa-web]/node-102/dyatt-[topology/pod-1/protpaths-101-102/pathep-[lindawa-vpc-to-srv37]]/conndef/conn-[vlan-826]-[0.0.0.0]   What is the BD name if I know the VLAN encap used for L2Out? lindawa@apic1:~> moquery -c l2extRsEBd | grep -A 20 -B 2 vlan-43 childAction : dn : uni/tn-lindawa/l2out-l2out-test/rseBd encap : vlan-43 forceResolve : yes lcOwn : local modTs : 2018-04-12T05:23:44.593+00:00 monPolDn : uni/tn-common/monepg-default rType : mo rn : rseBd state : formed stateQual : none status : tCl : fvBD tContextDn : tDn : uni/tn-lindawa/BD-recreate tRn : BD-recreate tType : name tnFvBDName : recreate uid : 0 Switch moquery <moquery -c fabricExplicitGEp> -  Find vPC Domain VIP(s), same command is applicable to switch as well.    What other EPG are configured to use leaf101 port 1/34. leaf101# moquery -c fvIfConn | grep dn | grep eth1/34 dn : uni/epp/rtd-[uni/tn-lindawa/out-l3out-ospf/instP-all-ext]/node-101/stpathatt-[eth1/34]/conndef/conn-[unknown]-[192.168.33.3/24]  What VLAN is associated with an interface? leaf101# moquery -c nwPathEp -f 'nw.PathEp.id=="eth1/16"' Total Objects shown: 1 # nw.PathEp id : eth1/16 allocState : allocated childAction : descr : dn : sys/conng/path-[eth1/16] fabricPathDn : topology/pod-1/paths-101/pathep-[eth1/16] lcOwn : local modTs : 2018-03-26T05:51:53.874+00:00 monPolDn : uni/fabric/monfab-default name : nativeEncap : vlan-1940 rn : path-[eth1/16] status : vlanScope : global vlanScopeSupport : supported leaf101#  What speeds are the leaf port configured to operate at? leaf101# moquery -c ethpmPhysIf | grep -E 'dn|operSpeed' dn : sys/phys-[eth1/33]/phys operSpeed : 10G dn : sys/phys-[eth1/34]/phys operSpeed : 1G   ... View more

Here is the troubleshooting checklist which should be ready before customers/partners contact Cisco TAC:   Firmware Version of APIC and Switch Hardware Model Download Switch and APIC techsupport logs Problem description (Symptoms with details) Business impact (eg, what kind of services are impacted, how many users are affected?) When did the problem start What changes done before the problem started What troubleshooting actions that has been performed, and what's the initial analysis.  If you couldn't provide all the information, just try your best to provide as much details as you could.  Useful commands set to get more info of the problem details, and start initial troubleshooting.  On APIC: # show version # show controller # show switch if It's Endpoint connectivity issue, using following show commands to retrieve the EP details, alternatively, you may use EP tracker tool under APIC GUI Operations.  # show endpoints ip x.x.x.x #  show endpoints mac x:x:x:x:x:x:x   e.g.  apic1# show endpoints ip 10.10.10.100 Legends: (P):Primary VLAN (S):Secondary VLAN Dynamic Endpoints: Tenant : lindawa Application : lindawa-vmm AEPg : lindawa-db End Point MAC           IP Address    Node        Interface                           Encap     Multicast Address ----------------- ---------------------------------------- ---------- ------------------------------ --------------- --------------- 00:50:56:91:5B:F0 10.10.10.100   101 102   vpc lindawa-vpc-to-srv37 vlan-829 not-applicable   On Switch: # show module # show vlan ex | grep <encap_vlan> # show vrf  # show ip route vrf <vrf-name> # show endpoint ip x.x.x.x #  show endpoint mac x:x:x:x:x:x:x # show system internal epm vpc (if vPC is configured) # show system internal epm endpoint ip  x.x.x.x # show system internal epm endpoint mac x:x:x:x:x:x:x # show logging ip access-list internal packet-log deny | grep <ip/mac>         // you may use this command to check contract drops on leaf switch, if it's vPC case, remember to check this command on both vPC member.      Here is an example for a detailed problem description.  ACI version: 3.1(1i) Inter-Pod connectivity issue within same EPG. ping from the src in pod1 could not be able to reach the dst in pod2. src ip: 10.15.20.22 (pod1, leaf201/202) dst ip: 10.15.20.59 (pod2, leaf203/204) Recent changes: customer did hardware replacement for both IPN routers(from Nexus N9K-C93180YC-EX to  N9K-C9372PX-E ), then started seeing this issue.   In some complicated scenarios, we also need to collect audit log, event history, fault record history .etc from APIC, this is very general information which is good to have before opening TAC case, please attach it to TAC case once case is opened.    Login to one of APIC by ssh, then: #bash Then run below:     mkdir /tmp/tac-outputs     cd /tmp/tac-outputs     show running-config > show_run.txt     icurl 'http://localhost:7777/api/class/firmwareARunning.xml' > firmwareARunning.xml     icurl 'http://localhost:7777/api/class/fabricNode.xml' > fabricNode.xml     icurl 'http://localhost:7777/api/class/faultInfo.xml' > faultInfo.xml     icurl 'http://localhost:7777/api/class/aaaModLR.xml?order-by=aaaModLR.created|desc&page-size=100000' > aaaModLR.xml     icurl 'http://localhost:7777/api/class/faultRecord.xml?order-by=faultRecord.created|desc&page-size=100000' > faultRecord.xml icurl 'http://localhost:7777/api/class/eventRecord.xml?order-by=eventRecord.created|desc&page-size=100000' > eventRecord.xml     cd /tmp     tar zcvf tac-outputs.tgz tac-outputs     cp tac-outputs.tgz /data/techsupport Please download the file tac-outputs.tgz and upload it to TAC case.       ... View more

  Introduction   The unmanaged mode is also known as Network only switching, which is introduced in Brazos release.  It adds the flexibility for customer to use only network automation for service appliance. If a device is configured as “un-managed” i.e. managed setting is set to false, APIC does not program the device. APIC only allocates the network resources and programs the VLAN/VXLAN on fabric side. The configuration of the L4-L7 device is left to be done by customer. This article will give a configuration example of the unmanaged mode using ASAv in routed mode, how to verify and troubleshoot scenarios will also be given as reference.  Prerequisites   Following settings are NOT needed when a device cluster is configured un-managed Device Package Connectivity Information for the device cluster (vnsLDevViP) and devices (CDev) -Management IP, Credentials, In-band connectivity information) Please note that APIC still needs to know the topology information (LIF, CIF) for the device cluster and devices. This is needed so that APIC can program the appropriate ports on leaf and also can use this information for troubleshooting wizard purposes. Requirements   Integration of L4-7 services in ACI requires specific bridge domain networking configuration.  The requirements can be summarized as:   Each unique interface/direction must be mapped to a unique Bridge Domain ARP Flooding must be enabled for each Bridge Domain L2 Unknown Unicast must be set to Flood for each Bridge Domain Unicast Routing must be enabled when the consumer EPG and destination end point are in different subnets Components Used   ASAv version: 9.4(2) APIC version: 2.3(1f) VMware integration Centos7 VM as a web client Centos7 VM as a web server   Network Diagram     logical relationship diagram as below.   Configuration part  Basic configuration on the ASAv.     ASA Version 9.4(2) ! hostname ASAv-1 interface GigabitEthernet0/0 nameif inside security-level 100 ip address 10.10.10.10 255.255.255.0 ! interface GigabitEthernet0/1 nameif outside security-level 0 ip address 20.20.20.20 255.255.255.0 ! access-list out extended permit icmp any any access-list out extended permit http any any access-group out in interface outside   L4-L7 Configurations on ACI side   Before configuring the L4-L7 integration,  make sure the A ccess Policies/VMM Domain configured and interfaces up and in service, the basic configuration of the two EPGs, BDs etc will not be included in this article. From the output below, the web client and web server are learned locally, also we can see they belongs to different BDs.      Leaf101# show system internal epm endpoint mac 0050.5691.5bf0 MAC : 0050.5691.5bf0 ::: Num IPs : 1 Vlan id : 22 ::: Vlan vnid : 11193 ::: VRF name : lindawa:prod BD vnid : 16285616 ::: VRF vnid : 2490374 Phy If : 0x16000001 ::: Tunnel If : 0 Interface : port-channel2 Flags : 0x80005c05 ::: sclass : 49158 ::: Ref count : 5 EP Create Timestamp : 11/01/2017 10:52:31.488217 EP Update Timestamp : 11/01/2017 21:06:31.088306 EP Flags : local|vPC|MAC|host-tracked|sclass|timer| :::: Leaf101# show system internal epm endpoint mac 0050.5691.6bb9 MAC : 0050.5691.6bb9 ::: Num IPs : 1 Vlan id : 29 ::: Vlan vnid : 11210 ::: VRF name : lindawa:prod BD vnid : 16089028 ::: VRF vnid : 2490374 Phy If : 0x16000001 ::: Tunnel If : 0 Interface : port-channel2 Flags : 0x80004c05 ::: sclass : 49156 ::: Ref count : 5 EP Create Timestamp : 11/01/2017 15:53:33.982626 EP Update Timestamp : 11/01/2017 21:06:31.086101 EP Flags : local|vPC|MAC|sclass|timer| ::::   Important Notice:  There is no need to create EPGs explicitly for the firewall internal and external interfaces, instead of that, shadow EPGs will be created by APIC automatically, we will go there later in this article.    Step1:  Creating l4-l7 device     Step2:  Creating a service graph     Step3.  Apply service graph.        Verification Part     Network Label configuration is also done automatically to ASAv.     Then in order to verify the connectivity, you can run the ping test from the web client to server and vice   versa .           On the firewall side, you could be able to see the hit count for the acl is actually increasing.    ASAv-1# show access-list access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096) alert-interval 300 access-list out; 2 elements; name hash: 0x5589cfea access-list out line 1 extended permit icmp any any (hitcnt=4) 0x4f3e126c access-list out line 2 extended permit http any any (hitcnt=0) 0xb4296acc ASAv-1#   The logical topology looks like below. The contract is applied between the provider and consumer, however the contracts which are applied between the shadow EPGs will not be shown on this graph view, we can check the zoning-rules on leaf.            Verify if the shadow EPGs get deployed from the leaf101   Dynamic Endpoints: Tenant : lindawa L4L7 Device : [uni/tn-lindawa/lDevVip-lindawa-ASAv]-ctx-prod L4L7 Graph : lindawa-ASAv-2ctxprod-N-BD_Web-C-consumer End Point MAC IP Address Node Interface Encap Multicast Address ----------------- ---------------------------------------- ---------- ------------------------------ --------------- --------------- 00:50:56:91:33:C1 --- 101 102 vpc lindawa-vPC-SVR37 vlan-3041 not-applicable Tenant : lindawa L4L7 Device : [uni/tn-lindawa/lDevVip-lindawa-ASAv]-ctx-prod L4L7 Graph : lindawa-ASAv-2ctxprod-N-prod-C-provider End Point MAC IP Address Node Interface Encap Multicast Address ----------------- ---------------------------------------- ---------- ------------------------------ --------------- --------------- 00:50:56:91:A0:5A --- 101 102 vpc lindawa-vPC-SVR37 vlan-3022 not-applicable Tenant : lindawa Application : lindawa-svcs AEPg : lindawa_svr End Point MAC IP Address Node Interface Encap Multicast Address ----------------- ---------------------------------------- ---------- ------------------------------ --------------- --------------- 00:50:56:91:5B:F0 --- 101 102 vpc lindawa-vPC-SVR37 vlan-3001 not-applicable Tenant : lindawa Application : lindawa-web AEPg : web End Point MAC IP Address Node Interface Encap Multicast Address ----------------- ---------------------------------------- ---------- ------------------------------ --------------- --------------- 00:50:56:91:31:A5 --- 101 102 vpc lindawa-vPC-SVR37 vlan-3018 not-applicable 00:50:56:91:6B:B9 --- 101 102 vpc lindawa-vPC-SVR37 vlan-3018 not-applicable Total Dynamic Endpoints: 6 Total Static Endpoints: 0 apic1# Leaf101# show vlan ex | grep -A 2 -B 2 ASA 18 lindawa:prod active Eth1/33, Po2 22 lindawa:lindawa-svcs:lindawa_svr active Eth1/33, Po2 27 lindawa:lindawa_ASAvctxprod:provider active Eth1/33, Po2 29 lindawa:lindawa-web:web active Eth1/33, Po2 30 lindawa:lindawa_ASAvctxprod:consumer active Eth1/33, Po2 18 enet CE vxlan-16285616 22 enet CE vlan-3001 27 enet CE vlan-3022 29 enet CE vlan-3018 30 enet CE vlan-3041 Leaf101# show system internal epm vlan 27 detail // Shadow EPG for firewall provider connector VLAN 27 VLAN type : FD vlan hw id : 22 ::: sclass : 32776 access enc : (802.1Q, 3022) fabric enc : (VXLAN, 11214) Object store EP db version : 4 BD vlan id : 18 ::: BD vnid : 16285616 ::: VRF vnid : 2490374 Valid : Yes ::: Incomplete : No ::: Learn Enable : Yes pol_ctrl_flags: Endpoint count : 1 ::: Local Endpoint count : 1 :::: Leaf101# show system internal epm vlan 30 detail // Shadow EPG for firewall consumer connector VLAN 30 VLAN type : FD vlan hw id : 23 ::: sclass : 32777 access enc : (802.1Q, 3041) fabric enc : (VXLAN, 11233) Object store EP db version : 4 BD vlan id : 28 ::: BD vnid : 16089028 ::: VRF vnid : 2490374 Valid : Yes ::: Incomplete : No ::: Learn Enable : Yes pol_ctrl_flags: Endpoint count : 1 ::: Local Endpoint count : 1 :::: :::: Troubleshooting part   As mentioned in the beginning, each unique interface/direction must be mapped to a unique Bridge Domain.  Which means, the two EPGs should belongs to different BDs.    If the web client and web server are in same BD,   then  only the shadow EPG for the ASAv(inside interface) would be created, if you check the connection connectors, you would see that APIC allocates same the encap vlan and sclass for both consumer and provider connectors, which means only one shadow EPG will be created here.       Now, if we would like to apply the same service graph template to a different pair of EPGs, what would happen? Short answer is, the service graph instance will not be deployed successfully.       Fault F1690 is raised with description: Configuration is invalid due to Encap is already allocated for a virtual cluster interface.     This is because the  consumer/provider connectors on the firewall has already been taken.  Here is the correct procedure in order to do so.    1. You need to modify the ASAv VM, an extra pair of network adapters need to be created.      2.Then go to the L4-L7 devices, create two more concreted interfaces and cluster interfaces for the ASAv device.    Select the device, then right-click on it, choose the "Create Concrete Interface" option, add two more interfaces here: inside2, outside2.    Create two cluster interfaces as well, don't forget this. You need to use the concreate interfaces created just now. 3. While applying the graph template, do NOT forget to manually change the consumer/provider connectors to the corresponding cluster interfaces, see as below. Otherwise, you are going to see the same fault as before.   In order to understand how the contract works for this use case, let's take a look at the zoning-rules and stats in policy-mgr on leaf switch.    Before applying the service graph.   Leaf101# show zoning-rule scope 2490374 Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority ======= ====== ====== ======== ====== ===== ====== ======== 4152 0 16386 implicit enabled 2490374 permit any_dest_any(15) 4153 0 0 implicit enabled 2490374 deny,log any_any_any(20) 4154 0 0 implarp enabled 2490374 permit any_any_filter(16) 4155 0 15 implicit enabled 2490374 deny,log any_vrf_any_deny(21) 4104 0 16388 implicit enabled 2490374 permit Leaf101# Leaf101# show system internal policy-mgr stats | grep 2490374 Rule (4104) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-16388-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4152) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-16386-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4153) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-any-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4154) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-any-f-implarp) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4155) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-15-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Leaf101#   After applying the service graph.   so here we can see how does the contract get applied between the EPGs. The contract is applied between the web client(sclass: 49156) and the shadow EPG(sclass: 32777) of ASAv Outside interface. The contract is applied between the web client(sclass: 49158) and t he shadow EPG(sclass: 32776) of ASAv Inside interface.   Leaf101# show zoning-rule scope 2490374 Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority ======= ====== ====== ======== ====== ===== ====== ======== 4152 0 16386 implicit enabled 2490374 permit any_dest_any(15) 4153 0 0 implicit enabled 2490374 deny,log any_any_any(20) 4154 0 0 implarp enabled 2490374 permit any_any_filter(16) 4155 0 15 implicit enabled 2490374 deny,log any_vrf_any_deny(21) 4104 0 16388 implicit enabled 2490374 permit any_dest_any(15) 4158 49156 32777 default enabled 2490374 permit fully_qual(6) 4159 32777 49156 default enabled 2490374 permit fully_qual(6) 4160 32776 49158 default enabled 2490374 permit src_dst_any(8) 4161 49158 32776 default enabled 2490374 permit src_dst_any(8) Leaf101# leaf101# show zoning-filter filter default FilterId Name EtherT ArpOpc Prot MatchOnlyFrag Stateful SFromPort SToPort DFromPort DToPort Prio Icmpv4T Icmpv6T TcpRules ======== =========== ====== ========= ======= ====== ======= ======= ==== ==== ==== ========= ======= ======== ======== default any unspecified unspecified unspecified no no unspecified unspecified unspecified unspecified def unspecified unspecified    The policy-mgr stats also show the same changes.    Leaf101# show system internal policy-mgr stats | grep 2490374 Rule (4104) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-16388-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4152) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-16386-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4153) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-any-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4154) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-any-f-implarp) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4155) DN (sys/actrl/scope-2490374/rule-2490374-s-any-d-15-f-implicit) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4158) DN (sys/actrl/scope-2490374/rule-2490374-s-49156-d-32777-f-default) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4159) DN (sys/actrl/scope-2490374/rule-2490374-s-32777-d-49156-f-default) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4160) DN (sys/actrl/scope-2490374/rule-2490374-s-32776-d-49158-f-default) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Rule (4161) DN (sys/actrl/scope-2490374/rule-2490374-s-49158-d-32776-f-default) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0 Leaf101#   Reference links:   Cisco APIC Layer 4 to Layer 7 Services Deployment Guide Service Graph Design with Cisco Application Centric Infrastructure White Paper Cisco Application Centric Infrastructure Best Practices Guide   ... View more

Is it supported or NOT supported? It's a frequently asked question.  Before APIC, release 2.3(1f), transit routing was not supported within a single L3Out profile. In APIC, release 2.3(1f) and later, you can configure transit routing with a single L3Out profile, with the following limitations: If the VRF is unenforced, an external subnet (l3extSubnet) of 0.0.0.0/0 can be used to allow traffic between the routers sharing the same L3EPG. If the VRF is enforced, an external default subnet (0.0.0.0/0) cannot be used to match both source and destination prefixes for traffic within the same L3EPG. To match all traffic within the same L3EPG, the following prefixes are supported: IPv4 0.0.0.0/1—with External Subnets for the External EPG 128.0.0.0/1—with External Subnets for the External EPG 0.0.0.0/0—with Import Route Control Subnet, Aggregate Import IPv6 0::0/1—with External Subnets for the External EPG 8000::0/1—with External Subnets for the External EPG :0:0/0—with Import Route Control Subnet, Aggregate Import Alternatively, a single default subnet (0.0.0.0/0) can be used when combined with a VzAny contract. For example: Use a VzAny providing contract and an L3EPG consuming contract (matching 0.0.0.0/0), or a VzAny consuming contract and L3EPG providing contract (matching 0.0.0.0/0). And use the subnet 0.0.0.0/0—with Import/Export Route Control Subnet, Aggregate Import, and Aggregate Export. The external documentation was also updated yesterday by the doc team https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Transit_Routing.html#concept_CBBCBA5750D84E4BB49CD727FDCF547A ... View more