Hi William,   Firstly the datasheet. If you can find a copy of the Cisco Live presentation BRKACK-3101, then I think you'll get all the information  you want there.  Another option is the DCAC9K courseware - there are some flowcharts in there that are not too bad.   And if given about 90 mins with a whiteboard, I can also explain how traffic is forwarded.  I don't have 90 mins, and I type slower than I talk, so here goes.   A frame arrives at a leaf from an EndPoint. The leaf examines the encapsulation (VLAN/VXLAN) of the arriving frame to determine the Source EPG, then removes the encapsulation. The leaf examines the destination of the arriving frame and looks at it Station Tables to see if it know the destination MAC IF the destination MAC is the leaf MAC, then route the packet - see explanation for routed packets later IF the destination MAC is found, determine the destination EPG from the Station Table. IF the source and destination MAC are in the same EPG,  then forward the frame: 1. If the destination MAC is local to the leaf, encapsualte it in the appropriate encapsualtion (VLAN/VXLAN) and forward it. 2. If the destination MAC is remote, encapsulate the entire frame (minus the original VLAN/VXLAN header) in an iVXLAN encapsulation packet addressed to the VTEP of the destination leaf and the VNID of the Bridge Domain. ELSEIF source and destination MAC are in different EPGs Check to see if a contract exists that allows the frame. IF the frame is allowed, then: 1. If the destination MAC is local to the leaf, encapsualte it in the appropriate encapsualtion (VLAN/VXLAN) and forward it. 2. If the destination MAC is remote, encapsulate the entire frame (minus the original VLAN/VXLAN header) in an iVXLAN encapsulation packet addressed to the VTEP of the destination leaf and the VNID of the Bridge Domain. ELSE (ie there is no contract or the frame is forbidden), drop the frame. ELSEIF the destination MAC is unknown, forward according to the L2 Unknown Unicast setting - by default it will encapsulate the entire frame (minus the original VLAN/VXLAN header) in an iVXLAN encapsulation packet addressed to the VTEP L2 UNKNOWN MAC anycast address of the PROXY and the VNID of the Bridge Domain.   That pretty much takes care of L2 traffic. But you also need to understand how iVXLAN encapulation works - that a look at https://tools.ietf.org/html/draft-smith-vxlan-group-policy-05 for some clues - it is not the whole story, but the most importnat part is desribed here.   Now L3 traffic.  I don't have time to include External L3 traffic in this story, but for IP packets destined to subnets that are part of the known subnets for a particular VRF, the story is very simlar to layer 2.  REcall I started with:   A frame arrives at a leaf from an EndPoint. The leaf examines the encapsulation (VLAN/VXLAN) of the arriving frame to determine the Source EPG, then removes the encapsulation. The leaf examines the destination of the arriving frame and looks at it Station Tables to see if it know the destination MAC IF the destination MAC is the leaf MAC, then route the packet   Here is the "Route the packet" logic   IF the destination IP is found, determine the destination EPG from the Station Table. IF the source and destination IP are in the same EPG,  then forward the packet: 1. If the destination IP is local to the leaf, encapsualte it in the appropriate encapsualtion (VLAN/VXLAN) and forward it.  This may invlove having to ARP for the destination MAC address first. 2. If the destination IP is remote, encapsulate the entire frame (minus the original VLAN/VXLAN header) in an iVXLAN encapsulation packet addressed to the VTEP of the destination leaf and the VNID of the VRF.    ELSEIF source and destination IPs are in different EPGs Check to see if a contract exists that allows the packet. IF the packet is allowed, then: 1. If the destination IP is local to the leaf, encapsualte it in the appropriate encapsualtion (VLAN/VXLAN) and forward it. 2. If the destination IP is remote, encapsulate the entire frame (minus the original VLAN/VXLAN header) in an iVXLAN encapsulation packet addressed to the VTEP of the destination leaf and the VNID of the VRF. ELSE (ie there is no contract or the frame is forbidden), drop the packet. ELSEIF the destination IP is unknown, encapsulate the entire frame (minus the original VLAN/VXLAN header) in an iVXLAN encapsulation packet addressed to the VTEP L3 UNKNOWN IP anycast address of the PROXY and the VNID of the VRF.   My flight is boarding. That's all for now I hope this helps   Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem     ... View more

Hi  williammanurung, You have asked a question that reveals one of the greatest features of ACI. The answer is: A Bridge Domain (and its subnets) only exist on the Leaf Switches where it is needed. [The same is true for VRFs and EPGs] So, if you have a leaf switch that has no EndPoints for a particular Bridge Domain, then the Bridge Domain will not exist on that leaf.  But the moment you assign an Endpoint attached to a leaf switch to an EndPoint Group that is linked to that Bridge Domain, the Bridge Domain will be created on the leaf (and if necessay, the VRF will also be created on that leaf too) and the subnet(s) for that BD will also suddenly appear on the leaf.  You'll see them by issuing a  apic1# fabric Leaf_ID show ip route vrf Tenant_Name:VRF_Name command on the APIC.  To see if the Bridge domain exists on a particular leaf use the following command: apic1# fabric Leaf_ID show vlan [extended] But that's not all.  The moment that there the last EndPoint for a particular BD is removed from a leaf, ACI dynamically removes the configration for the BD and Subnets. This is WHY you buy ACI, so that all this is managed for you, and you don't have to worry about it.  Just think of the number of times you have seen configuration on a switch that is no longer needed. With ACI, this doesn't happen. I hope this helps Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem ... View more

Hi Shred, The bug you are porbably talking about is this one: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvn30425/?rfs=iqvred My philosophy is to always take the Suggested release on the software download page.  Sometimes you will see that the suggested version is NOT the latest listed, but after a week or two the suggested version will be updated to the latest release (or a new release added) I hope this helps Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem ... View more

Hi PatrickH1, You are correct. In fact, after I had first posted my answer I later remembered this so I came back and  sneakily added that sentence "All you have to do in the EPG config is to make sure the EPG is linked to a Physical Domain that contains VLAN ID  that was used in the mapping." to cover my guilt.  And as far as I know, there is no other way to add a Physical Domain to the EPG. You would think that since (using this method) the EPG and AAEP are linked, then the EPG would be implicitly associated with every Physical Domain linked to teh AAEP, but it isn't.  I suspect the reason for this is historical - this method of associating AAEPs with EPGs didn't appear until version 2 or 3, so I guess it was added as an afterthought enhancement.   ... View more

Ok - I should have begun by asking WHY you want to know. Firstly, to see the IP addresses of lo1 and lo1023 on a leaf, issue the show ip interface brief vrf overlay-1 command.  To get it all from one place, issue the command from an APIC, but prefix the command with fabric followed by a list of your Leaf IDs apic1# fabric 101, 102 show ip interface brief vrf overlay-1 | egrep -v un ---------------------------------------------------------------- Node 101 (Leaf101) ---------------------------------------------------------------- IP Interface Status for VRF "overlay-1"(4) Interface Address Interface Status (lo0) vlan14 10.0.0.30/27 protocol-up/link-up/admin-up lo0 10.0.16.64/32 protocol-up/link-up/admin-up lo1 10.0.192.67/32 protocol-up/link-up/admin-up lo1023 10.0.0.32/32 protocol-up/link-up/admin-up ---------------------------------------------------------------- Node 102 (Leaf102) ---------------------------------------------------------------- IP Interface Status for VRF "overlay-1"(4) Interface Address Interface Status (lo0) vlan13 10.0.0.30/27 protocol-up/link-up/admin-up lo0 10.0.16.66/32 protocol-up/link-up/admin-up lo1 10.0.192.67/32 protocol-up/link-up/admin-up lo1023 10.0.0.32/32 protocol-up/link-up/admin-up [Edit - my first attempt was wrong - here goes again]  Back to your question "how do i get the leaf switch details [that represent the target of a tunnel]?" Like you said, you can find the tunnel destination by issuing a show interface tunnelX - but to find the actual leaf switch that points to you have to mentally  map that destination IP back to a given Leaf based on the output of the command above. As far as I know, there is no command that tells you that "This tunnel maps to this leaf" unless you wanted to do some real fancy stuff piping the output of the show endpoint command into sed or awk.       ... View more

Hi @PatrickH1, Firstly, my answers are my opinion, and may not reflect the "official" Cisco approach, and there may be much better answers out there than mine (so I encourage others to chip in). When should I use different AEPs? A: This is a question I would ask to see if someone understood the whole Access Policy Chain concept. But if you look at what an Interface Policy Group does, you'll see a fairly simple concept of collecting a set of statndard interface policies (CDP, LLDP etc) and putting them in a named group.  But the bit that is missing from that group is the set of VLANs allowed - or more precisely, the set of Encapsulations allowed, given that there may be VLANs and VXLANs. And of course these sets of encapsulations are kept in Encapsulation Domains which are linked to either a VLAN Pool or a VXLAN Pool - except Cisco didn't call them Encapsulation Domans, but rather Physical Domains, VMM Domains etc.  So what is needed is an entity that gives me a list (ie a Profile) of Encapsulation Domains that tell the Interface Policy Group what VLANs/VXLANs are to be included in this interface profile. So Cisco invented the Encapsulation Domain Profile to do just that - except instead of calling it an Encapulsaton Domain Profile (which I thnk is relatively easy to understand), they called it an Attachable Access Entity Profile (which still is beyond my comprehension) or various variations thereof.  So if an AAEP is a collection of VLAN/VXLANs, the act of linking an Interface Profile to an AAEP makes it possible for whatever Interfaces are linked to the Interface Policy Group to accept traffic on those VLANs/VXLANs. Think of the AAEP as a place where Encapsulations (VLANX/VXLANs) get together with interfaces and have a party. The only VLANs a given interface can dance with are the ones at the party. I have referred to this concept in the past as an Attachable Access Entity Party. Google it. In which case should I connect a Firewall/Loadbalancer with L3OUT or L4/7 Serivcegraph? A: I'm going to pass on this one for now to allow someone with more patience to explain. For me, I'd just say to use a L3Out if you want to keep your sanity. [This is NOT the official Cisco answer] Which advantages do I have with L4/7 Serivcegraph unmanaged? A: Using a Managed Service graph means you configure the L4/7 device within ACI. This is not a pretty prospect, but has the advantage of potentially re-configuring say your load balancer, when a change occurs.  An example might be that when a new server is added to the EPG, the managed Load Balancer is reconfigured to add that server to the Servel Pool. Unmanaged means you have to configure you Firewall/Load Balancer using the tools you already use to manage the device. And gives you a better chance of keeping your sanity. Is there any other way to add static ports to an EPG other than under EPG -> Static Port? A: Yes. And it pushes your understanding of the AAEP as well.  Instead of defining Static Mappings under the EPG, instead go to Fabric > Access Policies > Policies > Global > Attachable Access Entity Profiles In the Work Pane, you will see a secton called Application EPGs. Click the [+] icon to add an EPG/Vlan ID and as a result, every port in this Access Policy Chain is now configured to classify any traffic arriving with that VLAN tag into the EPG.  All you have to do in the EPG config is to make sure the EPG is linked to a Physical Domain that contains VLAN ID that was used in the mapping. From my experience, this is the method preferred by Cisco Advanced Services whenever they do a Network Centric Install. I often wonder why this method is not included in the Cisco training courses.   I hope this helps. Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.     ... View more

Hi  Prakin,  Also how do i check the loopback 0 and loopback 1 IP address of leaf switches from APIC CLI ?. The simplest way to check IP addresses of the leaves/spines and APICs is from the CLI of the APIC. Use the command acidag fnvread to check the IPs assigned to the switches, and acidag avread to check the IP addresses of the APICs - which will be the first IP addresses of the VTEP address range allocated during installation (ie 10.0.0.1; 10.0.0.2 and 10.0.0.3 if the deafult 10.0.0.0/16 VTEP range is chosen). And the command aciadiag hwcheck will ping each switch and APIC for you. Also, since each device is a Linux box, you can also use the quintessential ifconfig command if you really want to see the specific IP address of a spefic loopback address. You can also see some of this information in the GUI. Fabric > Inventory > Fabric Membership is a good place to start.  After providing the VTEP pool /16 assignment to apic, by default what interfaces are created, as i can see l0, l1, l1023? You got it!  l0, l1, l1023  is correct. Also check out Fabric > Inventory >  Podx > Leafxxx > Tunnel Interfaces Bottom Line Generally you don't need to know any of these VTEP addresses. The fabric looks after itself and all you need to worry about is making sure your Access Policy Chains are complete, and your Tenant configuration is sound. I hope this helps Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem ... View more

Hi  fthgfhng, Definately something stange going on there - it does seem (now that you've finally established that the code is version 13.1(2m)[Switch]/ 3.1(2m) [ACI]) that this could well be related to the bug you mentioned. (CSCvi51338)   Is doing a firmware upgrade possible? That's what I'd have to recommend at this stage.   ... View more

Hi  fthgfhng , I'm a little confused by your references to the documentation for the Nexus 9000 in NX OS mode and the APIC bug. "NX-OS" and "APIC" have virtually no relationship with each other. Are you using the  9336C-FX2 in NX-OS mode or in ACI mode? If you are using it in ACI mode, the NX-OS manual is useless to you. The show version command output you pasted is VERY STRANGE - it has no version information at all!  It SHOULD look more like this: apic1# fabric 101 show version ---------------------------------------------------------------- Node 101 (Leaf101) ---------------------------------------------------------------- Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php Software BIOS: version 07.64 kickstart: version 13.2(3o) [build 13.2(3o)] system: version 13.2(3o) [build 13.2(3o)] PE: version 3.2(3o) BIOS compile time: 05/16/2018 kickstart image file is: /bootflash/aci-n9000-dk9.13.2.3o.bin kickstart compile time: 10/17/2018 09:18:13 [10/17/2018 09:18:13] system image file is: /bootflash/auto-s system compile time: 10/17/2018 09:18:13 [10/17/2018 09:18:13] Hardware cisco N9K-C9396PX ("supervisor") Intel(R) Core(TM) i3- CPU @ 2.50GHz with 16267264 kB of memory. Processor Board ID SAL1914CNMH Device name: Leaf101 bootflash: 62522368 kB Kernel uptime is 38 day(s), 22 hour(s), 01 minute(s), 16 second(s) Last reset at 900000 usecs after Wed Nov 07 13:21:29 2018 AEDT Reason: reset-by-installer System version: 13.2(3i) Service: Upgrade plugin Core Plugin, Ethernet Plugin [BTW - a tip for pasing CLI output - use the Insert Code function on the edit toolbar]     ... View more

Hi Patrick, ACI reserves VLAN 4095, but you can use VLANs 1-4094. Having said that, there is probably a best-practice document somewhere that says you should not use VLAN IDs above 3967 (and usually VLAN 3967 is recommended for the ACI Infrastructure VLAN) because someone somewhere may have reserved some of those VLANs somewhere - such as the case you mentioned where Cisco reserves VLANs 3968 to 4095 on the Nexus 9K in NXOS mode. So my advice would be to use caution. Avoid if possible. I hope this helps Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem ... View more