ugochunw123,     For operStateReason: notConnect you will want to look into the physical cabling of the peer device as well as speed/negotiation policies tied to the interface in question.   As for the above response, you will want to be more concerned with EPG/Policy usage if you saw the port in an up(out-of-service) state:   Example 1: Check physical connectivity/negotiation b/w the Leaf and Peer device leaf2# show int eth 1/3 | head Ethernet1/3 is down (notconnect) admin state is up, Dedicated Interface Belongs to po2 Hardware: 1000/10000 Ethernet, address:  MTU 9222 bytes, BW 0 Kbit, DLY 1 usec reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, medium is broadcast full-duplex, 10 Gb/s, media type is 10G FEC (forward-error-correction) : disable-fec   Example 2: Check EPG assignment to determine if port is actually in use leaf2# show int eth 1/9 | head Ethernet1/9 is up (out-of-service) admin state is up, Dedicated Interface Belongs to po4 Hardware: 1000/10000 Ethernet, address: MTU 9000 bytes, BW 10000000 Kbit, DLY 1 usec reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, medium is broadcast full-duplex, 10 Gb/s, media type is 10G FEC (forward-error-correction) : disable-fec   -Gabriel ... View more

jvetrive,     Please take a moment to view the following white paper on endpoint learning behaviors in ACI: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html   In it you will find some answers to your queries as well as surrounding info on various features that impact/change the way ACI learns:          Prior to Cisco ACI Release 3.0(1k), if the Limit IP Learning To Subnet option was enabled when the bridge domain was configured for unicast routing, you could experience an impact of 120 seconds as the bridge domain endpoint table is flushed and endpoint learning for IP addresses would be paused (for 120 seconds).          When migrating Layer 3 gateway (L3GW) connectivity to Cisco ACI, you can mitigate this impact by enabling the Limit IP Learning To Subnet option when the bridge domain is configured as a Layer 2–only bridge domain. After you have enabled the option, wait 120 seconds for the timer to expire. Then enable the Unicast Routing option. Because you are not learning IP endpoints on the bridge domain (because it is an L2BD), the 120-second timer will not affect the learning of new MAC-based endpoints.     As for your initial question, it looks like it has been asked historically within the support forums: https://community.cisco.com/t5/application-centric/modification-on-unknown-unicast-policy/td-p/3417884   -Gabriel ... View more

6Maximo,     L3outs do not partake in endpoint learning in the manner that EPGs with static bindings do. At first thought, I would not think your l3out config is having bearing on IP movement.   If you are saying that you have hits when running " less /var/log/dme/log/epmc-trace.txt | grep -i moved", then you should be able to grab the sclass of the epgs as well as the MACs that the IPs are moving between. In most cases, this should map to standard EPGs as opposed to l3outs.   TAC should definitely be able to help clarify the source of dup IPs. -Gabriel ... View more

sentenced,     Without looking into the contents of the snapshot, that error looks like it may be flagged as part of XML/JSON parsing. That is, your working scenario looks to be in XML, whereas your non working appears to be a json file. With that, they are not exactly equal files to compare (fabric vs tenant wide aside).   XMLvs JSON aside, it looks like that error can be seen if something along the following lines is happening: 1. The json is trying to be parsed as xml. 2. The json is broken or invalid.   One question I have, is how did we specifically get the fabric snapshot onto the fabric? Was it generated just via the "Create a snapshot now" button from within the GUI? If so, it is interesting to me that one snapshot would be xml while the other is json, assuming they are both truly using the same "defaultOneTime" config export policy (Admin > Import/Export > Export Policies > Configuration > defaultOneTime)   -Gabriel ... View more

aswokei,     I'm not sure what you mean by it showing up as a single port channel, but technically each individual leaf will only have a single port channel but with a shared the VPC ID. From the looks of your output, it is indeed recognizing that each leaf sees that individual port channel as part of a VPC.   For the ports coming up as suspended, you will need to take a look at your LACP settings on both the ACI side and the Peer device side. Some clarifying questions that may help you do so:   1. What is your port-channel policy set to on the VPC Policy group? (lacp-active, passive, etc) 2. What are the peer devices settings when it comes to lacp? (is it some switch/end host, what is it's bond mode/lacp settings) 3. Assuming both are set to actively participate in LACP, we can then verify that LACPDUs are actively being sent and received by running this command on each leaf switch: show lacp counters interface port-channel 6  (or 5, depending on which leaf you run the command on).   -Gabriel ... View more

aswokei,     Depending on your model of switch, the upper 6-12 ports are Fabric Ports which means they are only meant to be connected into spines and not configured for host connectivity.   From the ACI Fabric Hardware Installation Guide:   Connecting Leaf Switches to Spine Switches For optimal forwarding between endpoints, you must connect each leaf switch (Cisco Nexus 93108TC-EX, 93108TC-FX, 93120TX, 93128TX, 93180LC-EX, 93180YC-EX, 93180YC-FX, 9332PQ, 9372PX, 9372PX-E, 9372TX, 9372TX-E, 9396PX, or 9396TX) to every spine switch (Cisco Nexus 9336PQ, 9504, 9508, or 9516) in the same ACI fabric. The following table lists the number of ports that you can connect on each type of leaf switch and the supported speeds for those ports. Leaf Switch Maximum Number of Uplink Connections Supported Transmission Speeds (Uplink Ports) Cisco Nexus 93108TC-EX 6 QSFP28 fixed ports 10, 40, and 100 Gigabits 1 Cisco Nexus 93108TC-FX 6 QSFP28 fixed ports 10, 40, and 100 Gigabits Cisco Nexus 93120TX 6 QSFP+ fixed ports 40 Gigabits Cisco Nexus 93128TX with M6PQ or M4PQ-E uplink module 6 QSFP+ ports 40 Gigabits with M12PQ uplink module 8 QSFP+ ports (leftmost 8 ports are supported on 12-port module) 1, 10, or 40 Gigabits 1 Cisco Nexus 93180LC-EX 6 QSFP28 fixed ports 40, and 100 Gigabits Cisco Nexus 93180YC-EX 6 QSFP28 fixed ports 10, 40, and 100 Gigabits 1 Cisco Nexus 93180YC-FX 6 QSFP28 fixed ports 10, 40, and 100 Gigabits 1 Cisco Nexus 9332PQ 6 QSFP+ fixed ports 10 or 40 Gigabits 1 Cisco Nexus 9372PX Cisco Nexus 9372PX-E Cisco Nexus 9372TX Cisco Nexus 9372TX-E 6 QSFP+ ports 40 Gigabits Cisco Nexus 9396PX Cisco Nexus 9396TX with M6PQ or M4PQ-E uplink module 6 QSFP+ ports 40 Gigabits with M12PQ uplink module 12 QSFP+ ports 10 or 40 Gigabits 1 1   For 10-Gigabit support, use a QSFP-to-SFP adapter (such as CVR-QSFP-SFP10G) with a SFP+ or SFP transceiver. The overview section of the same guide goes into model specific port usage. -Gabriel ... View more

aswokei,     A port being in an "out-of-service" state implies that the port is up, however no policy has been applied to it.   YOu mentioned that access policies are in place and that there is some association to an EPG. What sort of association to the EPG is there (static binding/vmm domain?) and are any faults being flagged on the epg?   In general, if a port is out of service it is either because: There was no configuration set to that port to explicitly state usage (such as an EPG static binding) There is a config set, however we are missing some set of access policies to make this config valid (check for faults)   If you do have any faults, please let us know what they are as they tend to indicate what may be missing.   -Gabriel ... View more

mmacdonald70,     Switches running ACI mode do not support POAP.   That aside, I am not personally familiar with the capabilities of POAP. Assuming you get a full range of commands and image initializations on Standalone NXOS with POAP, that makes me think the only pieces you could potentially automate through POAP are:   Grab the ACI image from some tftp server and move it into bootflash run the "no boot nxos" command reload After that, one would still need to console into the switch while in loader and manually boot into the ACI image: > boot bootflash:aci-image-name   Maybe someone else with more POAP experience can think of something else?   -Gabriel ... View more

craigdidata,     If you are expecting that traffic from one Bare metal host will be able to reach another Baremetal (or even interface to interface, switched through the simulated Leaf nodes), I do not believe that is the case. The APIC simulator is mainly for API testing. The VMM integration piece works only in that an APIC is able to send/receive updates to vCenter via the OOB mgmt. This allows for an extension of the control plane.   With that said, VMs that were using that DVS would not be able to traverse the simulated leaves for the same reason your Bare metal hosts aren't able to communicate or be learned the way you are expecting.   Please refer to this guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/simulator/install/b_APIC_Simulator_Installation_Guide/b_APIC_Simulator_Installation_Guide_chapter_010.html   Note The simulator server provides no data path switching between external network devices. You must connect the external servers to the tenant management network through additional interfaces on the external servers.   Note The simulator server provides only the capability to validate service network configuration and service function configuration. As shown in the figure, data packets do not traverse through the simulator.   -Gabriel ... View more

bchintak,     While it is true that we will not learn IPs unless "Unicast Routing" is enabled on a BD, please do not freely enable this setting if a BD has already been designed with L2 flows in mind.   For example, if a subset of these L2 endpoints have been doing some sort of Active/Active NIC teaming, this is acceptable with an L2 BD. The moment you enable Unicast routing, if the nic team/bond type settings mean that this particular host has been sourcing a single IP across two MACs, this will cause issues once "Unicast Routing" is enabled unless the NIC team/Bond type is addressed beforehand.   This Whitepaper on Endpoint Learning goes into more details around learning behaviors and considerations: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html   -Gabriel ... View more

rocky1,     Please refer to the following documentation for some tools that may be helpful for getting familiar with the API: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/api/rest/b_APIC_RESTful_API_User_Guide/using_tools_for_api_development_and_testing.pdf   Personally, I am fond of using the API Inspector when I am trying to figure out what calls are being made via the advanced GUI.   After getting the call or class in question, the Managed Object browser (visore) is a very useful minimal tool to understand the query a bit deeper, as well as see what available attributes exist, and there is a button to easily query for all children (or the parent) of the object you queried against.   -Gabriel ... View more