Hello evereyone,   Recently, we had some virtual machine on hypervisor connected to aci fabric (3.0.2h) in vpc to 2 leaves (93108) When booting up new virtual machines, the private ip and mac are learned but only on one leaf (via show endpoint ip) causing some connectivity issue  ==> When shutting the link toward leaf where ep table is not filled the connectivity is restored.   I can see data coming on both leaves when both link are up with elam and tcpdump tool.   The hypervisor has an ip in infrastructure vlan 3967 (integration) and as the virtual machine, ep in this vlan is only learned on one side but from arp table perspective, the ip is shown with    my server : 10.205.232.104 and  48df.3706.c778 and connected to Po29 of my vpc pair   1102# show ip arp | grep 48df.3706.c778 10.205.232.104 00:01:25 48df.3706.c778 vlan3 ==> vlan3 is in fact vlan 3967   1102# show endpoint ip 10.205.232.104 Legend: s - arp O - peer-attached a - local-aged S - static V - vpc-attached p - peer-aged M - span L - local B - bounce H - vtep +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ 1102#   1102# show mac address-table address 48df.3706.c778 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 36 48df.3706.c778 dynamic - F F po29 * 21 48df.3706.c778 dynamic - F F po29 * 3 48df.3706.c778 dynamic - F F po29   Advanced output from ep learning in vlan 3967 for my server :   MAC : 48df.3706.c778 ::: Num IPs : 0 Vlan id : 3 ::: Vlan vnid : 16777209 ::: BD vnid : 16777209 VRF name : overlay-1 ::: VRF vnid : 16777199 phy if : 0x1600001c ::: tunnel if : 0 ::: Interface : port-channel29 Ref count : 4 ::: sclass : 0 Timestamp : 02/11/1971 06:09:25.581000 ::: Learns Src: EPM EP Flags : local|vPC|MAC|sclass|timer| Aging: Timer-type : HT ::: Timeout-left : 767 ::: Hit-bit : Yes ::: Timer-reset count : 1412 PD handles: [L2]: Hdl : 0xd291609 ::: Hit: Yes ::::   As you will noticed, we don't have any IP associated with this mac on vlan 3 (3967) and the line L3 is missing.     For me it's strange, because from arp perspective the ip is seen in the good vlan and with good mac address.   Best regards,   Thank you  all and   best   wishes for the festive season   !    Yoann     ... View more

This issue is still on 3.0 and 3.1 release.   Yoann ... View more

Hello guy,   We have two stack of cisco 3850 which are both running 3.7.4.e ios-xe. On both stack we have the same behavior as core process cpu showing high usage while top process is only at 15% approximately :   On first stack:  Core 0: CPU utilization for five seconds: 95%; one minute: 94%; five minutes: 94% Core 1: CPU utilization for five seconds: 9%; one minute: 9%; five minutes: 10% Core 2: CPU utilization for five seconds: 2%; one minute: 3%; five minutes: 2% Core 3: CPU utilization for five seconds: 4%; one minute: 2%; five minutes: 4% Core 4: CPU utilization for five seconds: 1%; one minute: 3%; five minutes: 3% Core 5: CPU utilization for five seconds: 1%; one minute: 2%; five minutes: 2% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 6876 1928275 39568377 187 15.44 15.35 15.33 0 pdsd 11097 2742434 16701480 148 1.81 1.64 1.60 0 iosd 6276 1608511 12168463 30 1.52 1.57 1.62 1088 fed 6278 82344 12408233 31 0.19 0.10 0.10 0 stack-mgr   On 2nd stack :  Core 0: CPU utilization for five seconds: 90%; one minute: 91%; five minutes: 91% Core 1: CPU utilization for five seconds: 7%; one minute: 4%; five minutes: 5% Core 2: CPU utilization for five seconds: 5%; one minute: 4%; five minutes: 4% Core 3: CPU utilization for five seconds: 2%; one minute: 3%; five minutes: 3% Core 4: CPU utilization for five seconds: 2%; one minute: 3%; five minutes: 3% Core 5: CPU utilization for five seconds: 97%; one minute: 96%; five minutes: 96% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 6277 2143119 40480418 38 17.38 17.20 17.19 1088 fed 6874 759078 42008504 406 14.92 15.25 15.24 0 pdsd   Does anyone seen this issue ? is it critical ? I've seen many post which had the same behavior but on older release and everyone said that it should be ok after upgrade to 3.7.4.e but apparently not..     Thanks in advance,   Yoann       ... View more

Hello networking world,   I'm currently seeing some syslog regarding obersver1.db (database locked)   <1027> Mar 08 08:18:38 XXXX %LOG_LOCAL0-3-SYSTEM_MSG [E4208012][transition][major][topology/pod-1/node-1/lon/svc-ifc_observer/rpl-1-1] [GenericSQLiteException] ErrorCode=5. Msg=database is locked. SQLiteError at base::Bool db::SQLiteStatement::exec():148. . Path=/data2/dbstats/observer_1.db <1027> Mar 08 08:29:43 XXXX %LOG_LOCAL0-3-SYSTEM_MSG [E4208012][transition][major][topology/pod-1/node-1/lon/svc-ifc_observer/rpl-1-1] [GenericSQLiteException] ErrorCode=5. Msg=database is locked. SQLiteError at base::Bool db::SQLiteStatement::exec():148. . Path=/data2/dbstats/observer_1.db <1027> Mar 08 08:29:43 XXXX%LOG_LOCAL0-3-SYSTEM_MSG [E4208012][transition][major][topology/pod-1/node-1/lon/svc-ifc_observer/rpl-1-1] [GenericSQLiteException] ErrorCode=5. Msg=database is locked. SQLiteError at base::Bool db::SQLiteStatement::exec():148. . Path=/data2/dbstats/observer_1.db I've found a cisco bug regarding this "issue" https://quickview.cloudapps.cisco.com/quickview/bug/CSCuz82822   But we're running aci 3.0.2h at the moment, our release seems to be "impacted" by this behavior.   Does anyone seen this issue from aci 3.x ?    Note : If you look through /data2/dbstats, as we can see on the observer_1.db line, there is admin as the group while root is the one for the others file.   -rw------- 1 ifc root 8117248 Nov 3 08:26 observer_template.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:26 observer_2.db -rw-r--r-- 1 ifc root 8688640 Nov 3 08:26 observer_3.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:26 observer_5.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:26 observer_6.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:26 observer_8.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:26 observer_9.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:27 observer_11.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:27 observer_12.db -rw-r--r-- 1 ifc root 8119296 Nov 3 08:27 observer_14.db -rw-r--r-- 1 ifc root 8126464 Nov 3 08:27 observer_15.db -rw-r--r-- 1 ifc root 8119296 Nov 3 08:27 observer_17.db -rw-r--r-- 1 ifc root 8119296 Nov 3 08:27 observer_18.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:27 observer_20.db -rw-r--r-- 1 ifc root 9075712 Nov 3 08:27 observer_21.db -rw-r--r-- 1 ifc root 8758272 Nov 3 08:27 observer_23.db -rw-r--r-- 1 ifc root 8752128 Nov 3 08:27 observer_24.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:27 observer_26.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:27 observer_27.db -rw-r--r-- 1 ifc root 8204288 Nov 3 08:28 observer_29.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:28 observer_30.db -rw-r--r-- 1 ifc root 8117248 Nov 3 08:28 observer_32.db -rw-r--r-- 1 ifc root 8288256 Nov 23 10:19 observer_7.db -rw-r--r-- 1 ifc root 8269824 Mar 8 09:10 observer_25.db -rw-r--r-- 1 ifc root 35417088 Mar 8 09:12 observer_28.db -rw-r--r-- 1 ifc root 27639808 Mar 8 09:12 observer_4.db -rw-r--r-- 1 ifc root 8925184 Mar 8 09:13 observer_16.db -rw-r--r-- 1 ifc root 8592384 Mar 8 09:14 observer_31.db -rw-r--r-- 1 ifc root 230865920 Mar 8 09:14 observer_22.db -rw-r--r-- 1 ifc root 8188928 Mar 8 09:14 observer_10.db -rw-r--r-- 1 ifc root 8385536 Mar 8 09:14 observer_13.db -rw-r--r-- 1 ifc root 8714240 Mar 8 09:14 observer_19.db -rw-r--r-- 1 ifc admin 8292352 Mar 8 09:14 observer_1.db -rw-r--r-- 1 ifc root 10083328 Mar 8 09:14 observer_255.db   (We don't have any impact on production but strange behavior)   Have a nice day,   Yoann   ... View more