04-06-2021 01:42 AM
Hello,
We have some promlems with Cisco Nexus 3064 switch.
Version information
Software BIOS: version 4.1.0 NXOS: version 7.0(3)I6(2) BIOS compile time: 02/02/2017 NXOS image file is: bootflash:///nxos.7.0.3.I6.2.bin NXOS compile time: 10/17/2017 19:00:00 [10/18/2017 06:48:10] Hardware cisco Nexus3064 Chassis Intel(R) Celeron(R) CPU P4505 @ 1.87GHz with 3903096 kB of memory. Processor Board ID FOC1634EK8W
Symptoms:
1. The following entries appear in the log every second:
2021 Apr 6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_HASH_COLLISION: L2 ENTRY unit=0 mac=0c:80:63:b5:a5:5b vlan=3624 port=0x0c000003 2021 Apr 6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_HASH_COLLISION: L2 ENTRY unit=0 mac=50:ff:20:4b:75:30 vlan=3682 port=0x08000833 2021 Apr 6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_HASH_COLLISION: L2 ENTRY unit=0 mac=0c:b6:d2:8b:0e:a8 vlan=3585 port=0x08000821 2021 Apr 6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0 2021 Apr 6 11:10:55 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0 2021 Apr 6 11:10:55 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0 2021 Apr 6 11:10:56 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0 2021 Apr 6 11:10:56 Izhevsk20 %MTM-SLOT1-2-MTM_BUFFERS_FULL: MTM buffers are full for unit 0. MAC tables might be inconsistent. Pls use l2 consistency-checker to verify. ...
2. There are very few entries in the MAC table with about 40,000 entries in reality:
# show mac address-table count MAC Entries for all vlans : Dynamic Address Count: 2656 Overlay Address Count: 0 Static Address (User-defined) Count: 0 Secure Address Count: 0
3. High CPU load:
# show system resources Load average: 1 minute: 4.23 5 minutes: 4.25 15 minutes: 4.32 Processes : 480 total, 5 running CPU states : 71.92% user, 14.77% kernel, 13.30% idle CPU0 states : 99.00% user, 0.00% kernel, 1.00% idle CPU1 states : 45.19% user, 30.76% kernel, 24.03% idle Memory usage: 3903096K total, 2779640K used, 1123456K free Current memory status: OK
4. L2 consistency checker FAILED:
Consistency check: FAILED Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen, + - primary entry using vPC Peer-Link, (T) - True, (F) - False Missing entries in the HW MAC Table VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ * 19 0024.1db2.d141 dynamic 0 F F Eth1/51 * 19 0e50.bbbf.d9b1 dynamic 0 F F Eth1/51 * 19 848a.8d2e.b084 dynamic 0 F F Eth1/51 * 54 0018.ae31.bc3d dynamic 0 F F Eth1/51 * 54 0018.ae3a.daaf dynamic 0 F F Eth1/51 * 54 3c97.0ebe.c003 dynamic 0 F F Eth1/51 * 54 408d.5cfe.66c3 dynamic 0 F F Eth1/51 * 54 b42e.992b.ee7b dynamic 0 F F Eth1/51 ...
(many hundreds of lines).
5. After reboot the switch ploblem instantly starts over again.
6. According to the table from https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/scalability/703I71/b_Cisco_Nexus_3000_Series_NXOS_Verified_Scalability_Guide_703I71.html the device accepts 128k mac-addresses.
7. vPC is disabled:
# show feature | inc vpc vpc 1 disabled
How to solve this prombem?
Solved! Go to Solution.
04-06-2021 06:58 AM
I found the solution. One of the clients was sending a pvst packet like root change every 30 seconds. It caused MAC Learning Disabled and then MAC Learning Enabled. The MAC table did not have time to fill up until the next pvst packet.
I have filtered pvst packets from this client's interface and everything became fine.
04-06-2021 02:19 AM
- This bug report could be related : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg96323
In general , use latest advisory software release for the particular n3k and check if the problem persists afterwards.
M.
04-06-2021 05:09 AM
I think we have a different case. We have not reached the capacity limit. On the contrary, we cannot fill the mac-table. I think the key point is this:
2021 Apr 6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0 2021 Apr 6 11:10:55 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0 2021 Apr 6 11:10:55 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0 2021 Apr 6 11:10:56 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
04-06-2021 05:51 AM
- You may suffer from a forwarding loop or a malicious device inducing a network storm. Perhaps network traffic at the particular ports could be monitored and analyzed , also check this document :
https://isatisserver.com/index.php/article/item/download/66_ad3841de8c834e9dbdf4df2b9f605b98
The particular message is described in that book and explained.
M.
04-06-2021 06:58 AM
I found the solution. One of the clients was sending a pvst packet like root change every 30 seconds. It caused MAC Learning Disabled and then MAC Learning Enabled. The MAC table did not have time to fill up until the next pvst packet.
I have filtered pvst packets from this client's interface and everything became fine.
11-26-2023 02:32 AM
I am also getting same . How do you filtered pvst would please give a example
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide