Cisco Nexius C3064PQ MAC Table problem with log "MTM buffers are full for unit 0"

[waspagv]

Hello, 

We have some promlems with Cisco Nexus 3064 switch.

Version information

Software
  BIOS: version 4.1.0
  NXOS: version 7.0(3)I6(2)
  BIOS compile time:  02/02/2017
  NXOS image file is: bootflash:///nxos.7.0.3.I6.2.bin
  NXOS compile time:  10/17/2017 19:00:00 [10/18/2017 06:48:10]


Hardware
  cisco Nexus3064 Chassis
  Intel(R) Celeron(R) CPU        P4505  @ 1.87GHz with 3903096 kB of memory.
  Processor Board ID FOC1634EK8W

Symptoms:

1. The following entries appear in the log every second:

2021 Apr  6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_HASH_COLLISION: L2 ENTRY unit=0 mac=0c:80:63:b5:a5:5b vlan=3624 port=0x0c000003
2021 Apr  6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_HASH_COLLISION: L2 ENTRY unit=0 mac=50:ff:20:4b:75:30 vlan=3682 port=0x08000833
2021 Apr  6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_HASH_COLLISION: L2 ENTRY unit=0 mac=0c:b6:d2:8b:0e:a8 vlan=3585 port=0x08000821
2021 Apr  6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2021 Apr  6 11:10:55 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2021 Apr  6 11:10:55 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2021 Apr  6 11:10:56 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2021 Apr  6 11:10:56 Izhevsk20 %MTM-SLOT1-2-MTM_BUFFERS_FULL: MTM buffers are full for unit 0. MAC tables might be inconsistent. Pls use l2 consistency-checker to verify.
...

2. There are very few entries in the MAC table with about 40,000 entries in reality:

# show mac address-table count
MAC Entries for all vlans :
Dynamic Address Count:                 2656
Overlay Address Count:                    0
Static Address (User-defined) Count:       0
Secure Address Count:                      0

3. High CPU load:

# show system resources
Load average:   1 minute: 4.23   5 minutes: 4.25   15 minutes: 4.32
Processes   :   480 total, 5 running
CPU states  :   71.92% user,   14.77% kernel,   13.30% idle
        CPU0 states  :   99.00% user,   0.00% kernel,   1.00% idle
        CPU1 states  :   45.19% user,   30.76% kernel,   24.03% idle
Memory usage:   3903096K total,   2779640K used,   1123456K free
Current memory status: OK

4. L2 consistency checker FAILED:

Consistency check: FAILED
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen, + - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
Missing entries in the HW MAC Table
   VLAN     MAC Address      Type      age     Secure NTFY    Ports
---------+-----------------+--------+---------+------+----+------------------
* 19       0024.1db2.d141    dynamic   0          F    F  Eth1/51
* 19       0e50.bbbf.d9b1    dynamic   0          F    F  Eth1/51
* 19       848a.8d2e.b084    dynamic   0          F    F  Eth1/51
* 54       0018.ae31.bc3d    dynamic   0          F    F  Eth1/51
* 54       0018.ae3a.daaf    dynamic   0          F    F  Eth1/51
* 54       3c97.0ebe.c003    dynamic   0          F    F  Eth1/51
* 54       408d.5cfe.66c3    dynamic   0          F    F  Eth1/51
* 54       b42e.992b.ee7b    dynamic   0          F    F  Eth1/51
...

(many hundreds of lines).

5. After reboot the switch ploblem instantly starts over again.

6. According to the table from https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/scalability/703I71/b_Cisco_Nexus_3000_Series_NXOS_Verified_Scalability_Guide_703I71.html the device accepts 128k mac-addresses.

7. vPC is disabled:

# show feature | inc vpc
vpc                    1          disabled

How to solve this prombem?

[marce1000]

           -  This bug report could be related : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg96323

  In general , use latest advisory software release for the particular n3k and check if the problem persists afterwards.

 M.

[waspagv]

I think we have a different case. We have not reached the capacity limit. On the contrary, we cannot fill the mac-table. I think the key point is this:

2021 Apr  6 11:10:53 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2021 Apr  6 11:10:55 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2021 Apr  6 11:10:55 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2021 Apr  6 11:10:56 Izhevsk20 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0

[marce1000]

 - You may suffer from a forwarding loop or a malicious device inducing a network storm. Perhaps network traffic at the particular ports could be monitored and analyzed , also check this document :

                     https://isatisserver.com/index.php/article/item/download/66_ad3841de8c834e9dbdf4df2b9f605b98

 The particular message is described in that book and explained.

 M.

[waspagv · Accepted solution]

I found the solution. One of the clients was sending a pvst packet like root change every 30 seconds. It caused MAC Learning Disabled and then MAC Learning Enabled. The MAC table did not have time to fill up until the next pvst packet.

I have filtered pvst packets from this client's interface and everything became fine.

[aguntukkk]

I am also getting same . How do you filtered pvst would please give a example