Nexus 9K doesn't age out dynamic mac entries

[EdvinasKairys11982]

Hello,

We got couple of N9K's - N9K-C9372PX-E  7.0(3)I7(8)

For some reason they doesn't age out old MAC entries. Where're no custom configurations, the age time is left a default:

Switch# show mac address-table aging-time
Aging Time
----------
1800

Dynamic Local Address Count: 48137

Because of that - our mac address table has grown to 48K, which i think is quite near limits.

ARP entries ages out successfully - after about 25mins, but the MAC address is hanging here for more than 3 hours already. 

Switch# show mac address-table address fa16.3e8b.3b4d
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
+ 2232 fa16.3e8b.3b4d dynamic 0 F F Po3707

If i delete it manually - ir never reappears, what proves that it's really inactive.

Maybe you have some advices ? 

Tried to search for a bug - but not successful. There's no official bug reported.

[balaji.bandi]

is tha MAC address are unique one ?

try to lowere the age time - mac address-table aging-time 120

[EdvinasKairys11982]

yes, the addresses are uniq ones.

Hmm, i think it's not good practice to make it lower than the arp-cache timeout which is  1500 secs. But i think i can try to lower mac addres ageing from 1800 to 1700 and see if it takes effect.

Any info about effect of changing aging time ? will it flush all mac addresses ?

[balaji.bandi]

I do not believe it flush any MAC it lower the value

[MHM Cisco World]

there is any vPC ?

[EdvinasKairys11982]

Yes it's in VPC straight through topology which consists of two Nexuses. Do you think it's somehow related to that ?

It's strange, we have other switches in other VPC domains and they doesnt have this glitch. We have even the same model, but different software version of switches in other domain and everything is OK.

Last year we upgraded these two switches 7.0(3)I7(8), but as i remember this problem was before the upgrade also.

Any ideas ?

[balaji.bandi]

Is the same MAC List on another side ? (both the side same MAC count ?)

i forgot to ask in the first instance, is this MAC only for on certain VLAN 

[EdvinasKairys11982]

Yes, both sides has the same problem and the same mac count.

[MHM Cisco World]

...

check this bug 

[EdvinasKairys11982]

Thanks, as i understamd this is just a 'cosmetic' bug.

I havr other pair of devices where age is shown as zero, but it ages out mac addresses correctly.

[MHM Cisco World]

So for other this bug is useful but for this pair the issue is different?

[EdvinasKairys11982]

@MHM Cisco World This bug, like described, is doesn't impact actual behaviour - it just show zero, But it shouldnt ipact the actual behaviour of ageing out entries. 

And about 'Inactivity/Absolute' timers. I see it's a port-security options. We're not using port-security at all. That feature is not even enabled on our Nexus switches. Basic mac learning/ageing out should work without any features like port-security...

Thanks !

[MHM Cisco World]

.....

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/nx-os-software/213906-nexus-9000-mac-move-troubleshooting-and.html

I do some search about this issue and find the above issue about N9K with fast MAC movement, 
what make me think about this frozen Mac address in table is aged is equal to Zero, so after we exclude the port-security and find this issue and how to detect it. 

hope this help you.

[EdvinasKairys11982]

thanks again for your effotr i appreciate that !

I see that doc is mainly talking about mac-move thing. But i'm quite sure that the mac's isn't moving between interfaces. Anyhow. tried to enable deeper level of l2 logging - changed it to 5. Also doc says that if the nexus notices "too many mac-addresses in a short duration" it can disable learning. This is quite interesting as I know - that interfaces can 'spit out' about ~250macs instantly. Im finding hard time to relate that to mac ageing out as the whole topic is not about learning, it's more about ageing entries out.. will try to see if i will have some new logs.

Also i was able to debug the mac address to these commands:

SwitchA#   show system internal l2fm l2dbg macdb address fa16.3e8b.3b4d
Legend
------
Db:  0-MACDB, 1-GWMACDB, 2-SMACDB, 3-RMDB,    4-SECMACDB  5-STAGEDB
Src: 0-UNKNOWN, 1-L2FM, 2-PEER, 3-LC, 4-HSRP
     5-GLBP, 6-VRRP, 7-STP, 8-DOTX, 9-PSEC 10-CLI 11-PVLAN
     12-ETHPM, 13-ALW_LRN, 14-Non_PI_MOD, 15-MCT_DOWN, 16 - SDB
     17-OTV, 18-Deounce Timer, 19-AM, 20-PCM_DOWN, 21 - MCT_UP
     22-VxLAN, 23-L2RIB 24-CTRL, 25-UFDM
Slot:0 based for LCS 31-MCEC 20-OTV/ORIB 

 VLAN: 2232 MAC: fa16.3e8b.3b4d FE ID: 0
    Time                     If/swid    Db   Op                 Src Slot FE
    Fri Jul 16 13:06:36 2021 0x16000e7a 0  AGE                  3   0    0   

 VLAN: 2232 MAC: fa16.3e8b.3b4d
    Time                     If/swid    Db   Op                 Src Slot FE
    Fri Jul 16 12:13:46 2021 0x16000e7a 0  INSERT               3   31   0   
    Fri Jul 16 12:13:46 2021 0x16000e7a 0  INSERT               2   0    15  
    Fri Jul 16 12:13:46 2021 0x16000e7a 0  MAC_NOTIF_AM_MOVE    1   0    15  
    Fri Jul 16 12:13:46 2021 0x16000e7a 0  UPDATE               3   0    0   
    Fri Jul 16 12:13:46 2021 0x16000e7a 0  REFRESH_DETECT       3   0    15  
    Fri Jul 16 12:13:46 2021 0x7e000e7b 0  RESET_LL_UNDERWAY    2   0    15  
    Fri Jul 16 13:06:36 2021 0x16000e7a 0  SET_LOCAL_AGE        3   0    15  
    Fri Jul 16 13:06:36 2021 0x7e000e7b 0  SEND_AGE_TO_PEER     1   0    15  
SwitchA# show system internal l2fm event-history debugs | include fa16.3e8b.3b4d
    [104] l2fm_handle_mtm_age_notfn(9647): Add entry to age notification to be sent to vPC peer. count=1, mac fa16.3e8b.3b4d vlan 2232 intf 0x7e000e7b age 0 flags 4194567
    [104] l2fm_mac_regist_remove_entry(5992): Received request to remove entry from Delete reg db is_reg: 0 for MAC fa16.3e8b.3b4d
    [104] l2fm_handle_mac_move_generic_l2_entry(14820): Ignoring entry if_index 0x16000e7a, vl 2232 mac fa16.3e8b.3b4d state 3
    [104] l2fm_macdb_insert(7501): slot 0 fe 0 mac fa16.3e8b.3b4d vlan 2232 flags 0x400107 hints 0 E8 NL lc  : if_index 0x16000e7a old_if_index 0
    [104] l2fm_mcec_rmdb_delete(232): Trying to delete an entry not present in RMDB  fa16.3e8b.3b4d
    [104] l2fm_mcec_rmdb_delete(217): Deleting MAC fa16.3e8b.3b4d vlan 2232 from RMDB
    [104] l2fm_send_ntfn_to_am(15689): Sending old_index = 0x0, new_index = 0x16000e7a vlan: 2232 mac: fa16.3e8b.3b4d is_del: 0 
    [104] l2fm_mac_regist_add_entry(5569): Adding node to delete registration database is_reg: 1 immed_notif: 1 MAC: fa16.3e8b.3b4d, ifindex: 0x90108b8, phy ifindex: 0x16000e7a 
    [104] l2fm_macdb_insert(7501): slot 31 fe 0 mac fa16.3e8b.3b4d vlan 2232 flags 0x2400107 hints 0 E8 NL  : if_index 0x16000e7a old_if_index 0