Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
952
Views
3
Helpful
12
Replies

Ocassional uknown unicast flood on nexus 9k

Go to solution
enkisan
Level 1
Level 1

‎02-27-2024 11:12 AM

Hello community!

I have two 93180YC-EX in VPC, NX-OS mode:

 

BIOS: version 07.66
NXOS: version 7.0(3)I7(9)

 

 

Sometimes we experience unknown unicast flood for ~1-3min due to disappeared MAC. This situation arises every several hours randomly. At the time of flood i see errors on VPC primary:

 

 

# show system internal l2fm errors

1) Event:E_DEBUG, length:95, at 220748 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac 0050.566d.6c95, vlan 1230 is missing from macdb
2) Event:E_DEBUG, length:95, at 220729 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac 0025.90cd.b380, vlan 1230 is missing from macdb
3) Event:E_DEBUG, length:95, at 220709 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac 5254.00e0.2870, vlan 1230 is missing from macdb
4) Event:E_DEBUG, length:95, at 220678 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac 5254.006d.a281, vlan 1230 is missing from macdb
5) Event:E_DEBUG, length:95, at 220646 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac 0025.9033.061c, vlan 1230 is missing from macdb
6) Event:E_DEBUG, length:95, at 220628 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac 0050.5600.d660, vlan 1230 is missing from macdb
7) Event:E_DEBUG, length:95, at 220615 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac 0025.9065.efb0, vlan 1230 is missing from macdb
8) Event:E_DEBUG, length:95, at 220602 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac 000c.29db.4023, vlan 1230 is missing from macdb
9) Event:E_DEBUG, length:95, at 220584 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac fc34.9711.ab5e, vlan 1230 is missing from macdb
10) Event:E_DEBUG, length:95, at 220567 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_reset_ll_underway_for_req(4413): mac ac1f.6b48.b8a4, vlan 1230 is missing from macdb

 

 

and VPC secondary:

 

 

# show system internal l2fm errors

1) Event:E_DEBUG, length:118, at 824756 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x1600001b Vlan first 1230 last 1230
2) Event:E_DEBUG, length:137, at 824750 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x1600001b vlan 1230
3) Event:E_DEBUG, length:118, at 824718 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x16000013 Vlan first 1230 last 1230
4) Event:E_DEBUG, length:137, at 824711 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x16000013 vlan 1230
5) Event:E_DEBUG, length:118, at 824679 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x16000012 Vlan first 1230 last 1230
6) Event:E_DEBUG, length:137, at 824672 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x16000012 vlan 1230
7) Event:E_DEBUG, length:118, at 824638 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x16000011 Vlan first 1230 last 1230
8) Event:E_DEBUG, length:137, at 824632 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x16000011 vlan 1230
9) Event:E_DEBUG, length:118, at 824587 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x16000010 Vlan first 1230 last 1230
10) Event:E_DEBUG, length:137, at 824581 usecs after Tue Feb 27 21:01:47 2024
    [102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x16000010 vlan 1230

 

 

No particular MAC i could to highlight, always different. Didn't notice any MAC overflow also (same count on both switches):

 

 

# sh mac address-table count
Legend:
DLAC - Dynamic Local Address Count
DRAC - Dynamic Remote Address Count
SLAC - Static Local Address (User Defined) Count
SRAC - Static Remote Address (User Defined) Count
SAC - Secure Address Count

MAC Entries for all VLANS:
Dynamic Local Address Count:                                    6341
Dynamic Remote Address Count:                                      0
Static Remote Address (User-defined) Count:                        0
Static Local Address (User-defined) Count:                         0
Secure Address Count:                                              0
Total MAC Addresses in Use (DLAC + DRAC + SLAC + SRAC + SAC):   6341

 

 

Anyone point the direction please in which troubleshoot the root cause of this problem. Any help appreciated!

Labels:
0 Helpful
1 Accepted Solution

Accepted Solutions

Go to solution
enkisan
Level 1
Level 1
In response to MHM Cisco World

‎02-29-2024 11:02 AM - edited ‎02-29-2024 11:18 AM

I have found the problem. I forgot to enable portfast for several access switches connected to my VPC pair. Thus a STP TCN was occurring in the VLAN when these access ports upped or downed/upped causing rebuilding MAC table.

Thank again to all participants!

View solution in original post

0 Helpful
12 Replies 12

Go to solution
Reza Sharifi
Hall of Fame
Hall of Fame

‎02-27-2024 11:18 AM

Hi,

Do all the trunk ports including vPC peer-link (if you are using vPC) have the same vlans on both switches?

HTH

0 Helpful

Go to solution
enkisan
Level 1
Level 1
In response to Reza Sharifi

‎02-27-2024 11:28 AM

Trunk VPCs looks to access switches and have different allowed VLANs. Besides VPCs there are some orphan ports with own VLAN sets. Anyway i have noticed problem with only single (and most used) VLAN.

0 Helpful

Go to solution
Reza Sharifi
Hall of Fame
Hall of Fame
In response to enkisan

‎02-27-2024 12:01 PM

What is the output of "sh vpc" or "sh vpc det"?

HTH

0 Helpful

Go to solution
enkisan
Level 1
Level 1
In response to Reza Sharifi

‎02-27-2024 12:15 PM

primary:

# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : primary
Number of vPCs configured         : 26
Peer Gateway                      : Enabled
Dual-active excluded VLANs        : 6-9,172
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled, timer is off.(timeout = 240s)
Delay-restore status              : Timer is off.(timeout = 30s)
Delay-restore SVI status          : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router    : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id    Port   Status Active vlans
--    ----   ------ -------------------------------------------------
1     Po100  up     10,14,16,18-20,23,30-31,172,250,420,503,512,516,5
                    20,526,529,531,1202-1203,1210,1220,1222-1223,1230
                    ,1700-1900,2007,2053,2701-2706,3000

vPC status
----------------------------------------------------------------------------
Id    Port          Status Consistency Reason                Active vlans
--    ------------  ------ ----------- ------                ---------------
1     Po1           up     success     success               250,512,1220,1
                                                             230
2     Po2           up     success     success               18,250,1220,12
                                                             30,1813,1828,1
                                                             836,1847,1849,
                                                             1859,1869,1871
                                                             ,1875,2703
3     Po3           up     success     success               250,516,1220,1
                                                             230,1813,1839,
                                                             1873,2703
4     Po4           up     success     success               250,1220,1230,
                                                             1859
5     Po5           up     success     success               250,503,512,12
                                                             20,1230,1713-1
                                                             714,1813
6     Po6           up     success     success               250,1220,1230

7     Po7           up     success     success               250,503,1220,1
                                                             230
8     Po8           up     success     success               250,503,512,12
                                                             20,1230,1828
9     Po9           up     success     success               250,516,1220,1
                                                             230,1878,2706
10    Po10          up     success     success               10,14,16,18-20
                                                             ,23,30-31,250,
                                                             503,1210,1220,
                                                             1222-1223,1230
                                                             ,1700-1900,200
                                                             7,2053
11    Po11          up     success     success               250,1220,1230

12    Po12          up     success     success               250,1220,1230

13    Po13          up     success     success               250,1220,1230,
                                                             2702
14    Po14          up     success     success               250,503,512,52
                                                             9,531,1220,123
                                                             0
16    Po16          up     success     success               18,250,512,122
                                                             0,1230
17    Po17          up     success     success               250,503,1220,1
                                                             230
18    Po18          up     success     success               18,250,529,531
                                                             ,1220,1230
19    Po19          up     success     success               10,172,250,503
                                                             ,1220,1230
20    Po20          up     success     success               10,250,1220,12
                                                             30,3000
24    Po24          up     success     success               18,1220

26    Po26          up     success     success               18,1220

27    Po27          up     success     success               250,1230,2703

28    Po28          up     success     success               250,1220,1230,
                                                             1730
29    Po29          up     success     success               18,1220

44    Po44          up     success     success               10,14,16,18-20
                                                             ,23,30-31,172,
                                                             1210,1222,1700
                                                             -1900,2007,205
                                                             3,3000
50    Po50          up     success     success               18,250,512,526
                                                             ,1220,1230,170
                                                             9,1731,1735,18
                                                             25,2053,2704-2
                                                             706

Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.

secondary:

# sh vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : secondary
Number of vPCs configured         : 26
Peer Gateway                      : Enabled
Dual-active excluded VLANs        : 6-9,172
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled, timer is off.(timeout = 240s)
Delay-restore status              : Timer is off.(timeout = 30s)
Delay-restore SVI status          : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router    : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id    Port   Status Active vlans
--    ----   ------ -------------------------------------------------
1     Po100  up     10,14,16,18-20,23,30-31,172,250,420,503,512,516,5
                    20,526,529,531,1202-1203,1210,1220,1222-1223,1230
                    ,1700-1900,2007,2053,2701-2706,3000

vPC status
----------------------------------------------------------------------------
Id    Port          Status Consistency Reason                Active vlans
--    ------------  ------ ----------- ------                ---------------
1     Po1           up     success     success               250,512,1220,1
                                                             230
2     Po2           up     success     success               18,250,1220,12
                                                             30,1813,1828,1
                                                             836,1847,1849,
                                                             1859,1869,1871
                                                             ,1875,2703
3     Po3           up     success     success               250,516,1220,1
                                                             230,1813,1839,
                                                             1873,2703
4     Po4           up     success     success               250,1220,1230,
                                                             1859
5     Po5           up     success     success               250,503,512,12
                                                             20,1230,1713-1
                                                             714,1813
6     Po6           up     success     success               250,1220,1230

7     Po7           up     success     success               250,503,1220,1
                                                             230
8     Po8           up     success     success               250,503,512,12
                                                             20,1230,1828
9     Po9           up     success     success               250,516,1220,1
                                                             230,1878,2706
10    Po10          up     success     success               10,14,16,18-20
                                                             ,23,30-31,250,
                                                             503,1210,1220,
                                                             1222-1223,1230
                                                             ,1700-1900,200
                                                             7,2053
11    Po11          up     success     success               250,1220,1230

12    Po12          up     success     success               250,1220,1230

13    Po13          up     success     success               250,1220,1230,
                                                             2702
14    Po14          up     success     success               250,503,512,52
                                                             9,531,1220,123
                                                             0
16    Po16          up     success     success               18,250,512,122
                                                             0,1230
17    Po17          up     success     success               250,503,1220,1
                                                             230
18    Po18          up     success     success               18,250,529,531
                                                             ,1220,1230
19    Po19          up     success     success               10,172,250,503
                                                             ,1220,1230
20    Po20          up     success     success               10,250,1220,12
                                                             30,3000
24    Po24          up     success     success               18,1220

26    Po26          up     success     success               18,1220

27    Po27          up     success     success               250,1230,2703

28    Po28          up     success     success               250,1220,1230,
                                                             1730
29    Po29          up     success     success               18,1220

44    Po44          up     success     success               10,14,16,18-20
                                                             ,23,30-31,172,
                                                             1210,1222,1700
                                                             -1900,2007,205
                                                             3,3000
50    Po50          up     success     success               18,250,512,526
                                                             ,1220,1230,170
                                                             9,1731,1735,18
                                                             25,2053,2704-2
                                                             706

Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.
0 Helpful

Go to solution
Reza Sharifi
Hall of Fame
Hall of Fame
In response to enkisan

‎02-27-2024 12:23 PM

vPC peer-link looks good. Is there any error in the output of "show vpc consist"?

HTH

0 Helpful

Go to solution
enkisan
Level 1
Level 1
In response to Reza Sharifi

‎02-27-2024 12:38 PM

nothing suspicious here also:

 

# show vpc consistency-parameters vpc 1

    Legend:
        Type 1 : vPC will be suspended in case of mismatch

Name                        Type  Local Value            Peer Value
-------------               ----  ---------------------- -----------------------
delayed-lacp                1     disabled               disabled
mode                        1     active                 active
Switchport Isolated         1     0                      0
Interface type              1     port-channel           port-channel
LACP Mode                   1     on                     on
Virtual-ethernet-bridge     1     Disabled               Disabled
Speed                       1     10 Gb/s                10 Gb/s
Duplex                      1     full                   full
MTU                         1     1500                   1500
Port Mode                   1     trunk                  trunk
Native Vlan                 1     1                      1
Admin port mode             1     trunk                  trunk
STP Port Guard              1     Default                Default
STP Port Type               1     Default                Default
STP MST Simulate PVST       1     Default                Default
lag-id                      1     [(2000,                [(2000,
                                  0-23-4-ee-be-1, 8001,  0-23-4-ee-be-1, 8001,
                                  0, 0), (8000,          0, 0), (8000,
                                  44-e4-d9-36-85-0, 1,   44-e4-d9-36-85-0, 1,
                                  0, 0)]                 0, 0)]
Allow-Multi-Tag             1     Disabled               Disabled
Vlan xlt mapping            1     Disabled               Disabled
vPC card type               1     N9K TOR                N9K TOR
Allowed VLANs               -     250,510,512,1220,1230  250,510,512,1220,1230
Local suspended VLANs       -     -                      -

# show vpc consistency-parameters global

    Legend:
        Type 1 : vPC will be suspended in case of mismatch

Name                        Type  Local Value            Peer Value
-------------               ----  ---------------------- -----------------------
STP MST Simulate PVST       1     Enabled                Enabled
STP Port Type, Edge         1     Normal, Disabled,      Normal, Disabled,
BPDUFilter, Edge BPDUGuard        Disabled               Disabled
STP MST Region Name         1     ""                     ""
STP Disabled                1     None                   None
STP Mode                    1     Rapid-PVST             Rapid-PVST
STP Bridge Assurance        1     Enabled                Enabled
STP Loopguard               1     Disabled               Disabled
STP MST Region Instance to  1
 VLAN Mapping
STP MST Region Revision     1     0                      0
Interface-vlan admin up     2     172,250,526,1220,1223, 172,250,526,1220,1223,
                                  1230,2701-2706         1230,2701-2706
Interface-vlan routing      2     172,250,526,1220,1223, 172,250,526,1220,1223,
capability                        1230,2701-2706         1230,2701-2706
QoS (Cos)                   2     ([0-7], [], [], [],    ([0-7], [], [], [],
                                  [], [])                [], [])
Network QoS (MTU)           2     (9216, 0, 0, 0, 0,     (9216, 0, 0, 0, 0,
                                  9216)                  9216)
Network Qos (Pause:         2     (F, F, F, F, F, T)     (F, F, F, F, F, T)
T->Enabled, F->Disabled)
Input Queuing (Bandwidth)   2     (0, 0, 0, 0, 0, 0)     (0, 0, 0, 0, 0, 0)
Input Queuing (Absolute     2     (F, F, F, F, F, F)     (F, F, F, F, F, F)
Priority: T->Enabled,
F->Disabled)
Output Queuing (Bandwidth   2     (0, 0, 0, 0, 0, 0)     (0, 0, 0, 0, 0, 0)
Remaining)
Output Queuing (Absolute    2     (T, F, F, F, F, F)     (T, F, F, F, F, F)
Priority: T->Enabled,
F->Disabled)
Allowed VLANs               -     10,14,16,18-20,23,30-3 10,14,16,18-20,23,30-3
                                  1,172,250,420,503,510, 1,172,250,420,503,510,
                                  512,516,520,526,529,53 512,516,520,526,529,53
                                  1,1202-1203,1210,1220, 1,1202-1203,1210,1220,
                                  1222-1223,1230,1700-19 1222-1223,1230,1700-19
                                  00,2007,2053,2701-2706 00,2007,2053,2701-2706
                                  ,3000                  ,3000
Local suspended VLANs       -     -                      -

 

 

 
 

 

 

0 Helpful

Go to solution
MHM Cisco World
VIP
VIP
In response to enkisan

‎02-27-2024 12:45 PM

There is nothing wrong in your config but I think it is not optimal. 

Unknown flood happened when SW missing mac in table, so first point we must check it is this mac connect to orphan, are this Mac appear in both or one NSK or not appear in both NSK. 

One of case of flood

as role the Arp aging must be less than Mac aging  to eliminate the unknown flood, arp in host try to connect to other host will aging this make host send broadcast ask Mac, if maç aging before arp the host not send arp it send packet with Mac address unknown for both nsk and make both nsk flood this mac

So as I mentioned above first point check maç address shown in log

MHM

Go to solution
enkisan
Level 1
Level 1
In response to MHM Cisco World

‎02-27-2024 01:23 PM

Thanks for your suggestions. I'll investigate the MAC situation further and keep you updated. As for aging, i have defaults on both switches: 1800 (MAC) > 1500 (ARP), so there shouldn't be issues, i think.

Go to solution
enkisan
Level 1
Level 1
In response to MHM Cisco World

‎02-29-2024 11:02 AM - edited ‎02-29-2024 11:18 AM

I have found the problem. I forgot to enable portfast for several access switches connected to my VPC pair. Thus a STP TCN was occurring in the VLAN when these access ports upped or downed/upped causing rebuilding MAC table.

Thank again to all participants!

0 Helpful

Go to solution
MHM Cisco World
VIP
VIP
In response to enkisan

‎02-29-2024 11:08 AM

Thanks for update me

Have a nice day 

MHM

0 Helpful

Go to solution
MHM Cisco World
VIP
VIP

‎02-27-2024 11:21 AM

Did you use 

Ip arp sync

MHM

0 Helpful

Go to solution
enkisan
Level 1
Level 1
In response to MHM Cisco World

‎02-27-2024 11:29 AM

'ip arp synchronize' is active within VPC domain

Customers Also Viewed These Support Documents
Review Cisco Networking for a $25 gift card