02-27-2024 11:12 AM
Hello community!
I have two 93180YC-EX in VPC, NX-OS mode:
BIOS: version 07.66
NXOS: version 7.0(3)I7(9)
Sometimes we experience unknown unicast flood for ~1-3min due to disappeared MAC. This situation arises every several hours randomly. At the time of flood i see errors on VPC primary:
# show system internal l2fm errors
1) Event:E_DEBUG, length:95, at 220748 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac 0050.566d.6c95, vlan 1230 is missing from macdb
2) Event:E_DEBUG, length:95, at 220729 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac 0025.90cd.b380, vlan 1230 is missing from macdb
3) Event:E_DEBUG, length:95, at 220709 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac 5254.00e0.2870, vlan 1230 is missing from macdb
4) Event:E_DEBUG, length:95, at 220678 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac 5254.006d.a281, vlan 1230 is missing from macdb
5) Event:E_DEBUG, length:95, at 220646 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac 0025.9033.061c, vlan 1230 is missing from macdb
6) Event:E_DEBUG, length:95, at 220628 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac 0050.5600.d660, vlan 1230 is missing from macdb
7) Event:E_DEBUG, length:95, at 220615 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac 0025.9065.efb0, vlan 1230 is missing from macdb
8) Event:E_DEBUG, length:95, at 220602 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac 000c.29db.4023, vlan 1230 is missing from macdb
9) Event:E_DEBUG, length:95, at 220584 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac fc34.9711.ab5e, vlan 1230 is missing from macdb
10) Event:E_DEBUG, length:95, at 220567 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_reset_ll_underway_for_req(4413): mac ac1f.6b48.b8a4, vlan 1230 is missing from macdb
and VPC secondary:
# show system internal l2fm errors
1) Event:E_DEBUG, length:118, at 824756 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x1600001b Vlan first 1230 last 1230
2) Event:E_DEBUG, length:137, at 824750 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x1600001b vlan 1230
3) Event:E_DEBUG, length:118, at 824718 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x16000013 Vlan first 1230 last 1230
4) Event:E_DEBUG, length:137, at 824711 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x16000013 vlan 1230
5) Event:E_DEBUG, length:118, at 824679 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x16000012 Vlan first 1230 last 1230
6) Event:E_DEBUG, length:137, at 824672 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x16000012 vlan 1230
7) Event:E_DEBUG, length:118, at 824638 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x16000011 Vlan first 1230 last 1230
8) Event:E_DEBUG, length:137, at 824632 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x16000011 vlan 1230
9) Event:E_DEBUG, length:118, at 824587 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_walk_flush_req_reload_ps(1905): Requested Reload of MACs on Intf 0x16000010 Vlan first 1230 last 1230
10) Event:E_DEBUG, length:137, at 824581 usecs after Tue Feb 27 21:01:47 2024
[102] l2fm_l2rib_reload_peersync_macs_on_vlan_port(1846): Successfully sent reload request for Peer Synced MACs on PO 0x16000010 vlan 1230
No particular MAC i could to highlight, always different. Didn't notice any MAC overflow also (same count on both switches):
# sh mac address-table count
Legend:
DLAC - Dynamic Local Address Count
DRAC - Dynamic Remote Address Count
SLAC - Static Local Address (User Defined) Count
SRAC - Static Remote Address (User Defined) Count
SAC - Secure Address Count
MAC Entries for all VLANS:
Dynamic Local Address Count: 6341
Dynamic Remote Address Count: 0
Static Remote Address (User-defined) Count: 0
Static Local Address (User-defined) Count: 0
Secure Address Count: 0
Total MAC Addresses in Use (DLAC + DRAC + SLAC + SRAC + SAC): 6341
Anyone point the direction please in which troubleshoot the root cause of this problem. Any help appreciated!
Solved! Go to Solution.
02-29-2024 11:02 AM - edited 02-29-2024 11:18 AM
I have found the problem. I forgot to enable portfast for several access switches connected to my VPC pair. Thus a STP TCN was occurring in the VLAN when these access ports upped or downed/upped causing rebuilding MAC table.
Thank again to all participants!
02-27-2024 11:18 AM
Hi,
Do all the trunk ports including vPC peer-link (if you are using vPC) have the same vlans on both switches?
HTH
02-27-2024 11:28 AM
Trunk VPCs looks to access switches and have different allowed VLANs. Besides VPCs there are some orphan ports with own VLAN sets. Anyway i have noticed problem with only single (and most used) VLAN.
02-27-2024 12:01 PM
What is the output of "sh vpc" or "sh vpc det"?
HTH
02-27-2024 12:15 PM
primary:
# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 26
Peer Gateway : Enabled
Dual-active excluded VLANs : 6-9,172
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po100 up 10,14,16,18-20,23,30-31,172,250,420,503,512,516,5
20,526,529,531,1202-1203,1210,1220,1222-1223,1230
,1700-1900,2007,2053,2701-2706,3000
vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
1 Po1 up success success 250,512,1220,1
230
2 Po2 up success success 18,250,1220,12
30,1813,1828,1
836,1847,1849,
1859,1869,1871
,1875,2703
3 Po3 up success success 250,516,1220,1
230,1813,1839,
1873,2703
4 Po4 up success success 250,1220,1230,
1859
5 Po5 up success success 250,503,512,12
20,1230,1713-1
714,1813
6 Po6 up success success 250,1220,1230
7 Po7 up success success 250,503,1220,1
230
8 Po8 up success success 250,503,512,12
20,1230,1828
9 Po9 up success success 250,516,1220,1
230,1878,2706
10 Po10 up success success 10,14,16,18-20
,23,30-31,250,
503,1210,1220,
1222-1223,1230
,1700-1900,200
7,2053
11 Po11 up success success 250,1220,1230
12 Po12 up success success 250,1220,1230
13 Po13 up success success 250,1220,1230,
2702
14 Po14 up success success 250,503,512,52
9,531,1220,123
0
16 Po16 up success success 18,250,512,122
0,1230
17 Po17 up success success 250,503,1220,1
230
18 Po18 up success success 18,250,529,531
,1220,1230
19 Po19 up success success 10,172,250,503
,1220,1230
20 Po20 up success success 10,250,1220,12
30,3000
24 Po24 up success success 18,1220
26 Po26 up success success 18,1220
27 Po27 up success success 250,1230,2703
28 Po28 up success success 250,1220,1230,
1730
29 Po29 up success success 18,1220
44 Po44 up success success 10,14,16,18-20
,23,30-31,172,
1210,1222,1700
-1900,2007,205
3,3000
50 Po50 up success success 18,250,512,526
,1220,1230,170
9,1731,1735,18
25,2053,2704-2
706
Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.
secondary:
# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 26
Peer Gateway : Enabled
Dual-active excluded VLANs : 6-9,172
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po100 up 10,14,16,18-20,23,30-31,172,250,420,503,512,516,5
20,526,529,531,1202-1203,1210,1220,1222-1223,1230
,1700-1900,2007,2053,2701-2706,3000
vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
1 Po1 up success success 250,512,1220,1
230
2 Po2 up success success 18,250,1220,12
30,1813,1828,1
836,1847,1849,
1859,1869,1871
,1875,2703
3 Po3 up success success 250,516,1220,1
230,1813,1839,
1873,2703
4 Po4 up success success 250,1220,1230,
1859
5 Po5 up success success 250,503,512,12
20,1230,1713-1
714,1813
6 Po6 up success success 250,1220,1230
7 Po7 up success success 250,503,1220,1
230
8 Po8 up success success 250,503,512,12
20,1230,1828
9 Po9 up success success 250,516,1220,1
230,1878,2706
10 Po10 up success success 10,14,16,18-20
,23,30-31,250,
503,1210,1220,
1222-1223,1230
,1700-1900,200
7,2053
11 Po11 up success success 250,1220,1230
12 Po12 up success success 250,1220,1230
13 Po13 up success success 250,1220,1230,
2702
14 Po14 up success success 250,503,512,52
9,531,1220,123
0
16 Po16 up success success 18,250,512,122
0,1230
17 Po17 up success success 250,503,1220,1
230
18 Po18 up success success 18,250,529,531
,1220,1230
19 Po19 up success success 10,172,250,503
,1220,1230
20 Po20 up success success 10,250,1220,12
30,3000
24 Po24 up success success 18,1220
26 Po26 up success success 18,1220
27 Po27 up success success 250,1230,2703
28 Po28 up success success 250,1220,1230,
1730
29 Po29 up success success 18,1220
44 Po44 up success success 10,14,16,18-20
,23,30-31,172,
1210,1222,1700
-1900,2007,205
3,3000
50 Po50 up success success 18,250,512,526
,1220,1230,170
9,1731,1735,18
25,2053,2704-2
706
Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.
02-27-2024 12:23 PM
vPC peer-link looks good. Is there any error in the output of "show vpc consist"?
HTH
02-27-2024 12:38 PM
nothing suspicious here also:
# show vpc consistency-parameters vpc 1
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name Type Local Value Peer Value
------------- ---- ---------------------- -----------------------
delayed-lacp 1 disabled disabled
mode 1 active active
Switchport Isolated 1 0 0
Interface type 1 port-channel port-channel
LACP Mode 1 on on
Virtual-ethernet-bridge 1 Disabled Disabled
Speed 1 10 Gb/s 10 Gb/s
Duplex 1 full full
MTU 1 1500 1500
Port Mode 1 trunk trunk
Native Vlan 1 1 1
Admin port mode 1 trunk trunk
STP Port Guard 1 Default Default
STP Port Type 1 Default Default
STP MST Simulate PVST 1 Default Default
lag-id 1 [(2000, [(2000,
0-23-4-ee-be-1, 8001, 0-23-4-ee-be-1, 8001,
0, 0), (8000, 0, 0), (8000,
44-e4-d9-36-85-0, 1, 44-e4-d9-36-85-0, 1,
0, 0)] 0, 0)]
Allow-Multi-Tag 1 Disabled Disabled
Vlan xlt mapping 1 Disabled Disabled
vPC card type 1 N9K TOR N9K TOR
Allowed VLANs - 250,510,512,1220,1230 250,510,512,1220,1230
Local suspended VLANs - - -
# show vpc consistency-parameters global
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name Type Local Value Peer Value
------------- ---- ---------------------- -----------------------
STP MST Simulate PVST 1 Enabled Enabled
STP Port Type, Edge 1 Normal, Disabled, Normal, Disabled,
BPDUFilter, Edge BPDUGuard Disabled Disabled
STP MST Region Name 1 "" ""
STP Disabled 1 None None
STP Mode 1 Rapid-PVST Rapid-PVST
STP Bridge Assurance 1 Enabled Enabled
STP Loopguard 1 Disabled Disabled
STP MST Region Instance to 1
VLAN Mapping
STP MST Region Revision 1 0 0
Interface-vlan admin up 2 172,250,526,1220,1223, 172,250,526,1220,1223,
1230,2701-2706 1230,2701-2706
Interface-vlan routing 2 172,250,526,1220,1223, 172,250,526,1220,1223,
capability 1230,2701-2706 1230,2701-2706
QoS (Cos) 2 ([0-7], [], [], [], ([0-7], [], [], [],
[], []) [], [])
Network QoS (MTU) 2 (9216, 0, 0, 0, 0, (9216, 0, 0, 0, 0,
9216) 9216)
Network Qos (Pause: 2 (F, F, F, F, F, T) (F, F, F, F, F, T)
T->Enabled, F->Disabled)
Input Queuing (Bandwidth) 2 (0, 0, 0, 0, 0, 0) (0, 0, 0, 0, 0, 0)
Input Queuing (Absolute 2 (F, F, F, F, F, F) (F, F, F, F, F, F)
Priority: T->Enabled,
F->Disabled)
Output Queuing (Bandwidth 2 (0, 0, 0, 0, 0, 0) (0, 0, 0, 0, 0, 0)
Remaining)
Output Queuing (Absolute 2 (T, F, F, F, F, F) (T, F, F, F, F, F)
Priority: T->Enabled,
F->Disabled)
Allowed VLANs - 10,14,16,18-20,23,30-3 10,14,16,18-20,23,30-3
1,172,250,420,503,510, 1,172,250,420,503,510,
512,516,520,526,529,53 512,516,520,526,529,53
1,1202-1203,1210,1220, 1,1202-1203,1210,1220,
1222-1223,1230,1700-19 1222-1223,1230,1700-19
00,2007,2053,2701-2706 00,2007,2053,2701-2706
,3000 ,3000
Local suspended VLANs - - -
02-27-2024 12:45 PM
There is nothing wrong in your config but I think it is not optimal.
Unknown flood happened when SW missing mac in table, so first point we must check it is this mac connect to orphan, are this Mac appear in both or one NSK or not appear in both NSK.
One of case of flood
as role the Arp aging must be less than Mac aging to eliminate the unknown flood, arp in host try to connect to other host will aging this make host send broadcast ask Mac, if maç aging before arp the host not send arp it send packet with Mac address unknown for both nsk and make both nsk flood this mac
So as I mentioned above first point check maç address shown in log
MHM
02-27-2024 01:23 PM
Thanks for your suggestions. I'll investigate the MAC situation further and keep you updated. As for aging, i have defaults on both switches: 1800 (MAC) > 1500 (ARP), so there shouldn't be issues, i think.
02-29-2024 11:02 AM - edited 02-29-2024 11:18 AM
I have found the problem. I forgot to enable portfast for several access switches connected to my VPC pair. Thus a STP TCN was occurring in the VLAN when these access ports upped or downed/upped causing rebuilding MAC table.
Thank again to all participants!
02-29-2024 11:08 AM
Thanks for update me
Have a nice day
MHM
02-27-2024 11:21 AM
Did you use
Ip arp sync
MHM
02-27-2024 11:29 AM
'ip arp synchronize' is active within VPC domain
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide