Unicast Flooding on Nexus 7018 (vPC enabled)

[wandering_997]

hi all,

I have a serious problem with nexus 7018, there're unicast flooding on one n7k, named n7k-1, which is the member of vPC domain combined with 2 N7Ks.

And I think I've already got the reason why this happened.

///////////////////////////

That is, some modules lost several addresses in their mac-address-tables.

///////////////////////////

N7K-1# show module

Mod  Ports  Module-Type                      Model              Status

---  -----  -------------------------------- ------------------ ------------

1    48     10/100/1000 Mbps Ethernet XL Mod N7K-M148GT-11L     ok

2    48     1000 Mbps Optical Ethernet XL Mo N7K-M148GS-11L     ok

5    48     1000 Mbps Optical Ethernet XL Mo N7K-M148GS-11L     ok

7    48     1000 Mbps Optical Ethernet XL Mo N7K-M148GS-11L     ok

8    48     1000 Mbps Optical Ethernet XL Mo N7K-M148GS-11L     ok

......

N7K-1# show mac address-table count

MAC Entries for all vlans :

Dynamic Address Count:                 5692

Static Address (User-defined) Count:      0

Secure Address Count:                     0

N7K-1#

N7K-1# show mac address-table 1 dynamic | in "dynamic" | wc -l

5598

N7K-1#

N7K-1# show mac address-table 2 dynamic | in "dynamic" | wc -l

5595

N7K-1#

N7K-1# show mac address-table 5 dynamic | in "dynamic" | wc -l

5665

N7K-1#

N7K-1# show mac address-table 7 dynamic | in "dynamic" | wc -l

5605

N7K-1#

N7K-1# show mac address-table 8 dynamic | in "dynamic" | wc -l

5593

N7K-1#

See, all mac counts are different from the global mac-address-table on N7K-1.

But, N7K-2 does not have this problem, all modules are ok.

N7k-2#

N7k-2# show mac address-table count

MAC Entries for all vlans :

Dynamic Address Count:                 5692

Static Address (User-defined) Count:      0

Secure Address Count:                     0

N7k-2#

N7k-2#

N7k-2#

N7k-2# show mac address-table 1 dynamic | in "dynamic" | wc -l

5692

N7k-2#

N7k-2# show mac address-table 2 dynamic | in "dynamic" | wc -l

5692

N7k-2#

N7k-2# show mac address-table 5 dynamic | in "dynamic" | wc -l

5692

N7k-2#

N7k-2# show mac address-table 7 dynamic | in "dynamic" | wc -l

5692

N7k-2#

N7k-2# show mac address-table 8 dynamic | in "dynamic" | wc -l

5692

N7k-2# 

I had clean the mac-address-table, and all mac-address-tables had been synced fine, and the unicast flooding went away.

.

But it came back again after a few hours.

How could I fix the mac-address sync function between the modules ?

Thanks.

Dayong

[Jerry Ye]

What version of code? NXOS should not have that issue since the default timers are already following best practice.

Can you post the vpc domain config?

Regards,

jerry

[wandering_997]

hi jerry,

In fact, this is the second n7k chassis which has mac-address sync-failed issue I've met.

The current NXOS version is 5.1(5), and I think it's maybe a software bug.

Anyway, the unicast flooding was triggered by mac-address losing on the modules.

By the way, best practice is just best practice, it can not cover everything.

//////////////////////////////////////////////////////

N7K-1# show run vpc

version 5.1(5)

feature vpc

vpc domain 10

  peer-switch

  role priority 20480

  peer-keepalive destination 1.1.1.2 source 1.1.1.1

  delay restore 40

  peer-gateway

  reload restore

  ip arp synchronize

interface port-channel98

  description VPC_Peer-link

  switchport

  switchport mode trunk

  spanning-tree port type network

  vpc peer-link

N7K-1# show vpc brief

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 10 

Peer status                     : peer adjacency formed ok     

vPC keep-alive status           : peer is alive                

Configuration consistency status: success

Type-2 consistency status       : success

vPC role                        : primary                      

Number of vPCs configured       : 115

Peer Gateway                    : Enabled

Peer gateway excluded VLANs     : -

Dual-active excluded VLANs      : -

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans   

--   ----   ------ --------------------------------------------------

1    Po98   up     1,10,111-122,124-128,131-134,142,144,152-153,188-1

                   89,255,333                                       

//////////////////////////////////////////////////////

[Jerry Ye]

When this happened, any mac address disappeared from the peer link? Are these mac addresses belong to orphan hosts? I might have the bug ID for you but would like to verify if these 2 conditions.

Best practice is what Cisco recommend customer to do in general deployment scenario. Of course, this will not cover some odd/corner-case design.

Regards,

jerry

[wandering_997]

hi jerry,

When this happened, no mac address disappeared from the peer link, and the total number of mac address was not changed on both chassises.  And The mac addresses that disapeared could be found in arp table, that's why unicast-flooding occured.

All the hosts, that have the disapeared mac addresses, are connected to layer-2 access switches which connect N7Ks via vPC links.

thanks for helping,

dayong

[Jerry Ye]

Sorry for the late response, I was busy with work.

I suspected you are hitting this bug, CSCtt37768.

Regards,

jerry

[wandering_997]

hi Jerry,

This bug is mostly close to my issue.

I found all mac addresses lost on line-cards are learned from the N5K which connected with VPC links, and clear mac address table manually could resolve this problem for a few hours.

Anyway, thank you very much.

Regards,

dayong