Microsoft NLB multicast not working well with Nexus 5k and 7k..NEED HELP!!!!

[dannyngo99]

Question: Hi all,

We need your expertise to help us clarify the NLB configuration on our 5k or 7k switches. We are in the process of migration our email exchange servers from 2003 to 2010. We are implementing Microsoft NLB (network load balancing) on our CAS servers (Client Access servers). These two server are VM guest machines .Their physical VMware hosts are directly connected to 2k switches on port eth101/1/33-36.
Our exchange consultant insisted us to add the following command into either 5k or 7k to allow NLB multicast traffic passing through to all end users hosts.

here is the command requested by our consultant.

Arp 10.156.2.132 03bf.0a9c.0284 ARPA
Mac-address-table static 03bf.0a9c.0284 Vlan 10 interface Ethernet101/1/33-36

I tried to add the above commands into either 5k and 7k . it bounced back with "unrecognized command error".

Note: Many users out there are experiencing the same issue we are having now.

Below are some links regarding configuring NLB on Cisco switch with Microsoft and VMware:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006525
https://supportforums.cisco.com/thread/2091841?decorator=print&displayFullThread=true
http://arstechnica.com/civis/viewtopic.php?f=10&t=1150623
http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/unicast/configuration/guide/l3_ip.html#wp1196870
https://communities.cisco.com/thread/16234
http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml


Question#1
Am I missing anything here that is stopping me from entering these commands?

Question#2:
Since my computer is connected via a static ip address and via Vlan1 port. I am able to ping NBL cluster host Ip address 10.156.2.132 but nobody else who are on other  VLANs (subnets) 101,102,103,104,105 and vlan 50 (wireless subnet).

I am about to add the following command to 5k switch, but want to know if these commands might make it work.

switchport trunk allowed vlan 1,3,5,10-11,15,50,52

interface Ethernet101/1/33
  description Link to ESX # 1
  switchport mode trunk
switchport trunk allowed vlan 1,3,5,10-11,15,50,52

interface Ethernet101/1/34
  description Link to ESX # 1
  switchport mode trunk
switchport trunk allowed vlan 1,3,5,10-11,15,50,52

interface Ethernet101/1/35
  description Link to ESX # 2
  switchport mode trunk
switchport trunk allowed vlan 1,3,5,10-11,15,50,52

interface Ethernet101/1/36
  description Link to ESX # 2
  switchport mode trunk
switchport trunk allowed vlan 1,3,5,10-11,15,50,52

Any help are greatly appreciated.!!!
Thanks,
Danny.

[johgill]

Hi Danny,

In NX-OS, you put ARP entries on the L3 interface, so probably an SVI:

interface vlan 10

ip arp 10.156.2.132 03bf.0a9c.0284

Qestion #2) Your VLAN1 needs to be routed, where is your rouer?  Can the router reach the other destinations?

I would suggest looking into using IGMP instead of this manual NLB configuration, it is very old and annoying to keep ARP entries and static mac entries.

Regards,

John

[dannyngo99]

Hi Johgill,

Thanks very much for your reply. I did add this statement below to 7k switch with no luck.

7k is also acting as the router. I can ping to Nic#1 server1at 10.156.2.128 and server2 at 10.156.2.129.

But I was not able to ping to cluster virtual ip address 10.156.2.132 from 7k switch.

interface vlan 10

ip arp 10.156.2.132 03bf.0a9c.0284

any ideas??

Thanks,

Danny

[johgill]

Did you put the static mac address in as well on the N5k?

The ARP is needed on the 7k L3 interface, and the static mac is needed on n7k and on n5k.

Regards,

John

[dannyngo99]

Hi John,

We tried to enter static entry in 7k

mac address-table static 03bf.0a9c.0284 vlan 10 interface Po2.

we received the following error:

mac address-table static 03bf.0a9c.0284 vlan 10 interface Po2

We are now running 5.(1) image on 7k. We called Cisco support. They verified that we must upgrate the image software to 5.(2) in order to perform the above command. We are going to upgrade the software soon. I will keep you posted. Thanks very much for all your help.

Thanks,
Danny.

[dannyngo99]

Hi John,

Here is the error that we got from the previous reply:

We tried to enter static entry in 7k

mac address-table static 03bf.0a9c.0284 vlan 10 interface Po2.

we received the following error:

Error: Multicast/Broadcast MACs are not supported!

Thx,Danny.

[Lawrence Searcy · Accepted solution]

You need the command from version 5.2(1)

mac address-table multicast 03bf.xxxx.xxxx vlan <y> interface <int1>

to use NLB in multicast mode on the Nexus 7000. The nexus 7000 does not support IGMP multicast mode. Please see

CSCtt00284 Document the limitation of mac address-table multicast

To help others

There are 3 modes to Microsoft Network Load Balancing (NLB)

1.      1.Unicast

2.      2.Multicast

3.      3.IGMP multicast  (check the IGMP checkbox in the GUI while in multicast mode)

In general,every mode uses a different sending and receiving mac address while keeping the unicast virtual IP address (VIP) constant across all 3 modes. This concept makes switches flood traffic at layer 2 since the switch either never sees the destination mac address come in on any of its ports(and hence can’t learn it) or the multicast mac address floods. Either multicast mode, IGMP or normal multicast, also requires static ARP entries on the gateway router since Cisco routers will not learn an ARP reply with a multicast mac address tied to a unicast ip address.

Mac addresses in the 3 modes breakdown into the following components:

• The     first number in the mac address is the type of NLB configuration: 01=IGMP,     02=Unicast, 03=Multicast (Note: bit 2 is the administered locally     multicast space)
• The second number, (bf) is the same for unicast and multicast mode (not IGMP multicast mode     which uses the standard 01-00-5e mac address)
• The last two (IGMP multicast mode) or four (unicast or     multicast mode) numbers are the virtual IP address, i.e. c0=192, a8=168,     04=4, 0a=10 and thus the IP of 192.168.4.10 has a multicast mac address 03-BF-C0-a8-04-0a     while an IGMP multicast mac address would be 01-00-5e-7f-04-0a

Summary of configuration

NLB mode	Switch  configuration	Router  configuration
Unicast	Mac  address-table static 02bf.xxxx.xxxx vlan y interface <int1>  <int2>	Not  required – unicast mac address with unicast ip address
Multicast	Mac-address-table static 03bf.xxxx.xxxx  vlan y interface <int1>   <int2> n7k[5.2(1)]: mac address-table  multicast 03bf.xxxx.xxxx vlan y interface <int1> <int2>	Arp  <virtual IP address> 03-bh-xx-xx-xx-xx arpa
IGMP multicast	Mac address-table static 01005exx.xxxx  vlan y interface <int1>   <int2>	Arp  <virtual IP address> 01-00-5e-7f-xx-xx arpa

[Lawrence Searcy]

I should add if you do IGMP multicast mode on, say, a 6500, you need to make sure that IGMP snooping has an mrouter port to prevent the IGMP snooping from dropping the IGMP joins to 239.255.x.x. I typically do this quick and dirty by turning on multicast routing and putting ip pim sparse-mode on the layer 3 gateway interface.

[johgill]

Good point - it is highly encouraged to use the IGMP mode of NLB from the network point of view.  Supporting this age-old unicast IP/multicast mac trick keeps causing problems as we "forget" in modern network operating systems that some customers still rely on this.  Microsoft, Stonebeat, and Checkpoint all support IGMP mode now.

NX-OS will also require an mrouter port in order to align the direction of IGMP membership reports (joins) and multicast data.

[Lawrence Searcy]

I want to update my last post. The Nexus 7000 business unit has indicated that they support IGMP multicast.

There are 3 different options for IGMP multicast mode (with the 4th for multicast mode):

     Option 1: Static ARP + MAC-based L2 Multicast Lookups + Dynamic Joins (ip pim sparse-mode)

under the interface:

       ip arp 10.0.36.28  0100.5e7f.241c

       ip pim sparse-mode

under the vlan

      layer-2 multicast lookup mac

     Option 1A: Static ARP + MAC-based L2 Multicast Lookups + Dynamic Joins with IGMP Snooping Querier

under the interface:

       ip arp 10.0.36.28  0100.5e7f.241c

under the vlan

      ip igmp snooping querier 10.0.36.254

      layer-2 multicast lookup mac

     Option 2: Static ARP + MAC-based L2 Multicast Lookups + Static Joins + IP Multicast MAC

under the interface:

       ip arp 10.0.36.28  0100.5e7f.241c

under the vlan

      layer-2 multicast lookup mac

      ip igmp snooping static-group <multicast IP address mapped from multicast mac address> interface Ethernet8/2

     Option 2A: Static ARP + MAC-based L2 Multicast Lookups + Static Joins + Non-IP Multicast MAC (for multicast mode only)

under the interface:

       ip arp 10.0.36.28  03bf.0a00.241c

under the vlan

      layer-2 multicast lookup mac

global

     mac address-table multicast 03bf.0a00.241c vlan 36 interface Ethernet8/2

NOTE: all options include a static ARP entry. The different methods are just different ways to limit layer 2 multicast flooding.

[grissonwang]

Good solutions.  Do you know how to work it around on N1000v virtual switch on ESX host server? Because I have both N7K and N1K involved in the forwarding path to NLB servers. N1K also support IGMP, but not sure about "layer2-multicast lookup mac" since I didn't find this command on N1K. Any good suggestion? thanks.

by the way, My N7K is v5.2(1) while N1K is v4.0(4)

[dannyngo99]

Hi Grisson,

please verify with Cisco prior to upgrading your N1k software to a new version to ensure that v.4.2 (1) supports

"Mac Static entry". We were able to enter Mac static address after upgrading our 7k to new version. NLB traffic from servers arre passing through ok since then.Hope this helps.

Thanks,

Danny.

[dan.y]

Option 2: Static ARP + MAC-based L2 Multicast Lookups + Static Joins + IP Multicast MAC

under the interface:

       ip arp 10.0.36.28  0100.5e7f.241c

under the vlan

      layer-2 multicast lookup mac

      ip igmp snooping static-group <multicast IP address mapped from multicast mac address> interface Ethernet8/2

on the above option, for N7K, do we have to change the multicast lookup mode from IP to mac, under the vlan, or the mac: 0100.5e7f.241c still flapping, even the N7K already enable ip igmp snooping?

[Lawrence Searcy]

First, the layer-2 multicast lookup command just determines whether to use layer 3 or layer 2 to generate the TCAM lookup key. In this case, we have a unicast ip address using a multicast mac address. So we want to use layer 2 as the lookup key since layer 3 is not a multicast IP address.

For mac flapping, it means the switch is learning (i.e. it is the source mac address for an incoming packet) the same exact mac address on 2 different ports. Typically, I have seen this when the server is misconfigured to use active/active on dual NICs with the same mac address.

Do you need to use layer 2 for the lookup? Yes. It can't use the unicast IP address in the static ARP entry to look up the forwarding information in the multicast FIB.

The second command for the static group essentially tells the router to put multicast packets with the defined multicast mac address (since it is mac based lookup, it translates the multicast ip address to a mac address) out that interface.

[dan.y]

Thanks for your reply.   I have the following questions, please kindly check them.

1. If we don't change the mutlicast lookup mode to mac, and don't config ip igmp snooping static-group, the packet with destination mac: 0100.5e7f.241c still flapping to other ports with the same vlan on N7K, right?

2. If we don't change the mutlicast lookup mode to mac, and config ip igmp snooping static-group, the packet with destination mac: 0100.5e7f.241c still flapping to other ports with the same vlan on N7K, right?

3. We have to change the mutlicast lookup mode to mac, and config ip igmp snooping static-group, the packet with destination mac: 0100.5e7f.241c won't be flapping to other ports with the same vlan on N7K, right?