Multicast Issue in UCS

[marknigh1]

We have, what we believe, to be a multicast issue or design issue in our UCS environment.

Environment:

two (2) 6120 Fabric Interconnects configured in end-host mode

two (2) Chassis

four (4) v4.1 ESXi Blades

two (2) 3750 switches in one (1) stack.

one (1) 7606 router

We have configured a vyatta software-based router in a Virtual Machine. We have configured ospf on both the vyatta router and the 7606 router. The issue we are seeing that the DR election is completed but after the 40 second dead timer, the vyatta router loses the routes learned from the 7606 router. In addition, the DR router (the 7606) is lost for approximately 3 seconds. After which, the DR is elected again and the routes are learned.

On the 7606, We are receiving Hello's from the  Vyatta after the initial election that do not contain our IP (the 7606).  The 7606 debug of OSPF ADJACENCY says "cannot see ourself in hello from..." The  router then re-starts the Election  process...

It looks like we are having trouble  getting All the hello's to the Vyatta router, which is probably why it keeps  sending the neighbor election hello's which are getting through in both  directions.

Is ther

[Robert Burns]

Mark,

Can you verify if you've enabled IGMP snooping querier or Multicast routing (mrouter) on the 3750 stack?

Currently the 6100's do not support IGMP snooping querier which would explain your loss of OSPF adjacencies.

Instruction on enable IGMP snooping querier can be found here:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.2_25_see/configuration/guide/swigmp.html#wp1193337

Regards,

Robert

[marknigh1]

Robert,

I appreciate your response.

I did configure ip igmp snooping querier on the 3750 stack but without any difference. The 3750 stack is layer 2 only. It has a management IP address on it which is in a vlan that isn't associated with the vlan/subnet that we are wanting to create the adjancies. There is an optional ip address that can be configured after the ip igmp snooping querier command but I am not sure what IP address I should use.

I tried disabling ip igmp snooping on the 3750 stack without success. Shouldn't this eliminate the stack from "listening" to multicast packets thus the switch treats multicast similar to broadcast, flooding on all ports?

Do you have a suggested configuration?

[marknigh1]

Sorry, let me give alittle more specifics on my environement

vyatta router IP address is 192.168.0.2

Cisco 7606 IP address is 192.168.0.1

Vlan 612

3750 stack has one (1) SVI and the IP address is 10.0.0.1 in vlan 105.

[Robert Burns]

Any chance you're using LAN Pin Groups with your uplinks?

[marknigh1]

Yes we are using LAN pin groups. Each ESXi has four (4) vNICs. Two (2) assigned to a standard switch for Management and the last two (2) are assigned to a vDS.

The ip igmp snooping querier command isn't having effect. I do not have a interface for the vlan we are using to communicate between the vyatta and the 7606 router. Do I need to create a interface vlan x for this and assign an IP? This seems like a waste of an IP address.

Thanks for you help with this design/issue!!

[Manish Tandon]

Mark

As you mentioned multiple interfaces I am going to assume that its the Palo adapter present in this case.

If yes, then what you are seeing is the expected behavior on the Palo. The reserved space 224.0.0.x is not passed onto the host by the Palo firmware

due to filtering even if the switch was to flood it.

Other adapters like the Menlo's, Gen2's, Oplin etc do not have this issue.

To turn it on for Palo for all 224.0.0.x is something which is being looked into but is not as straight forward as it has other implications on the system.

(Btw, we *only* allowed 224.0.0.251 in 1.4 to be in the filter list for Oracle mDNS based installs as it uses that address).

You could try it on a Menlo or any other card (if thats an option).

Another workaround which you can try (it works on a bare metal OS like Linux for sure) is to put the vNIC in promiscuous mode (not the VM's but ESX's vmnic0/1 etc) so that if the switch floods it, the vNIC forwards it upstream towards the soft switch/host without applying filtering rules.

It would be good if you could open a TAC case on this so that we can look at it further offline and reference the enhancement id details etc.

Thanks

--Manish

[Manish Tandon]

Mark

For ESX environment we would need the output of some other commands too.

Better to take it offline.

A TAC case would be ideal so we can track it otherwise my email is mtandon@cisco.com

Thanks

--Manish

[HAROLD MEIER]

Manish,

Can you elaborate on this statement and/or point me to some Palo docs that may explain this further?

"If yes, then what you are seeing is the expected behavior on the Palo. The reserved space 224.0.0.x is not passed onto the host by the Palo firmware due to filtering even if the switch was to flood it."

This is likely unrelated, but I have a Microsoft IGMP Multicast NLB cluster on ESX hosts using Palo adapters, and we see inconsistencies that we can't nail down. We have IGMP snooping turned on up stream, and the multicast convergence seems to work fine. The NLB nodes however see a large number of TCP resets causing web servers to not send all the data.

[Manish Tandon]

Harold

I am not sure of the bulletin for this but can check up on that.

The enhancement id is CSCtj99010 and will provide you a headline if you were to query it using the bug toolkit.

ORARAC - VIC will not get MDNS packet with Filtering ON

Symptom: Server with Palo adaptor is unable to receive MDNS Packets with Multicast IP 224.0.0.251. Conditions: This issue is observed when Palo Adaptor is used in non-promiscous mode.

This issue is *only* for the Local Network Control Block i.e 224.0.0.0/24 range *and* Palo adapter.

http://www.iana.org/assignments/multicast-addresses/multicast-addresses.xml#multicast-addresses-1

mDNS in the above url is 224.0.0.251 and list contains OSPF (224.0.0.5/6 which Mark is running) addresses too.

If you look at the range of services, they are not your "typical" services run on hosts.

If you are using any other address range (have seen NLB implementations with 239.x.x.x typically) then its not the same issue we are talking abt i.e you need igmp snooping etc.

For 224.0.0.x, igmp snopping (and hence querier) doesn't apply. Its supposed to be flooded and is link-local.

In UCS as you know, multicast replication happens at at the nearest point to the receiver(s).

What this means is that if you have 2 servers (blades) under the same IOM, the FI(6100) sends 1 packet to the IOM,the IOM replicates it and sends to the 2 blades.

In case of Palo like adapter, if you have multiple vNICs on the same server (blade) interested in a group, the FI/IOM will send 1 frame to the adapter and the replication happens on the adapter. It is better utilization of the network bandwidth.

A check in place on the Palo adapter prevents the replication of the 224.0.0.0/24 space. We just opened it for 224.0.0.251 (enhancement id above) in 1.4 for Oracle install use case but did not do a blanket 224.0.0.0/24 open. The workaround is to put the Palo vNIC in promiscuous mode for such scenario currently. In a soft switch scenario, the soft switch should have put it in promiscuous mode which is why I asked for more show commands from Mark (he has a TAC case and we are following up) and if yes, there is some other issue on his network.

Hope it clears up some of the confusion.

Thanks

--Manish

[Nirmalbang]

We are facing this issue after upgrade to 2.1(3g), nic card drops the packets for any multicast address starting with x.0.0. Example. If we try getting multicast on 227.0.0.1 to 227.0.0.254 we do not receive multicast. Just to check we tried getting multicast on other address like 228.0.0.1 but we were unable to receive it. But when we tried getting multicast with say like 227.1.10.35 group we were able to get multicast. TAC has suggested us to upgrade the UCS firmware to latest.

Thanks

Amit