Couple networking issues with UCS Manager/6120's

[jhedman]

Hi, there everyone.

I'm curious if anyone can add a little value to a couple of questions i've got while doing a current implementation of UCS.  Both of these issues really are layer three issues that should be handled by a layer three switch upstream, but really I have nothing else to try.  Both of the things i'm trying to do work fine on siwtches and servers connected to standard Catalyst switches in this customers environment, but they don't work on UCS.

First, I"m trying to use host based NLB on systems hosted on the UCS B series servers.  These servers use unicast and multicase NLB and typically in a cisco environment, to make these forms of NLB work, you simply have to create a static ARP entry on the layer three switch upstream.  This works fine for servers and VM's on standard servers and Catalyst switches.  However, in this case, no matter what I do, the static arp entry on the layer 3 switch upstream from the 6120 doesn't seem to be working.  Is anyone aware of a way to add or modify the CAM/ARP table on the Nexus 6120 for UCS B systems?

On that same line, I have VM's that are built on a Cisco UCS blade server that's running VMware ESX4.  These VM's are going to be setup to use DHCP.  However, none of the VM's in any VLAN are getting their DHCP addresses?  Typically this is handled by putting the DHCP relay information on the layer 3 switch upstream, but we've done this and it still doesn't work.  Has anyone experienced this and if so, how did you fix it?  I've verified that if I set one of these VM's up with a static IP in the same VLAN that should have DHCP, the VM can ping it's default gateway which is an interface on the same layer 3 switch hosting the dhcp relay information.  It really seems like the DHCP request is never making it to the layer 3 switch.  Anyone seen anything like this?  Again, I'd like to try and add dhcp relay information to the Nexus 6120, but I can't seem to figure out any way to do this.

Again, both of these things are pretty standard easy to do things that work just fine on standard hosts and VM's living in the standard server and catalyst world, not sure why they won't work on UCS, but it's frustrating.

Thanks in advance.

[stechamb]

What are the connections northbound of the 6120?  Is it a Nexus5k?  Or some other 10GbE device?  Is 6120 in End Host Mode?  No funky Pin Groups?  No barmy VLAN setup (Fab A VLAN 3 but not on Fab B?) etc.

Have you created all the correct VLANs on northbound devices, and in UCS?

On your northbound switches can you use SPAN or similar to sniff the ports connect to UCS and see if the NLB and DHCP VLAN traffic is getting out?

Ta

Steve

[jhedman]

Thanks for the quick response.  So two things.  We've worked out the DHCP issue.  It turned out to be an issue on the InfoBlox system that handles the DHCP addressing.

Still have the issue with the Arp controls though.  I don't think the CAM/ARP tables are available for management on the 6120's.

As for connectivity upstream.  The 6120's are directly connected to two 6513's upstream using the 16 port 10Gb XL modules.  All routing appears to work just fine.  No issues there.  At this point, I need to figure out how to modify the ARP stuff or get that working so tha thte load balancing will work for VM's hosted on the ESX 4 servers within the UCS environment.  Definitely a lot of nuances I have to workout between UCS and standards servers/nexus/catalyst.

Thanks again, any input on the ARP issue on the 6120's would be very helpful.

[visaran]

Hi !

Did you find a solution to your problem. I am having trouble setting up NLB on UCS as well.

[jhedman]

Hi there...  So I did get NLB working, and it was a pain in the rear.  In the end, I ended up having to use Multicast NLB with single nic's on my vm's.  For some reason, unicast with multiple nics wouldn't work no matter what I did.  Not sure why.

Anyway, once i put the correct arp entry on my upstream 6500's (make sure you do both if it's multi-homed, the multicast nlb began to work just fine.  It's key to use the correct MAC address created by the windows NLB manager while creating the nlb configuration.  To that point, and maybe you're already doing this but make sure you allow the NLB manager to enable NLB on each nic.  Some people have a habit of checking the box on the adapter config before running NLB manager and creating the cluster.

It took quite a while to get it working, and I actually prefer to use unicast, but like I said, i gave up on that.  Hope that helps a little.

[visaran]

Thanks for the information @jhedman!

I am able to create a NLB cluster in multicast mode and it works fine in the local subnet. however the cluster IP cannot be reached from outside the local subnet.  My upstream switches connected to the 6120s are Nexus 5010. I tried to define a static arp entry on the 5010s & it doesn't seem to work.

[jhedman]

You will definitely need to have the static arp entry or NLB will not work properly.  This is a well known issue on pretty much all Cisco switches, as they are designed not to learn the MAC address of the NLB VIP.

[Manish Tandon]

Hi

What ethernet switching mode are you running the 6100's in?

Is it switch mode or the default end-host-mode?

In End-host-mode, unknown unicast flooding is not done and that could be the reason why unicast NLB does not work while multicast works.

In switch mode, uknown unicasts are flooded and unicast NLB should work there too.

Both switch mode and End-Host -Mode are fully supported modes on the 6100.

Let me know if you have any questions ..

Thanks

--Manish

[jhedman]

The switches are in end-host-virtualizer mode so if your statement is correct about the 6100's not forwarding unicast traffic across the broadcast domain, then yes, that would be why it's not working.  Thanks a ton for the heads up!

[loizosko]

did you find the solution?

usually when connected to the traditional switches you do static arp and tell which ports the mac address is coming from.

how this can be done in the ucs B series and get the microsoft NLB multicast mode working?

each of our 6120 has 2 uplinks (1 to each core) and unfortunately we don't have  vpc to make them in one

we are also using end host mode.

[jhedman]

I was actually able to get it to work just fine with single nic multicast NLB.  It's not the best way to leverage MS NLB, but it worked.  I don't like using the single NIC multicast mode just partially because the manager doens't even launch properly.

Unicast just simply didn't work at all.  But I was able to get Multicast to work just fine.

[loizosko]

you mean single uplink to your network or to your server?

i have 1 vnic assigned to the server

can you elaborate?

[lletterm]

RE: DHCP issue on 6120..

We also have an install of 6120's, Fab-A and Fab-B..each has a port-chan going to core-1 and Core-2..the ucs is server blades are trying to do a pxe cold boot and failing to get a dhcp address on teh port-chan's that connect to the core-2 device..The port-chan's that connect to the core-1 device work fine..Static IP's work fine across port-chan-2's, just no dhcp request..

fabrics are in EH mode

vlan are configured on both sides

cores are 7018's

nxos is 4.26

any help would be appreciated...

[abbharga]

Hi Lletterm,

Are core-1 and core-2 connected to each other? In simple words are they in the same l2 domain for the vlan on which you are having issues with the DHCP?

Looking at you nx-os version you are running a pre 2.0 UCSM code and if the core switches are not connected then you could be having the L2-disjoint issue.

./abhinav