ucs c210 nic ether-channel issue with 2 diffent NIC types

[vijendra Saravanamuthu]

hi guys,

its a ucs c210 which has 2 types of NICs. one that comes with the a 2 NICs gig ports on mother board (intel corpration 82576 gig network connection) and the other one is installed into the pci slot 4 ports (broadcom netxtreme bcm5709 1000base-T).  all we are trying to do is load balancing with 2 of the mother board NICS (vmnic 6 & 7) and 2 ports from the broadcom nics (vmnic 5 & 4).    all these 4 ports are in the vswitch0 nic teaming as active adaptors  and configured to use route based on ip hash.     we had cables plugged into nmnic 6 & 7 and all working.  when we decided to plug into vmnic 5 & 4 also, it brought everything down.. nothing is accessible.  I think the switch ports are configured correctly for port channeling..(though I didn't have access to the switch, I had network eng telling me its all configured correctly with vlan and  port channel mode on)  ...it appears the minute we use 2 different types of NICs (mother board nic and Broadcom nic) it brings it down. some kind of nics incompatibility.  the reason I say that is that when I used 4 ports on the Broadcom (vmnic 2,3,4, & 5) and load balance (and not use the motherboard nic at all) , all works fine. so its not the switch.

any idea?

thanks a bunch.

[Keny Perez]

Hi,
As seen here : http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/hw/C210M1/install/C210M1/overview.html#wp1210899 ; ( Figure 1-2 Rear Panel Features)  the onboard NIC ports are 10/100/1000 same as the Broadcom's ports as confirmed here:   http://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/1CS5709-Broadcom_quad-port_GbE_HBA.pdf  (Look for "Data Rate" in page 2)

Now those ports go to auto negotiate by default, what is the configured speed in the upstream ports? I am thinking about a weird scenario where some ports are configured with one speed while the other ports have a different speed negotiated.

Also, the port-channel negotiation protocol we use with UCS is LACP and as you can see here, mode on meaning "No LACP":   http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli/CLIConfigurationGuide/EtherChannel.html#wp1160773 ; Is i possible you can request a copy of the running config of the ports that are part of the port-channel in the upstream switch?

Also, have you confirmed that the drivers are at the right level for both cards?

Just to be sure, you are running ESXi on this server? (seems to be the case as you mentioned "vmnics" but wanted to be sure) if yes, what is the CIMC and ESXi version you are running?

HTH,

-Kenny

[vijendra Saravanamuthu]

thanks for the response Kenny.

as I said, it works fine if I use all 4 ports of Broadcom.   yes its esxi 5.0.  cimc is 1.4(3v).  I did use host upgrade utility to make sure all up todate. not that I mean that it was working with Broadcom& LOM before that upgrade. we purchased the Broadcom 2 weeks ago (since more VMs is being deployed on this c210). so I installed the Broadcom, upgraded the driver using host upgrade utility and added the switch conifg.. 

interface Port-channel31

switchport

switchport access vlan 477

switchport mode access

end

interface GigabitEthernet10/47

description to rack D9 torucs02 onboard NIC 0 Port 1

switchport

switchport access vlan 477

switchport mode access

spanning-tree portfast edge

channel-group 31 mode on

end

interface GigabitEthernet10/48

description to rack D9 torucs02 onboard NIC 1 Port 1

switchport

switchport access vlan 477

switchport mode access

spanning-tree portfast edge

channel-group 31 mode on

end

interface GigabitEthernet11/47

description to rack D9 torucs02 onboard NIC 0 Port 2

switchport

switchport access vlan 477

switchport mode access

spanning-tree portfast edge

channel-group 31 mode on

end

interface GigabitEthernet11/48

description to rack D9 torucs02 onboard NIC 1 Port 2

switchport

switchport access vlan 477

switchport mode access

spanning-tree portfast edge

channel-group 31 mode on

end

..we have a similar design in our datacenter..where c240 is being used with nexus switches.  here its c210 with 6500 series switch.. configs are pretty much the same other than vlan.

to me, I think we have to enable something in the bios for this nic to work with the LOM.  that's my guess. well..i have to shut down to look for what I need to enable..

thank you.

vijay

[Keny Perez]

I think you will have to open a TAC just to confirm cause I did a quick search by myself and it looks like someone else already did some testing ( a while ago) and the result was that doing the NIC teaming required to use the same card to have the expected results.

The only guide I have does not mentioned anything like doing a team with different cards (hardware):

http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-c-series-rack-servers/117637-technote-UCS-00.html#anc6

Again, my guessing is that there must be some sort of conflict between the way one cards works over the other and I just confirmed the channel mode is not a problem in C-series servers.

HTH,

-Kenny

[vijendra Saravanamuthu]

after some digging I came across this article in VMware.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2019871

"ESXi/ESX host loses network connectivity after adding new NICs or an upgrade"

it happens if I add a nic after esxi is installed and it messes up the numbering order.

I don't think want to try the workaround.