inquiry about M71KR-E and Windows

russ.givens · ‎01-04-2011

I'm new to UCS so not all the familiar. I have a few blades with the M71KR-E in them and am installing Windows on them.

1) I have on interface going to fabric A and the other interface going to fabric B. When I install Windows it sees both network interfaces, but I was wondering how to team them within the operating system. Do I have to go to Intels site and download appropriate drivers and teaming software since it sees the interfaces as Intel 82598EB 10 Gigabit AF Dual Port adapters?

2) Also Windows doesn't apparently have a driver for the fiber channel portion of the M71KR-E as they show up as unknown devices. Where do you get the drivers for the fiber channel controllers?

Thanks.

Jeremy Waldrop · ‎01-04-2011

If you enable failover on the vNIC then you do not need NIC teaming software within Windows but Windows will see 2 NICs. If you would still rather do OS side teaming you can but you will need to install the Intel teaming software.

On the FC side you need to download and load the Emulex FC driver. The driver bundle can be downloaded from the Cisco site.

russ.givens · ‎01-05-2011

So that raises a question to me. How do you configure the two network interfaces. Typically when you have two network interfaces in a machine and you would like to team them to use the teaming software and then configure the tcp/ip settings for the network teamed interface that is created with the software. If I don't team the interfaces how do you configure the two interfaces. You can only have one interface on a machine with a specific IP address. You can't have two interfaces with the same IP.

Thank you for you help.

Robert Burns · ‎01-05-2011

You can team the adapters still. You'll just need the appropriate Intel Pro set suite to support the advanced networking services (ANS). You can't aggregate the links totally (like an ether-channel or us IP based hasing) but you can use balancing options like ALB or Explicit failover.

Refer to:

http://www.intel.com/support/network/sb/cs-009747.htm#Modes

Regards,

Robert

Manish Tandon · ‎01-05-2011

Russ

I will just add some more color to what Robert mentioned.

In UCS from the adpater you cannot do active/active teaming. This is because the FI's are not vPC peers.

So for teaming you will have to use active/passive, ALB, SLB etc (and not 802.3ad).

Now to achieve teaming of failover capabilities there are 2 approaches.

Both are fully supported and you can take either.

Background on the M71KR-E or Q (also referred to as the Menlo for the FCoE ASIC) -

If you look at the attached diagram, a M71KR is characterized by 2 Ethernet PCI devices (the Intel 82598) and the FC chip from Qlogic or Emulex(depending on what flavor of the CNA you are running).

These PCI devices (2 ethernet and 2 FC) is what the OS sees in the device manager.

The 2 ethernet devices will be Local Area Connection#0 and #1 as seen in the network properties.

The card also has 2 DCBX ports which use the baclplane traces to connect to the IOM-A and B respectively.

By default, Local Area Conenction#0 is wired to DCBX0 while #1 to DCBX1.

It is important to note that the Intel PCI devices are not wired directly into the physical ports. They go through the "Menlo FCoE" ASIC.

Now regarding teaming -

Approach #1 (what has traditionally been done)-

You create a SP and set up 2 vNICS going to A and B Fabric with Fabric Failover disabled choosing the VLAN.

You download the proset teaming driver on the blade running windows for example. You unpack and install it.

You create a team and choose Active/Passive (no 802.3ad).

It creates an interface "Team#0" and you assign an IP to it. It will do active/passive amongst Local Area Connection #0 and #1.

If DCBX0 fails (and was the primary path), the driver will go via LocalArea Connection #1 (DCBX1) and vice versa.

You are correct in stating you assign an IP to Team#0 and not the Local Area Connection#0 or #1 and hence you have 1 IP subnet.

Approach #2 (Using fabric failover unique to UCS) -

You create the SP and setup 2 vNICs going to A and B with Fabric Failover enabled (so they will be A-B and B-A) choosing the VLANs reqd on both.

In this case the OS again will show you the 2 network devices Local Area Connection #0 and #1.

This is where the seamlessness of the solution comes in. There is *no* teaming driver to be downloaded/configured.

You assign an IP to Local Area Connection #0 for example 10.x.x.x.

This will go via DCBX0 ..Now if there is an issue upstream on Fabric A, the Menlo ASIC will transparently redirect Local Area Connection#0 to DCBX1 giving you active/passive. If Fabric A uplink is gone lets say, your Local Area Connection#0 will still stay up and is now going via Fabric B. If Fabric A comes up, the system reverts it back to its original path.

In this case you could potentially assign another IP subnet to Local Area Connection#1 for example 20.x.x.x and use that too which will go via DCBX1. DCBX0 and DCBX1 are backing each other and is being done by Menlo amongst other things.

So in this case failover is provided as part of the fabric. No teaming software version issues, nothing to configure, just a checkbox in the SP.

It works indepedently of the OS so for Windows/linux etc ..behavior is same. This is recommended for the flexibility it offers.

Fabric Failover is only available on the M71KR and the M81KR (the architecture of the 2 cards is different though).

Hope it helps

Thanks

--Manish

cweinhold · ‎01-06-2011

To add even more color:

The M72KR-E and -Q adapters (the second generation emulex and qlogic cards) don't do failover and have really terrible Windows teaming drivers. I pity anyone who has to use them. We performed the exact same teaming steps on four identical Windows machines and wound up with four different network configs (interface numbering, primary NIC election, etc). Qlogic's issues seemed less severe than Emulex's. E.g., with Emulex, if you boot Windows when the Team's primary NIC is in a down state, the Team won't ever be able to come up.

I hope Cisco doesn't stop selling the first generation M71KR-E and -Q. I'll happily give up a bit of SAN performance for the rock-solid Menlo drivers.

c.larsen · ‎03-30-2012

I am trying to setup NIC teaming using an M72KR-Q card with Windows 2008R2. Where did you get the QLogic software to install for allowing you to team the NIC's?