Showing results for 
Search instead for 
Did you mean: 

Can't configure adapter-fex on Broadcom 57810 in a C220 M3 server

I'm attempting to setup Adapter-FEX between a Cisco C220 M3 server with a Broadcom BCM57810 (Cisco P/N UCSC-PCIE-B3SFP) and a pair of Nexus 5596 switches.  The server is running Windows 2012 R2 (also tried 2008 R2).  I've setup both the Nexus switches and Broadcom card according to what documentation that I could find (which isn't much).  When configuring the Broadcom card, the port profiles do show up so that I can choose them (so it is seeing the Nexus switch).  I've tried various combinations of assigning the profiles to different channels on the Broadcom card.  However, when the server is booted up, both uplinks go into a down state and show "link failure" in the system log of the Nexus.  Yet when the system is undergoing POST tests and such (pre-Windows boot), both links are active and online.  Is there a configuration setting that I may be missing?  The Broadcom card is setup for AFEX mode with DCB enabled, and flow control is turned off for the channels that I was configuring.   The Nexus switches each have the VM-Fex license installed.
The setup that I'm trying to accomplish is rather simple.  I just need to have a minimum of two virtual adapters created which are on two separate VLANs.  One physical port on the Broadcom card goes to Nexus switch #1, and the second port goes to Nexus switch #2.  The Nexus switches are setup in a redundant fashion with VPCs between them.  All drivers on the Windows server are up-to-date from Cisco's website.  The Nexus switches never show any vethernet interfaces being created.  And manually created vethernet interfaces don't come online either.
Commands entered into the Nexus switches(and synced between them):
install feature-set virtualization
feature-set virtualization
vethernet auto-create
port-profile type vethernet PRIMARY_DATA_VLAN103
switchport access vlan 103
state enabled
port-profile type vethernet BACKUP_DATA_VLAN400
switchport access vlan 400
state enabled
interface Ethernet1/32
description New Backups Server
switchport mode vntag
no shutdown
Output of Nexus switch interface after server is booted up:
CU-DC-5K1# sh int virtual summ
Veth       Bound        Channel/  Port        Mac                VM
Interface  Interface    DV-Port   Profile     Address            Name
Total 0 Veth interfaces
CU-DC-5K1# sh int e1/32
Ethernet1/32 is down (Link not connected)
  Hardware: 1000/10000 Ethernet, address: 002a.6a48.e7a7 (bia 002a.6a48.e7a7)
  Description: New Backups Server
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is vntag
  auto-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 00:36:29
  Last clearing of "show interface" counters never
  30 seconds input rate 0 bits/sec, 0 packets/sec
  30 seconds output rate 0 bits/sec, 0 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
    0 unicast packets  71045 multicast packets  17043 broadcast packets
    88088 input packets  8066953 bytes
    0 jumbo packets  0 storm suppression bytes
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
    36425 unicast packets  11806 multicast packets  18962 broadcast packets
    67193 output packets  21437797 bytes
    0 jumbo packets
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 0 output discard
    0 Tx pause
  40 interface resets
Cisco Employee

You may hit this issue.

CSCun93741 - Broadcom 57810 Adapter FEX does not work when OptionROM is enabled



Thanks.  Actually I've had a TAC case open on this pretty much since I originally posted it, and it was a confirmed bug in the Broadcom firmware.  The CNA isn't sending the proper requests to the Nexus switch to setup the virtual adapters.  QLogic (which owns part of Broadcom) is supposedly working on a fix, but I have yet to see it.