UCS 240 M4 with VIC 1387 works with Nexus 3064, but not with 3850

[Elliot Dierksen]

I have an absolutely maddening issue. I have a Nexus 3064 that has been the switch for my storage and vMotion networks for several years, and it has worked great. I am trying to transition some of the 40G ports to a 3850 so I don't have to run the Nexus all the time. Its fans are much louder than the 3850's. On the 3850, I get CDP in both directions when connected to either switch, but IP doesn't work. I can't ping, and no ARP entry for the UCS. When connected to the Nexus, everything works properly. It is a C240-M4SX with a 1387 MLOM card, and it is running ESXi 7.0. The port configurations are simple, and I have stared at them quite a bit, but still no joy. Here is the Nexus config fragment.

interface Ethernet1/51
  description vm-ucs1 - vmnic3
  switchport access vlan 252
  spanning-tree link-type point-to-point

 Here is the 3850 config fragment.

interface FortyGigabitEthernet1/1/2
 description vm-ucs1 - NFS
 switchport access vlan 252
 switchport mode access
 switchport nonegotiate
 spanning-tree portfast

I see input and output packets on the 3850 side and the ESXi side, but IP fails. I am stumped at the moment. It is the exact same cable and QSFP. I moved it from the Nexus to the 3850. I plug in a different C240 M4 running TrueNAS (FreeBSD) with a Chelsio T580 card and it works fine.

[balaji.bandi]

what is the version code on CAt 3850

can you post below output - when the port output from nexus and when you move to 3850

show interface

show intercace Ethernet1/51

show intercace forty1/1/2

where is layer 3 SVI, when you connect 3850, is the nexus still live ?

[Elliot Dierksen]

Both the Nexus and the 3850 have different IP's on VLAN 252. The 3850 is 192.168.252.29 and the Nexus is 192.168.252.30. Even though L3 routing is possible, all the communication in this VLAN's is between hosts on 192.168.252.0/24. Here it is from the 3850.

EBD-3850#sh int fo1/1/2
FortyGigabitEthernet1/1/2 is up, line protocol is up (connected)
  Hardware is Forty Gigabit Ethernet, address is 0042.5ac4.e63e (bia 0042.5ac4.e63e)
  Description: vm-ucs1 - NFS
  MTU 9198 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 40Gb/s, link type is auto, media type is QSFP 40G SR4 SFP
  input flow-control is on, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 01:04:24
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     3096 packets input, 223325 bytes, 0 no buffer
     Received 3086 broadcasts (192 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 192 multicast, 0 pause input
     0 input packets with dribble condition detected
     2150 packets output, 204838 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

 And here it is from the Nexus.

EBD-N3K# sh int e1/51
Ethernet1/51 is up
admin state is up, Dedicated Interface
  Hardware: 40000 Ethernet, address: 18e7.2897.c0fa (bia 18e7.2897.c0fa)
  Description: vm-ucs1 - vmnic3
  MTU 1500 bytes, BW 40000000 Kbit , DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is access
  full-duplex, 40 Gb/s, media type is 40G
  Beacon is turned off
  Auto-Negotiation is turned on  FEC mode is Auto
  Input flow-control is off, output flow-control is off
  Auto-mdix is turned off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  EEE (efficient-ethernet) : n/a
    admin fec state is auto, oper fec state is off
  Last link flapped 00:00:30
  Last clearing of "show interface" counters never
  1 interface resets
  Load-Interval #1: 30 seconds
    30 seconds input rate 536 bits/sec, 0 packets/sec
    30 seconds output rate 1304 bits/sec, 0 packets/sec
    input rate 536 bps, 0 pps; output rate 1.30 Kbps, 0 pps
  Load-Interval #2: 5 minute (300 seconds)
    300 seconds input rate 424 bits/sec, 0 packets/sec
    300 seconds output rate 568 bits/sec, 0 packets/sec
    input rate 424 bps, 0 pps; output rate 568 bps, 0 pps
  RX
    0 unicast packets  5 multicast packets  30 broadcast packets
    35 input packets  2610 bytes
    0 jumbo packets  0 storm suppression packets
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    0 unicast packets  27 multicast packets  0 broadcast packets
    27 output packets  5175 bytes
    0 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause

 There is a link between the 3850 and the Nexus. When the host is connected to the Nexus, both the 3850 and the Nexus can ping the host IP (192.168.252.37).

[balaji.bandi]

First hand i would change the MTU in line with nexus :  

EBD-3850#sh int fo1/1/2
FortyGigabitEthernet1/1/2 is up, line protocol is up (connected)
  Hardware is Forty Gigabit Ethernet, address is 0042.5ac4.e63e (bia 0042.5ac4.e63e)
  Description: vm-ucs1 - NFS
  MTU 9198 bytes, BW 40000000 Kbit/sec, DLY 10 usec,

BD-N3K# sh int e1/51
Ethernet1/51 is up
admin state is up, Dedicated Interface
  Hardware: 40000 Ethernet, address: 18e7.2897.c0fa (bia 18e7.2897.c0fa)
  Description: vm-ucs1 - vmnic3
  MTU 1500 bytes, BW 40000000 Kbit , DLY 10 usec

change above to MTU on Cat 3850 and test it

still not working, clarify below :

There is a link between the 3850 and the Nexus. When the host is connected to the Nexus, both the 3850 and the Nexus can ping the host IP (192.168.252.37).

is this link still connected ?

[Elliot Dierksen]

The MTU on the 3850 system is 9198 (system mtu 9198). ESXi is configured with an MTU of 9000 for that VMkernel interface, and the 1387 is configured for MTU 9000. The link between the 3850 and the Nexus is still connected. I can ping the host with 9000 byte packets from both the 3850 and the Nexus.

EBD-N3K# ping 192.168.252.37 packet-size 9000
PING 192.168.252.37 (192.168.252.37): 9000 data bytes
9008 bytes from 192.168.252.37: icmp_seq=0 ttl=63 time=1.762 ms
9008 bytes from 192.168.252.37: icmp_seq=1 ttl=63 time=1.392 ms
9008 bytes from 192.168.252.37: icmp_seq=2 ttl=63 time=1.34 ms
9008 bytes from 192.168.252.37: icmp_seq=3 ttl=63 time=1.983 ms
9008 bytes from 192.168.252.37: icmp_seq=4 ttl=63 time=1.284 ms

--- 192.168.252.37 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 1.284/1.552/1.983 ms

EBD-3850#ping ip 192.168.252.37 size 9000
Type escape sequence to abort.
Sending 5, 9000-byte ICMP Echos to 192.168.252.37, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms

 Here is what the links look like between the 3850 and the Nexus.

EBD-3850#sh cdp nei
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
EBD-N3K.ebd.net(FOC1815R0U7)
                 Ten 2/0/47        130            R S I C N3K-C3064 Eth 1/48
EBD-N3K.ebd.net(FOC1815R0U7)
                 Ten 1/0/47        129            R S I C N3K-C3064 Eth 1/47

interface Ethernet1/47
  description 3750E Stack
  switchport mode trunk
  channel-group 48 mode active
!
interface Ethernet1/48
  description 3750E Stack
  switchport mode trunk
  channel-group 48 mode active
!
interface port-channel48
  description 3750E Stack
  switchport mode trunk

interface TenGigabitEthernet1/0/47
 description EBD-N3K
 switchport trunk allowed vlan 1,3-4094
 switchport mode trunk
 switchport nonegotiate
 power inline never
 channel-protocol lacp
 channel-group 47 mode active
!
interface TenGigabitEthernet2/0/47
 description EBD-N3K
 switchport trunk allowed vlan 1,3-4094
 switchport mode trunk
 switchport nonegotiate
 power inline never
 channel-protocol lacp
 channel-group 47 mode active
!
interface Port-channel47
 description EBD-N3K
 switchport trunk allowed vlan 1,3-4094
 switchport mode trunk
 switchport nonegotiate

[Elliot Dierksen]

I forgot to include the versions. 3850 (WS-C3850-12X48U to be specific) is Version 16.06.09 and the Nexus 3064 (N3K-C3064PQ-10GX to be specific) is version 9.3(10).

[balaji.bandi]

I was suspected due to MTU issue, since nexus see 1500 compare to Cat 3850.

Lets look other side  s Ten Gig interface between Cat 3850 and Nexus 3K this works as expected.

When you move the Link to 3850 to nexus using 40gig, from the host you able to reach Esxi ?

From 3850 SVI as source are you able to reach esxi ?

Can you also post show vlan , show stp , show ip route  information for that vlan ? (from nexus and 3850)

Sorry to ask you so much information, there may be small peice of config we missing, even though looks simple - until we see other information its hard to guess

[Elliot Dierksen]

I'll try to make a 40G connection between the 3850 and the Nexus. The part that really confuses me is that the FreeNAS units communicate on the 3850 40G ports. I even tested that with ESXi on the Nexus and FreeNAS on the 3850 while the VM's were running off the NFS data store, so the live traffic was going over the 10G TP link between the Nexus and the 3850.

No worries on asking for additional information. I also think it must be something small, but I haven't been able to figure out what. I have been beating my head against a wall on this for a couple days now.

[balaji.bandi]

Now it's time to go over your Physical topology with the corresponding interface to understand.

[Elliot Dierksen]

OK. To make it even more confusing, doing a 40G link between the 3850 and the Nexus works just fine.

What I was trying to do was have FreeNAS2 connected to Fo1/1/1 in the 3850 and vm-ucs1 connected to Fo1/1/2 in the 3850. Pinging to the FreeNAS worked just fine. I also had CDP neighbor info for the FreeNAS which is running ladvd to do that. I had CDP neighbor info for vm-ucs1 (ESXi 7.0), but no IP traffic worked. I couldn't ping ESXi, no ARP entries. Switching ports, the problem follows the ESXi host with the VIC 1387. I could ping vm-ucs1 when it was connected to the Nexus with traffic going over Po47 which is made up of Te1/0/47 and Te2/0/47 via LACP. I'll show config snippets of both sides. The only thing that is different is that I had to remove 'spanning-tree portfast' on the 3850 side and 'spanning-tree link-type point-to-point' on the Nexus side.

interface Ethernet1/47
  description 3750E Stack
  switchport mode trunk
  channel-group 48 mode active

interface Ethernet1/48
  description 3750E Stack
  switchport mode trunk
  channel-group 48 mode active

interface Ethernet1/49
  description FreeNAS
  switchport access vlan 252

interface Ethernet1/50
  description FreeNAS2
  switchport access vlan 252
  spanning-tree link-type point-to-point

interface Ethernet1/51
  description vm-ucs1 - vmnic3
  switchport access vlan 252
  spanning-tree link-type point-to-point

interface Ethernet1/52
  description vm-ucs2 - vmnic3
  switchport access vlan 252
  spanning-tree link-type point-to-point
!
interface port-channel48
  description 3750E Stack
  switchport mode trunk

EBD-N3K# sh cdp nei
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
vm-ucs1.oau.org     Eth1/6         129    S         VMware ESXi   vmnic2
vm-ucs2.oau.org     Eth1/8         122    S         VMware ESXi   vmnic2
EBD-3850.ebd.net    Eth1/47        176    R S I     WS-C3850-12X4 Ten1/0/47
EBD-3850.ebd.net    Eth1/48        133    R S I     WS-C3850-12X4 Ten2/0/47
EBD-3850.ebd.net    Eth1/49        167    R S I     WS-C3850-12X4 For1/1/2
freenas2.oau.org    Eth1/50        163    H         FreeBSD 12.2- cxl0
vm-ucs1.oau.org     Eth1/51        129    S         VMware ESXi   vmnic3
vm-ucs2.oau.org     Eth1/52        122    S         VMware ESXi   vmnic3

Total entries displayed: 8
EBD-N3K# sh int Eth1/49
Ethernet1/49 is up
admin state is up, Dedicated Interface
  Hardware: 40000 Ethernet, address: 18e7.2897.c0f8 (bia 18e7.2897.c0f8)
  Description: FreeNAS
  MTU 1500 bytes, BW 40000000 Kbit , DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is access
  full-duplex, 40 Gb/s, media type is 40G
  Beacon is turned off
  Auto-Negotiation is turned on  FEC mode is Auto
  Input flow-control is off, output flow-control is off
  Auto-mdix is turned off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  EEE (efficient-ethernet) : n/a
    admin fec state is auto, oper fec state is off
  Last link flapped 00:15:09
  Last clearing of "show interface" counters never
  1 interface resets
  Load-Interval #1: 30 seconds
    30 seconds input rate 352 bits/sec, 0 packets/sec
    30 seconds output rate 608 bits/sec, 0 packets/sec
    input rate 352 bps, 0 pps; output rate 608 bps, 0 pps
  Load-Interval #2: 5 minute (300 seconds)
    300 seconds input rate 248 bits/sec, 0 packets/sec
    300 seconds output rate 144 bits/sec, 0 packets/sec
    input rate 248 bps, 0 pps; output rate 144 bps, 0 pps
  RX
    7454 unicast packets  511 multicast packets  2 broadcast packets
    7967 input packets  67254246 bytes
    7452 jumbo packets  0 storm suppression packets
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    7507 unicast packets  59 multicast packets  588 broadcast packets
    8154 output packets  66929410 bytes
    7401 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause

EBD-N3K# sh spann vl 252

VLAN0252
  Spanning tree enabled protocol rstp
  Root ID    Priority    4348
             Address     0042.5ac4.e600
             Cost        1
             Port        49 (Ethernet1/49)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    8444   (priority 8192 sys-id-ext 252)
             Address     18e7.2897.c101
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po48             Altn BLK 1         128.4143 P2p
Eth1/49          Root FWD 1         128.49   P2p
Eth1/50          Desg FWD 1         128.50   P2p
Eth1/51          Desg FWD 1         128.51   P2p
Eth1/52          Desg FWD 1         128.52   P2p

Here is the same basic info from the 3850.

interface TenGigabitEthernet1/0/47
 description EBD-N3K
 switchport trunk allowed vlan 1,3-4094
 switchport mode trunk
 switchport nonegotiate
 power inline never
 channel-protocol lacp
 channel-group 47 mode active
!
interface TenGigabitEthernet2/0/47
 description EBD-N3K
 switchport trunk allowed vlan 1,3-4094
 switchport mode trunk
 switchport nonegotiate
 power inline never
 channel-protocol lacp
 channel-group 47 mode active
!
interface Port-channel47
 description EBD-N3K
 switchport trunk allowed vlan 1,3-4094
 switchport mode trunk
 switchport nonegotiate
!
interface FortyGigabitEthernet1/1/1
 description FreeNAS2 - NFS
 switchport access vlan 252
 switchport mode access
 switchport nonegotiate
 spanning-tree portfast
!
interface FortyGigabitEthernet1/1/2
 description vm-ucs1 - NFS
 switchport access vlan 252
 switchport mode access
 switchport nonegotiate
end

EBD-3850#sh cdp nei
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
EBD-N3K.ebd.net(FOC1815R0U7)
                 For 1/1/2         161            R S I C N3K-C3064 Eth 1/49
EBD-N3K.ebd.net(FOC1815R0U7)
                 Ten 2/0/47        159            R S I C N3K-C3064 Eth 1/48
EBD-N3K.ebd.net(FOC1815R0U7)
                 Ten 1/0/47        140            R S I C N3K-C3064 Eth 1/47
EBD-3750E-DESK.ebd.net
                 Ten 2/0/48        157             R S I  WS-C3750E Ten 1/0/2
EBD-3750E-DESK.ebd.net
                 Ten 1/0/48        157             R S I  WS-C3750E Ten 1/0/1
ap1.oau.org      Gig 1/0/16        128             T B I  AIR-SAP37 Gig 0
EBD-CR-2960.ebd.net
                 Gig 2/0/9         160              S I   WS-C2960C Gig 0/9
freenas2.oau.org Ten 2/0/45        171               H    FreeBSD 1 ix1
freenas2.oau.org Ten 1/0/45        171               H    FreeBSD 1 ix0
vm-ucs2.oau.org  Ten 1/0/43        160               S    VMware ES vmnic4
vm-ucs2.oau.org  Ten 2/0/43        160               S    VMware ES vmnic5
vm-ucs1.oau.org  Ten 2/0/42        167               S    VMware ES vmnic5
vm-ucs1.oau.org  Ten 1/0/42        167               S    VMware ES vmnic4

Total cdp entries displayed : 13
EBD-3850#sh int fo1/1/2
FortyGigabitEthernet1/1/2 is up, line protocol is up (connected)
  Hardware is Forty Gigabit Ethernet, address is 0042.5ac4.e63e (bia 0042.5ac4.e63e)
  Description: vm-ucs1 - NFS
  MTU 9198 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 40Gb/s, link type is auto, media type is QSFP 40G SR4 SFP
  input flow-control is on, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters 1d04h
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     11438 packets input, 67167317 bytes, 0 no buffer
     Received 3921 broadcasts (263 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 263 multicast, 0 pause input
     0 input packets with dribble condition detected
     10263 packets output, 67473136 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
EBD-3850#sh spann vl 252

VLAN0252
  Spanning tree enabled protocol rstp
  Root ID    Priority    4348
             Address     0042.5ac4.e600
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    4348   (priority 4096 sys-id-ext 252)
             Address     0042.5ac4.e600
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Fo1/1/2             Desg FWD 1         128.62   P2p
Po9                 Desg FWD 4         128.2385 P2p
Po42                Desg FWD 1         128.2418 P2p
Po43                Desg FWD 1         128.2419 P2p
Po47                Desg FWD 1         128.2423 P2p
Po48                Desg FWD 1         128.2424 P2p

[Elliot Dierksen]

Latest update is that with the FreeNAS on a 3850 40G port and the ESXi host on a Nexus 40G, with the 2nd 3850 40G port tied to a Nexus 40G port then everything works as expected. That still doesn't resolve my problem, so I still can't shut down the Nexus. I have some extra Chelsio T580 40G NIC's, so I'll try those as the storage network link instead of the VIC 1387. Even if that works, it still doesn't get me to where I wanted to be because the Chelsio cards are 'unknown' to the CIMC so it will make the fans run at higher speed. Trying to lower the normal noise level in my office was a big part of this project. Sigh... I guess I could look at some card that isn't considered to be 'unknown' by the CIMC, but I still can't figure out why the 1387's won't work correctly with the 3850.

[Elliot Dierksen]

It is definitely the VIC 1387 that is causing the problem. When I put a Chelsio T580 into the 2nd ESXi host and hooked it to the 3850 40G port, it works as expected. What the heck?

[balaji.bandi]

This is very interesting testing you did, Long way to reach here. 

as I am looking bizzard here - there is no support of cat 3850 here.

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/unified-computing-system-adapters/datasheet-c78-736683.html

check any upgrade required to accomplish your task.

or are you ok with your existing arrangements?

[Elliot Dierksen]

Are you saying that a VIC 1387 won't talk any switch but a Nexus? I find that a little hard to believe. Perhaps I need to alter some settings in the 1387 to make it talk to the 3850. That was what I was hoping to do. Prior to having the Nexus, I had an HTC 5800 switch that worked fine with VIC 1227 MLOM adapters.

[balaji.bandi]

As per the document i guess, that is what cisco tested. to confirm if you have access to TAC, contact TAC, so they can guide you in better, they may have local testing and results.

i know its looks blizzard that VIC only talks to nexus, but that is a reality I guess - I have not done any testing like this with VIC.

Just trying to help with my views and what I have done testing the past.