vNICS no connectivity from B-Series M3 to Gateways

danny.hallwood · ‎02-23-2015

Chassis - M5108

FIs - 6248

IOMs - 2204

Uplink > Brocade ICX6450(Layer2) > Cisco 5525x(Gateway sub interfaces)

Hi, I'm setting up a new, and my first UCS, single chassis with four blades to hosts a Hyper-V. The setup is quite straight forward, everything appears to be correct, the service profiles generate, I can make use of the KVM to connect and install 2012R2, the vNICs are generated and i've loaded the USC eNIC drivers, verified the vNICs by MAC address to that in UCSM to ensure the IP being set is on the correct vLAN.

But for all my trying, I cannot uncover where I've misconfigured the stack. When I give the vNICs an IP address in their vLAN they cannot arp the default gateway, windows reports them as ipv4 Connectivity: No network access.

I've have 7vNICs to each blade. As all are exhibiting the same problem i've been concentrating on vLAN 175, which is to be my blade host OS management interface.

Its as if the vLANS are not being forwarded across the uplink, over the FIs to the vNIC. I cant find any tooling such as ping to make use of to half the problem.

My vLAN to vNIC mapping is 1 to 1 per blade, I've tried setting the vLAN to native on the vNIC, no change.

Can anybody help with some debug tips for the uplink?

Im struggling to halve the problem to understand if its broken in the brocade trunk, as the KVM untagged ports do not prove this, of in the fabric interconnects.

The only peculiarity seems to be 'show int brief' from the nxos as these do not show veth 823/824 in vLAN 175.

Though the GUI: Servers>Service Profiles>HV_HOST2>vNICS>vNIC Host_OS>vLANSs>Host_OS does list vLAN 175 checked as its ID.

I cant set the vLANs in nxos :/

I've installed wireshark on the windows host, i can see ARP requests for the gateway leaving with I ping from command line, there is no response received.

I apologise for the lengthy post - trying to give as much detail i can think off -- really appreciate any help you may be able to offer.

Thanks

Danny

-----------

Some of my config which may help :

Concentrating on Server 2, vNIC HOST_OS, network 10.10.172.0/24 .1 is gateway, .102 the host NIC.

Eth1/3 on the FI are my uplinks to the Brocade.

I can see from the GUI: Equipment > Server2 > NICs than NIC-1 is the correct HW address as reported in windows.

It has VIF 823 and 824, and vNIC HOST_OS.

-----------

NXOS shows vLAN 175 Mapped to veth824 on Fabric B ::

175 VLAN0175 active Eth1/3, Veth795, Veth809

Veth823, Veth865

NXOS shows vLAN 175 Mapped to veth823 on Fabric A::

175 VLAN0175 active Eth1/3, Veth795, Veth809

Veth823, Veth865

-----------

Both veth's show as up on the respective FIs::

qsucs1-B(nxos)# show interface vethernet 823

Vethernet823 is up

Bound Interface is port-channel1281

Port description is server 1/2, VNIC HOST_OS

Hardware is Virtual, address is 002a.6afb.3080

Port mode is trunk

Speed is auto-speed

Duplex mode is auto

-----------

qsucs1-A(nxos)# show interface vethernet 824

Vethernet824 is up

Bound Interface is port-channel1280

Port description is server 1/2, VNIC HOST_OS

Hardware is Virtual, address is 002a.6afb.3620

Port mode is trunk

Speed is auto-speed

Duplex mode is auto

300 seconds input rate 0 bits/sec, 0 packets/sec

300 seconds output rate 0 bits/sec, 0 packets/sec

-----------

qsucs1-B(nxos)# show interface brief

--------------------------------------------------------------------------------

Ethernet VLAN Type Mode Status Reason Speed Port

Interface Ch #

--------------------------------------------------------------------------------

Eth1/1 1 eth fabric up none 10G(D) --

Eth1/2 1 eth fabric up none 10G(D) --

Eth1/3 1 eth trunk up none 10G(D) --

.........

--------------------------------------------------------------------------------

Vethernet VLAN Type Mode Status Reason Speed

--------------------------------------------------------------------------------

Veth795 175 virt trunk down nonParticipating auto

Veth798 1 virt trunk down nonParticipating auto

Veth799 1 virt trunk down nonParticipating auto

Veth802 1 virt trunk down nonParticipating auto

Veth804 1 virt trunk down nonParticipating auto

Veth805 1 virt trunk down nonParticipating auto

Veth807 1 virt trunk down nonParticipating auto

Veth809 175 virt trunk down nonParticipating auto

Veth812 1 virt trunk down nonParticipating auto

Veth813 1 virt trunk down nonParticipating auto

Veth816 1 virt trunk down nonParticipating auto

Veth818 1 virt trunk down nonParticipating auto

Veth819 1 virt trunk down nonParticipating auto

Veth821 1 virt trunk down nonParticipating auto

Veth823 1 virt trunk up none auto

Veth826 1 virt trunk up none auto

Veth827 1 virt trunk up none auto

Veth830 1 virt trunk up none auto

Veth832 1 virt trunk up none auto

Veth833 1 virt trunk up none auto

Veth835 1 virt trunk up none auto

Veth865 175 virt trunk down nonParticipating auto

Veth868 1 virt trunk down nonParticipating auto

Veth869 1 virt trunk down nonParticipating auto

Veth872 1 virt trunk down nonParticipating auto

Veth874 1 virt trunk down nonParticipating auto

Veth875 1 virt trunk down nonParticipating auto

Veth877 1 virt trunk down nonParticipating auto

--------------------------------------------------------------------------------

Ethernet VLAN Type Mode Status Reason Speed Port

Interface Ch #

--------------------------------------------------------------------------------

Eth1/1 1 eth fabric up none 10G(D) --

Eth1/2 1 eth fabric up none 10G(D) --

Eth1/3 1 eth trunk up none 10G(D) --

-----------

My Brocade, uplink switch config - this provides also vlan168 which is my KVM Lan.

1/1/1 1GB is a link to the Cisco 5525x with sub interfaces or each vLAN.

1/1/3 and 1/1/4 are my 1GB links to the Management Interfaces on the Fabric Interconnects/KVM (working)

1/2/1 and 1/2/3 are two 10GB links to the Eth3 on each of the Fabric Interconnects

-----------

Running config:

vlan 168 name HW_MGT by port

tagged ethe 1/1/1

untagged ethe 1/1/3 to 1/1/4

management-vlan

default-gateway 10.10.168.1 1

vlan 172 name VHPRDMGT by port

tagged ethe 1/1/1 ethe 1/2/1 ethe 1/2/3

!

vlan 173 name VHDEVSVC by port

tagged ethe 1/1/1 ethe 1/2/1 ethe 1/2/3

!

vlan 174 name VHPRDSVC by port

tagged ethe 1/1/1 ethe 1/2/1 ethe 1/2/3

!

vlan 175 name HOST_OS by port

tagged ethe 1/1/1 ethe 1/2/1 ethe 1/2/3

!

-----------

Show trunk:: Trunk two uplink ports to the FIs 1/2/1 and 1/2/3

Configured trunks:

Trunk ID: 65

Hw Trunk ID: 1

Ports_Configured: 2

Primary Port Monitored: Jointly

Ports PortName Port_Status Monitor Rx_Mirr Tx_Mirr Monitor_Dir

1/2/1 none enable off N/A N/A N/A

1/2/3 none enable off N/A N/A N/A

Operational trunks:

Trunk ID: 65

Hw Trunk ID: 1

Duplex: Full

Speed: 10G

Tag: Yes

Priority: level0

Active Ports: 2

Ports Link_Status port_state

1/2/1 active Forward

1/2/3 active Forward

-----------

Show vLAN:

PORT-VLAN 175, Name HOST_OS, Priority level0, Spanning tree On

Untagged Ports: None

Tagged Ports: (U1/M1) 1

Tagged Ports: (U1/M2) 1 3

Uplink Ports: None

DualMode Ports: None

Mac-Vlan Ports: None

Monitoring: Disabled

Walter Dey · ‎02-23-2015

Hi Danny

Can you please post a diagram of your setup !

- are your UCS FI in Ethernet end host mode

- you are not using Fibrechannel ?

- Normally all vlans are trunked from FI North bound

- How does your service profile look like ? to which fabric are the vnics connected ? A or B ? did you select hardware failover

- is each vlan available on fabric A and B ?

- W2012 sees 7 access interfaces, one vlan per interface ?

- do have North bound vlan L2 connectivity ?

Walter.

danny.hallwood · ‎02-23-2015

Hi Walter, thank you the help

Answers inline:

> Can you please post a diagram of your setup !

Ill draw one, i have a design document with each step detailed ive executed - i can email this in the interim if useful?

- are your UCS FI in Ethernet end host mode

Yes

- you are not using Fibrechannel ?

No - at this stage im deploying once 4 2012r2 DC builds to host a hyper visor, VMs to be hosted on CSV over local disks in the blades.

- Normally all vlans are trunked from FI North bound

I have 6 vlans with gateways on a cisco firewall. There is an intermediary switch between the firewall and the Fabric Interconnects. At the switch my hardware vlan(168) is untagged and presented over cat6 to the management interfaces of the fabric interconnects. The remaining 5 vlans remain tagged and are trunked over two 10Gbit links to eth3 on each Fabric. There are two more vlans in the fabric, 1 for CSV and 1 for Live Migration.

- How does your service profile look like ? to which fabric are the vnics connected ? A or B ? did you select hardware failover

There are interleaved across both, but all vlans are set to failover

- is each vlan available on fabric A and B ?

They are all added in the community space, this should then be available to each fabric.

- W2012 sees 7 access interfaces, one vlan per interface ?

Yes one vlan per vNIC presented to the hosts.

HOST_OS, CSV, LVMGR are type adaptor. There are then four more for VMs which are Adaptor and VM type.

- do have North bound vlan L2 connectivity ?

I have been struggling to test this - vlan 168 untagged at the switch is for sure working as i can connect to the KVM, mounth virtual media, install the OS.

Mapping the MAC address to a NIC in Windows and putting in my HOST_OS vlan, it cannot ping, or arp its gateway on the firewall.

The only drift from the design/ build detail is ive tried setting the vLANS to native. No joy.

Ive set the trunk t the firewall as LACP, no joy.

I'm not using any pinning -- tried this also, no joy.

- Danny

Walter.

danny.hallwood · ‎02-23-2015

Where im struggling, aside it not working! is being able to debug the configuration.

The output from nxos looks ok, to my untrained eye.

Is there anywhere short of from a built vNIC on a blade to run connectivity tests from, inside the Fabric on its command interfaces? It would be great to half the problem to concentrate debug efforts in a more targeted manner.

- Danny

Manuel Velasco · ‎02-23-2015

Can you confirm that you can ping between servers with vnics pointing to the same fabric and same vlan? How many uplinks do you have configured in UCSM for each fabric? Are you doing any type of disjoint layer 2?

danny.hallwood · ‎02-24-2015

HI Manuel,

Thank you for the reply.

I have tested between blade 3 and 4 and can confirm on the HOST_OS vNICs they can ping each other. This is in my vLAN 175.

Each Fabric has 1 Uplink, both to the same switch, which itself connected to a firewall that hosts the gateway IPs.

I think I have a spanning tree problem - both of my 10GB uplink ports are showing as blocked on the switch. I can't swear this was the case yesterday, my debug efforts may have triggered this, but certainly i need to clear the block first!

- Danny

danny.hallwood · ‎02-24-2015

Walter, Manuel

It is alive! ;)

Turns out, amazingly my original configuration with No LAN Pinning, No port channel, No trunk from the switch to the FI's was working.

My testing on Blade-2 was a poor choice, the blade was stuck at 84% configured, which I did not notice till today drilling around the GUI, after trying to debug why 3 to 4 could talk to each other but not to 2.

Removing the trunk/channel/pinning all sprang into life for 3 and 4 to reach their Gateways.

I'll rebuild Blade-2 from the template and hopefully all should be well

Thank you for your help

Danny

Manuel Velasco · ‎02-24-2015

I am glad it is working now.

danny.hallwood · ‎02-23-2015

OK, not proud - but hopefully gives a flavour.

2 minute sketch of the Environment.

Walter Dey · ‎02-23-2015

Hi Danny

Nice diagram ! can you add, which fabric A or B your vnics are connected to ?

Because mgt. and KVM are working (native vlan), something must be wrong with the trunk !

Therefore as Manuel suggested, try to ping 2 hosts, which are on the same vlan, and therefore L2 switched on the UCS FI.

Are all the vlans on the uplink trunk up ?

http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-manager/116095-configure-ucs-upstream-port-channel-00.html

UCS port channel configuration is statically set to Link Aggregation Control Protocol (LACP) mode active. This configuration cannot be modified; therefore, all upstream port-channel configurations must adhere to LACP mode active as well. Alternatively, you can configure the upstream switchports for LACP mode passive.

CLI Configuration

Here is a sample of the UCS interface configuration that cannot be modified:

UCS1-B(nxos)# show run interface eth1/19

!Command: show running-config interface Ethernet1/19
!Time: Fri Oct 12 20:25:59 2012

version 5.0(3)N2(2.11)

interface Ethernet1/19
  description U: Uplink
  pinning border
  switchport mode trunk
  switchport trunk allowed vlan 1,107,110-111,115,119,
      168,175,179,183,200-201,279,283,379,383,555-556
  channel-group 100 mode active 
  no shutdown

danny.hallwood · ‎02-24-2015

Hi,

I have tested between blade 3 and 4 and can confirm on the HOST_OS vNICs they can ping each other. This is in my vLAN 175.

I had no port channel configured at all, did not think id need this as only one physical uplink per Fabric to the switch.

I have now configured Port Channel on both a/1/3 and b/1/3 which are my uplink ports.

What I have now noticed, though cannot be sure if present yesterday, is both the 10GB uplink ports on the brocade switch are "blocked", so i suspect a spanning tree issue. Certainly needs fixing first!

---

Some output from the recommended tips,

* NXOS queries around trunk ports, shows vLANS assigned

* Brocade switch LACP configuration on my two uplink ports to the FIs

* VLAN for 168(KVM working) 175(HOST_OS not)

* Interfaces showing there is likely a ST issue.

Thanks Danny!

FI - NXOS - A: Trunk Queries 
!Command: show running-config interface Ethernet1/3
!Time: Tue Feb 24 13:29:55 2015
version 5.2(3)N2(2.23d)

interface Ethernet1/3
  description U: Uplink
  pinning border
  switchport mode trunk
  switchport trunk allowed vlan 1,168-177
  udld disable
  channel-group 1 mode active
  no shutdown

!Command: show running-config interface port-channel1
!Time: Tue Feb 24 13:35:58 2015
version 5.2(3)N2(2.23d)

interface port-channel1
  description U: Uplink
  switchport mode trunk
  pinning border
  switchport trunk allowed vlan 1,168-177
  speed 10000

  port-channel1 is up
  Hardware: Port-Channel, address: 002a.6afb.362a (bia 002a.6afb.362a)
  Description: U: Uplink
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s
  Input flow-control is off, output flow-control is off
  Switchport monitor is off 
  EtherType is 0x8100 
  Members in this channel: Eth1/3

— Switch LACP Configuration: 

ICX6450-24P Switch>show link-aggregate ethernet 1/2/1
System ID: 748e.f8b1.a2e0
Default Key:        2
Port  [Sys P] [Port P] [  Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/2/1       1        1    10000   Yes   S   Agg  Syn  Col  Dis  No   No   Ina

ICX6450-24P Switch>show link-aggregate ethernet 1/2/3
System ID: 748e.f8b1.a2e0
Default Key:        2
Port  [Sys P] [Port P] [  Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/2/3       1        1    10000   Yes   S   Agg  Syn  Col  Dis  No   No   Ina

— Switch VLAN 175 and 168: 

PORT-VLAN 175, Name HOST_OS, Priority level0, Spanning tree On
 Untagged Ports: None
   Tagged Ports: (U1/M1)   1 
   Tagged Ports: (U1/M2)   1   3 
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 168, Name HW_MGT, Priority level0, Spanning tree On
 Untagged Ports: (U1/M1)   3   4 
   Tagged Ports: (U1/M1)   1                                      
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled

Port    Link    State   Dupl Speed Trunk Tag Pvid Pri MAC            Name 
1/1/1   Up      Forward Full 1G    None  Yes N/A  0   748e.f8b1.a2e0      
1/1/2   Down    None    None None  None  No  1    0   748e.f8b1.a2e1     
1/1/3   Up      Forward Full 1G    None  No  168  0   748e.f8b1.a2e2      
1/1/4   Up      Forward Full 1G    None  No  168  0   748e.f8b1.a2e3          .........
.......    
1/2/1   Up      Blocked Full 10G   None  Yes N/A  0   748e.f8b1.a2f9      
1/2/2   ERR-DIS None    None None  None  No  1    0   748e.f8b1.a2fa      
1/2/3   Up      Blocked Full 10G   None  Yes N/A  0   748e.f8b1.a2fb      
1/2/4   ERR-DIS None    None None  None  No  1    0   748e.f8b1.a2fc      
mgmt1   Down    None    None None  None  No  None 0   748e.f8b1.a2e0