Cisco UCS B-Series 3.1 - Unable to ping GW via ESXi hosts/All Host can ping within chassis

jrobinson78 · ‎05-01-2017

Hoping someone can resolve this issue and point out an obvious flaw.

UCS B-Series all ESXi hosts on blades can ping each other via mgmt network (tagged 510) although they cannot ping the mgmt GW.

The FI indicate the following configuration on the 6248s:

nterface Ethernet1/1
description S: Server, Port-channel 1025
no pinning server sticky
switchport mode fex-fabric
switchport trunk native vlan 4044
switchport trunk allowed vlan 4044,4047
fex associate 1
udld disable
channel-group 1025
no shutdown

interface Ethernet1/2
description S: Server, Port-channel 1025
no pinning server sticky
switchport mode fex-fabric
switchport trunk native vlan 4044
switchport trunk allowed vlan 4044,4047
fex associate 1
channel-group 1025
no shutdown

interface Ethernet1/3
description S: Server, Port-channel 1025
no pinning server sticky
switchport mode fex-fabric
switchport trunk native vlan 4044
switchport trunk allowed vlan 4044,4047
fex associate 1
channel-group 1025
no shutdown

interface Ethernet1/4
description S: Server, Port-channel 1025
no pinning server sticky
switchport mode fex-fabric
switchport trunk native vlan 4044
switchport trunk allowed vlan 4044,4047
fex associate 1
channel-group 1025
no shutdown

I understand the B-series will pass control plane traffic over vlans 4044 and 4047, however I feel like the predefined UCS server port policy is applying the config and limiting data traffic upstream via the mgmt vlan 510.

I have a later instance of UCS 2.2 I believe and when I configure or select the server port via the GUI the configuration is as follows:

interface Ethernet1/4
description S: Server, Port-channel 1025
switchport mode fex-fabric
fex associate 1
channel-group 1025
no shutdown

As you can see its lacking the trunk configs specific to the SAM vlans.

My question is, would this inhibit connectivity to the ESXi hosts?? If so, how do I adjust the config??

Thanks for you assistance in advance.

I attached a config from one of the 6248s for additional reference in hopes to gain clarification or an answer.

-Jason

Wes Austin · ‎05-01-2017

If you can ping between hosts on the same L2 domain, the UCS is passing traffic correctly. Where is your layer 3 gateway? Is it the upstream switch?

On Server interfaces, I would not expect VLAN tagging, as this occurs on the vethernet interfaces:

interface Vethernet863
  description server 1/3, VNIC eth0-mgmt-a
  switchport mode trunk
  no lldp transmit
  no lldp receive
  no pinning server sticky
  pinning server pinning-failure link-down
  no cdp enable
  switchport trunk allowed vlan 510  <<<<<<<<<<<<<<<<<<<<<<<<<<
  bind interface port-channel1281 channel 863
  no shutdown

Are you also tagging a VLAN in ESXi? If so, you either need to tag in ESXi or in the UCS. Double tagging will cause connectivity issues.

I also see VLAN 510 on the uplink port-channel, which is correct.

interface port-channel11
  description U: Uplink
  switchport mode trunk
  pinning border
  switchport trunk allowed vlan 1,4,11,20,30,32,40-45,51-52,56,60-61,90,110,123,                                                                                                                      126,510,520,530-533,900-903,905-906
  speed 10000

Make sure you are learning the mac addresses for the ESXi management interfaces in the correct VLAN on the UCS and upstream switch. If you are, you likely have an upstream switching/routing issue to investigate.

Wes

jrobinson78 · ‎05-01-2017

I am tagging on the host as well as the mgmt vnic in UCS.

I have looked at the mac address table in the 7ks and 6248s and see no MAC associated to the ESXi hosts.

The GW is on a Firewall upstream, although this is just a layer 2 issue currently I am not concerned at the moment pinging outside of the mgmt network.

So, having the FIs with the "switchport trunk allowed VLAN 4044, 4047" is okay?

Kirk J · ‎05-01-2017

Greetings.

Just to be clear, the only UCSM/service profile 'tagging' you can enable is when you designate a specific vlan as 'native' on a vnic. You can only do this for one vlan per vnic, and use cases are usually for bare metal windows hosts, iscsi vnics for boot from san, and PXE booting nics.

Please try the following again:

On each vnic in your service profile, make sure no vlans are marked as 'native', but have the correct allowed vlans in the vlan's list.
Using ESXi DCUI, pick one vnic to use for managment, and confirm the mac address it is using. Set the vlan tagging in the esxi DCUI mgt network setup for your mgmt network vlan. Please confirm the MAC that vmware DCUI mgmt network setup shows actually matches the vnic you think it is in the service profile's MAC.

Not seeing the learned MACs, especially for the VMK0, which inherits the VMNIC0's hardware MAC, usually means the tagged traffic does not match the allowed vlan list.

You can ignore the 'server' port programming that the IOMs connect to on the FI's, as those VLANs are reserved for internal communication, and default vlans for IOM, Server CIMC discovery, etc.

It would be helpful if you can enable/disable ports for your esxi MGT vmnics, such that we can get one host that is pinned to the B side, and one host to the A side, and do a test ping. If only hosts pinned to the same side can communicate, then we have an issue with the VLAN pinning or port-channel programming between the FI uplinks and the upstream switch.

Please indicate what your expected esxi MGMT vlan is.

On each FI please run the following:

#connect nxos

nxos#show vlan id xx

Please confirm the uplink ports listing this vlan are the only ports/port-channel you expect the vlans on, and that you don't have a disjoint layer2 network.

Thanks,

Kirk...

jrobinson78 · ‎05-02-2017

Thanks Kirk.

I confirmed the following as you suggested.

On each vnic in your service profile, make sure no vlans are marked as 'native', but have the correct allowed vlans in the vlan's list.
Using ESXi DCUI, pick one vnic to use for managment, and confirm the mac address it is using. Set the vlan tagging in the esxi DCUI mgt network setup for your mgmt network vlan. Please confirm the MAC that vmware DCUI mgmt network setup shows actually matches the vnic you think it is in the service profile's MAC.

When I access the shell of the ESXi hosts I can see the VMKs having the same MACs as the VNICs. Is this okay? When I go in the shell and remove/add so VMKs take on a new MAC it doesnt resolve anything. Wouldnt I want to see the VNIC MAC on the 6200s MAC address list? When I do the change in the shell "remove/add" and the MAC is then different than the VNIC, what I see on the 6200 is that MAC. Wouldnt I want the MAC address list on the 6200 to show the ESXi host mgmt VNIC MACs that I have selected in order to ping the IP associated to the host from the 6200s?

Thanks again for the assistance.

jrobinson78 · ‎05-01-2017

Let me step back, just a bit in hopes to make a clearer picture.

The 6248s directly connected to the B-series chassis. Those fabric connections are the only connection providing connectivity to the ESXi hosts.

I cannot ping a host on the chassis from the 6248 and or the mgmt switch I have in the network. All other devices can be reached via the mgmt network. Although as I am sure you are completely aware the other systems, 7Ks, 6248s have mgmt ports directly connected to the mgmt sw. The chassis relies solely on the fabric uplinks IOM to 6248s.

All the hosts can ping each other on the chassis but I suspect thats because its learning the MACs of each host via the blade connection to backplane of the chassis.

jrobinson78 · ‎05-01-2017

I adjusted tagging.

Only tagging at the UCS. Still not able to ping the ESXi hosts from the 6248s.

The 6248s are not seeing the MAC addresses of the host vnics.

I see no faults on the FI to the IOMs.

I am still curious, having the server ports indicating "switchport trunk allowed vlan 4044, 4047" is not a problem??

I find it very odd. I cannot adjust and or add vlan tag to the port on the 6248. The server port policy is apparently adding them.

If this is a good config, and its associated to control plane traffic, why display it in this fashion? Its extremely confusing, goes against simple understanding. Previous versions of UCS did not display server port in this fashion which keeps me returning to the "delta".

I appreciate the assistance Wes.

Tom Lundy · ‎05-15-2017

Hi there,

I had a very similar problem with a lab I am building. Just to be clear I am a novice at this kind of data center config.

I have a UCS chassis connected to 2 5K's and ESXI hosts on blades. I could not ping the hosts either from mgmt. It turned out to be a few things, first I had tagged the vlans on both the ucs service profile vlans and the ESXI host config, I removed the native tagging on the service profile and left the tagging on the ESXI hosts.

Then I also had to make sure the vnic MAC addresses were not duplicates, this was done by creating separate MAC pools on service profile.

And I had to make sure that the vlan gateways were all ok on my 5K's, I had the ports as trunks allowing those vlans like you have.

I have the connections between the 5K's and the UCS as Uplnk Ports, not sure if thats what you had.

These few changes sorted it out for me anyway. Not sure if it helps, best of luck.

Tom.