HX240C M4- use as stanalone with Ubuntu 20.x - networking

[alanraczek11]

We have 3 HX240C M4's that were logically disconnected from the UCS manager (although you can still see them there). We need to use these servers independently to build a Kubernetes cluster. See the image below: 

Upon installation of Ubuntu it seems Ubuntu does not have drivers for the built in NIC's on the UCS rack servers. So it seems we need to use the fiber interconnects after all. So I believe I have to re-integrate the servers back into UCS somehow and then set up their IP addresses. I am absolutely a novice with UCS, seems a little daunting to me. Would anyone be able to detail how this all works and guide me on how to accomplish this? 

Once the networking is in place the developers building this K8 cluster will also install ceph for storage. Any help would be appreciated. I DO have a TAC case pending but want a little more insight. Just telling me to read the doc doesn't help much.

Thank you VM.

...ar

[itm.i11.team]

Hello,

I do have no experience with the HX240 M4 but as these servers are more or less the same as a  C240 M4 I guess you can treat them the same. So you could use them in standalone mode without connection to the Fabric Interconnects.

Every server has a CIMC (out-of-band managment) you can use instead of the UCS Manager running on the Fabric Interconnects.

Cisco also offers a Linux driver package (e.g. https://software.cisco.com/download/home/286281356/type/283853158/release/4.0(2i)) which might help you get the NICs running.

Regards

 Martin

[Kirk J]

Greetings.

From an OS perspective, there is no difference between standalone and UCSM integrated.

The UCSM integration helps automate the creation of the vnics and vHBAs instead of you manually creating them through the CIMC GUI on each server.  The actual vnic/NIC presented through the BIOS and to the OS is exactly the same.

If you have issues from a driver perspective in standalone mode you will also have an issue in integrated mode.

Sounds like you just need to add drivers  ucs-cxxx-drivers-linux.4.1.2c.iso 

Kirk...

[alanraczek11]

Kirk-

If I do a lshw -C in Ubuntu I see what amounts to be 10 network interfaces, product = VIC Ethernet NIC, capabilities: bus_master cap_list ethernet physical fibre 10000bt-fd. So I assume Ubuntu DID recognize the fiber interfaces. If we could use the fiber that would be great but I am not sure how the network sees the IP's on these interfaces via the fiber interconnects. I added static IP addresses via Ubuntu but these did not work. Can't ping. 

Is my assumption corrrect? That Ubuntu is seeing the fiber interfaces? We'd prefer to use them for the 40Gb throughput, but if it's too difficult we could try loading the other drivers. Or am I totally missing something here?

...Alan

[Kirk J]

If you haven't created vlan subinterfaces for tagging vlan traffic (https://wiki.ubuntu.com/vlan) , then you will want to make sure you mark the LAN at the service profile level (vnic, lan) as native to see if this is working correctly.

Also, see if you are learning any mac addresses at the FI level (connect nxos, show mac address-table vlan xx) ?

What does ethtool eth0,  ethtool -S eth0, or ethtool -i eth0 show?

Kirk...

[alanraczek11]

Some Screens:

That's an out of the box Ubuntu 18.04.5 install (note drivers in the download only go to 18.04.4). I ran the cmd on a configured interface. And can you lead me through how I get  " then you will want to make sure you mark the LAN at the service profile level (vnic, lan) as native to see if this is working correctly."? Not sure where I should be looking.

[Kirk J]

That will be set either under you service profile, network, vnic list, a service profile template, or a linked vnic template.

The native vlan option, selects 'untagged' traffic that has no vlan ID specified, add tags it with the vlan marked as native.

Kirk...

[alanraczek11]

Kirk-

Thank you for all the help BTW.

I can't get properties of the Service Profiles, I get this:

Ty.

[alanraczek11]

Here are the connectivity policies, none:

[Kirk J]

You appear to have a hyperflex system involved (or at least your service profile was created by a HX installer), so you will have a suborg.

You will need to scroll further down the screen until you find the suborg section, expand your suborg, and then find the vnic templates.

Kirk...

[alanraczek11]

Her is what is going on. We DID have this set up as a Hyperflex system but I decommissioned the servers as we needed to use the hardware for a Kubernetes cluster. A developer installed Ubuntu on all servers and we are trying to get the networking going. And I ended up here. Would ne nice to use the fiber interfaces as they are 40Gb. I will check what you said in your post above.

[alanraczek11]

I spent my day digging in the interface, not sure if the below is what you mean by "native" VLAN. To be clear there were no VLAN mods made in the od. We obviously have VLAN's on our switches. I drilled down through /lan/policies/root/sub-organizations/<Lab Name>/vNIC Templates. There is a native button there. Just why would I select native? Sorry i'm ignorant here, just wanting to understand.

[alanraczek11]

Any thoughts?

[Kirk J]

Yes, that is the native vlan designation.  You actually set that under the related vnic template under edit vlans.

If you aren't tagging at the os, then your traffic will be untagged, and will likely not get correctly switched to other devices that are in the same vlan, including the upstream default gateway, unless you add the native vlan tag, which will auto tag your untagged traffic with the vlan you designate as 'native'

Kirk...

[alanraczek11]

"that is the native vlan designation."

I know what native VLAN means in a switch. There is only one. So in UCS manager is this concept different? What I get from your information is I should mark all the templates VLAN's as native? Since I am not tagging in the OS. 

...Alan