Will configuring unused port on Fabric add adapters for vNICs?

jdish67 · ‎05-09-2024

I'm absolutely new to UCS and I've inherited a system that is discontinued and out of support. I've been tasked with moving 40TB of data off this UCS/vSphere system and onto a newer system supported by our server team. (Server team is unfamiliar with the UCS as well.) We're not sure how to move this volume of data with the current design, and hoping for some help/clues from this community. Apparently our physical path requires MTU 9000 but we're out of physical adapters. In short, is it possible to configure an unused port on my fabric interconnect so that it makes more physical adapters available to vSphere? If so, can I make it an uplink port that's connected to the same switch, or will I need to use a different switch?

My setup: A 5108 AC2 with five servers. It's using two 6324 fabric interconnects that are connected to a Nimble. Each FI has 5 ports: One that goes to the uplink switch, an empty one, one that goes to Nimble controller A, one that goes to Nimble controller B. There's also an unused 'Scalability' port that's labeled Ethernet 1/5/1 thru 1/5/4. (I assume these won't help.)

Each FI is connected to a 3750X switch stack which is port-channeled to my router and then onto the new server farm where we'd like to move all this data. The server team tells me I need the entire path to be MTU 9000, but our first hurdle is on vSphere, where I have no more available physical adapters. Any suggestions?

Steven Tardy · ‎05-10-2024

Don't need another physical switch uplink. Should be easy-peasy (if one is familiar with UCS). There is a pretty good guide out there written by Cisco TAC:

https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-b-series-blade-servers/117601-configure-UCS-00.html

Configure UCS Global QoS policy to MTU 9000 (or better yet 9216)
Configure UCS vNIC of each Service Profile to be MTU 9000.
Configure ESXi vSwitch to allow MTU 9000.
Configure OS NIC to use MTU 9000.
`ping` the new server with MTU 9000 and it should work.
(Careful with the `ping` syntax to make sure you are testing what you think you are testing.)

jdish67 · ‎05-13-2024

Hi Steven, Thanks for the reply.

If I understand correctly, you're saying I should change the MTU rather than add adapters/vNICs -- correct? Won't changing the MTU affect the other traffic on that path? The other traffic on the path is using an MTU of 1500, and since it's a production network requiring 24/7 uptime I'm uncomfortable making such a change without knowing it's full effects. That's why I was looking for a way to add another vSwitch ... but I can't add another vSwitch because there are no unused vmnic's, and there aren't any available physical adapters. See the screenshot below taken from vSphere:

Looking thru my UCS 5108 documentation, it looks like I can add vNICs but I'm a bit confused about where to start -- the vNIC template or the LAN connectivity policy? Is this the right place?

There are several options here... should I be creating a new MAC pool or using MACs from one of the other MAC pools? (I'd use the vNIC template but that template uses a 1500 MTU, which is what I want to change.) I assume I could copy the adapter, QOS, and Network control policies from the other vNICs that are working? Any clues much appreciated.

Steven Tardy · ‎05-14-2024

Yes. Adding a new vNIC to be added to a new vSwitch is a good option. When adding a new vNIC there is a non-obvious "Use vNIC Template" checkbox (why that checkbox isn't the second config option, the world may never know) which makes most of the configuration go away on that page in the screenshot. When you add a vNIC the Service Profile will go into pending reboot status where the vNIC will get added on next host reboot. After reboot a new vmnic (Physical adapters) will be seen by the OS.

Typically you'd create the new vNIC template with MTU 9000. Then add a new vNIC to the LAN Connectivity Policy (but not everyone uses LCP). Or add a new vNIC to the Service Profile Template (but not everyone uses SPT). Or add a new vNIC to the Service Profile.

I would re-use most settings your existing vNICs. Sucks that UCS Copy doesn't do what Copy does for 99% of the computer interface world and let you paste to make a duplicate. So you have to stare and compare or take screenshots of your existing vNIC template to make sure everything else matches.

I would re-use the same MAC pool. If the MAC pool doesn't have enough free MAC addresses, then add another MAC block to the existing pool. More MAC pools is more complexity with little benefit and I prefer to keep things simple. A MAC pool for iSCSI-A with a AA: prefix and iSCSI-B with a BB: prefix is cool and helps troubleshooting, but isn't required by any means.

You can configure the new vNIC to use a dedicated VLAN and a dedicated physical uplink out of the UCS Fabric Interconnect. In UCS this is called Disjoint Layer 2 (DJL2). Misconfigured DJL2 has caused more issues than I can count due to non-obvious gotchas which is why I initially recommended using re-using the same physical uplinks and only create new vNIC(s).