Get the Guide - KUBaM: Kubernetes on Cisco UCS made easy!
During a recent deployment of a Kuberenetes environment on Cisco UCS we noted that certain features of the UCS technology which provided unique server management abilities. These allowed us to consume the underlying infrastructure in a programmatic and abstracted fashion to deliver Bare Metal as a Service!
It's probably worth sharing these with a wider audience so I will split these into two blog posts as we have so much to share!
This first post will focus on the UCS Service Profile aspects with the following post (part 2) expanding on some of the UCS integrations such as the Python and Ansible SDKs plus accompanying container networking technology, Contiv.
Cisco UCS Service Profiles
Kubernetes is ultimately a cluster and with this comes the need to assure all of the cluster members are configured identically. UCS uses the concept of the Service Profile where a template of predefined configurations of components such as the network cards, storage connectivity setup, bios settings and many other elements are defined to provide a logical representation of a server. This ‘profile’ can be applied to multiple servers repeatedly thus assuring that the h/w configuration is consistent across the cluster and can be applied in a programmatic fashion. This facilitaes a DevOps methodology for provisioning and consuming the underlying cluster platform in an 'Infrastructure as Code' way.
Consistant Device Naming
One important requirement Container Clusters have is the need for Consistent Device Naming of the network connection configurations within all of the cluster hosts.
When there is no mechanism for the base Operating System to label Ethernet interfaces in a consistent manner, it becomes difficult to manage network connections across the cluster and this aspect is crucial when assuring traceable connectivity during the cluster configuration stage and potential network troubleshooting.
When Consistent Device Naming (CDN) functionality was introduced in Cisco UCS Manager Release 2.2(4), it allowed Ethernet Interfaces to be named in a persistent manner along with predicable nic placement and ordering within the target host Operating Systems which is key for cluster technologies to be robust.
Kubernetes Networking needs all of the servers in the cluster to have consistent placement and naming and the Service Profile using CDN assures this and can be configured as per the image below using the UCSM GUI, or using the UCS Python SDK:
Figure 1: KUBaM Consisyant Device Naming Setup
Storage Profile
To allow flexibility in defining the number of storage disks, roles and usage of these disks, plus other storage parameters, you can create Storage Profiles within UCS Manager. A storage profile encapsulates the storage requirements and is used for one or more service profiles as it specifies all of the storage requirements of a service profile.
This feature allowed us to define a consistent storage configuration and was consumed via the service profile throughout the cluster giving us a known state in a programmatic fashion. It worth noting, without this feature it would take manual intervention of going into the raid tool of each server and verify the settings individually which is a tedious and time consuming job.
Another advantage this storage profile approach gives us is we can see from a central management place, UCS Manager, where the storage profile has been applied successfully which greatly helps in the troubleshooting process.
We use the storage profile to configure the following characteristics via storage policies:
- LUN count and size
- Raid level of the LUN
Figure 2: KUBaM Storage Profile Setup
- Configure the number, type and role of disks in the disk group (via a defined Disk Group Configuration Policy)
Figure 2: KUBaM Storage Policy Setup
Once created, we associate the storage profile with a service profile template. This was achieved within the Python scripts constructed from the UCS Python SDK, and when the service profile is associated to a physical server, the logical LUNs are deployed as physical LUNs.
vMedia duel Image/ISO loading capabilities
PXE technologies are prevalent and successful throughout many enterprises but there are still considerations that will prevent an organization from adopting PXE for bare metal server deployment such as security concerns or network architectures. For these reasons, the unique ability of UCS to load multiple virtual media images to both deploy and customize the provisioning bare metal server may prove to be very appealing to many sys admins as they can now avoid the consequences of implementing PXE in their networks.
The approach we took for this implementation was to create a pre-boot Linux environment as an ISO file, and supplement this with an image file containing the Linux kickstart file for defining the unique custom setting such as IP address and Hostname per server. This can (and will) be improved upon over time by automatically generating the kickstart files but this initial approach overcomes the need for PXE in the network which has great value.
In order to consume these virtual configuration media artifacts, we use the vMedia capability in UCS Manager and the Service Profile as the mechanism to attach these images to the host servers as they cycle through the build process and all of this is done programmatically thanks to the rich UCS Python SDK.
Figure 3: vMedia Policy
Note: A web server for hosting the ISO and image files is required but the alignment of this this with the Service Profile is automated during the Python server build process to assure the correct configuration.
If you would like to know more about our work around many of these aspects, check out KuBAM!
OK, part 1 completed so onto part 2.........