cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Linux Bonding with Cisco VIC in Stand Alone Mode - RHEV Use Case

5891
Views
10
Helpful
0
Comments

One of the unappreciated features of the Cisco C-Series server is the ability to carve up the VIC into multiple virtual adapters.   This allows you to apply things like jumbo frames and COS to adapters on the same physical link.  It also makes dealing with trunked VLANs easier, eliminating the need to use VLAN interfaces in the OS.

In this document I am using a C220 M3 with a 1225 card for a Red Enterprise Virtualization Hypervisor host.  This is an excellent example of the flexibility of the Cisco VIC because on a single 2 port card I need to support a management network, a storage network, and a Linux bridge for VM networks.   I have also successfully used this feature of the VIC to create Ceph and SwiftStack servers with 4 10GBE ports and 4 way LACP bonds, with no issues.   There is an example of a more complex setup in the linked CiMC CLI how-to below.

This is what we want the layout of the server to look like.  There are 3 important layers, the L2 switches, the VIC, and the Linux kernel configuration.  I will walk through each of these.

Cisco VIC - vnic configuration.png

Exploded Server View:

RHEV-Server-Networks.png

It will look like this in RHEV-M:

RHEV-M Networks - vic.png

Related Documents:

*** Important bonding mode support information:

You should read and understand this KB Article from Red Hat, it is generally applicable to all Linux Distros.

https://access.redhat.com/solutions/67546

This is important to understand in the context of UCS.   KVM with Linux bridge only supports LACP/802.3ad bonding mode for load distribution/balancing.   The VIC in stand-alone mode fully supports LACP/802.3ad bonding mode for KVM, however, the VIC connected to Fabric Interconnects cannot be configured in 802.3ad bonds, only mode 5 and 6 - alb/tlb, which KVM does not support.  Unfortunately OVS no longer supports alb/tlb either so if you are using VIC+UCS, you are stuck using active/passive failover bonds or Fabric Failover.


Nexus 5596 Port Channel Configuration


This is a standard 2 switch with VPC peer link configuration.

Switch-A:

ATCOSPN5KA# feature lacp

ATCOSPN5KA# show feature | grep lacp

lacp                  1         enabled

ATCOSPN5KA# sho run int e1/36

!Command: show running-config interface Ethernet1/36

!Time: Fri Aug 12 20:52:02 2011

version 6.0(2)N2(5)

interface Ethernet1/36

  description VPC Port-channel to RHEV-Host-2

  switchport mode trunk

  channel-group 32 mode active

ATCOSPN5KA# sho run int po 32

!Command: show running-config interface port-channel32

!Time: Fri Aug 12 20:52:17 2011

version 6.0(2)N2(5)

interface port-channel32

  description To RHEV-Host-2

  switchport mode trunk

  vpc 32


Cisco VIC Configuration


It is much easier if you use the CIMC CLI for these activities rather than the GUI.

I created a CIMC CLI reference document which you can reference (CIMC CLI Command Reference )

Discover existing NIC configuration -

vietmeik@Skarn (~)$ ssh admin@10.255.152.95

admin@10.255.152.95's password:

C220-FCH1731V2GJ# scope chassis

C220-FCH1731V2GJ /chassis # sh adapter

PCI Slot Product Name   Serial Number  Product ID     Vendor

-------- -------------- -------------- -------------- --------------------

1        UCS VIC 1225   FCH1731J48Y    UCSC-PCIE-C... Cisco Systems Inc


NOTE - Depending on the server platform and VIC arrangement, MLOM vs PCI card for example the, Adapter number will be different.


C220-FCH1731V2GJ /chassis # scope adapter 1

C220-FCH1731V2GJ /chassis/adapter # show host-eth-if

Name             MTU  Uplink Port MAC Address       CoS VLAN PXE Boot iSCSI Boot

---------------- ---- ----------- ----------------- --- ---- -------- ----------

eth0             1500 0           C0:67:AF:31:A3:31 0   2    disabled disabled

eth1             1500 1           C0:67:AF:31:A3:32 0   2    disabled disabled

eth5             1500 1           C0:67:AF:31:A3:35 0   NONE disabled disabled

eth4             1500 0           C0:67:AF:31:A3:36 0   NONE disabled disabled

eth3             9000 1           C0:67:AF:31:A3:37 0   3003 disabled disabled

eth2             9000 0           C0:67:AF:31:A3:38 0   3003 disabled disabled

These are the completed VNICs, I will walk through setting them up in the following steps.  A note on this output - the VNICs are not listed in the PCI device order that you will see in the Linux kernel, nor are the MAC addresses allocated in order.  This can be confusing later as you will see (A CDET is open to fix this).

Important things to note in this output, the VLAN assignment and MTU:

C220-FCH1731V2GJ /chassis/adapter # scope host-eth-if eth0

C220-FCH1731V2GJ /chassis/adapter/host-eth-if # sh detail

Name eth0:

    MTU: 1500

    Uplink Port: 0

    MAC Address: C0:67:AF:31:A3:31

    CoS: 0

    Trust Host CoS: disabled

    PCI Order: 0

    VLAN: 2

    VLAN Mode: TRUNK

    Rate Limiting: OFF

    PXE Boot: disabled

    iSCSI Boot: disabled

    Channel Number: N/A

    Port Profile: N/A

    Uplink Failover: N/A

    Uplink Failback Timeout: N/A

These are some other interesting features available that could be useful but won't be investigated in this document:.

CoS: 0

Trust Host CoS: disabled

Rate Limiting: OFF

Rate Limiting:

An egress rate limiter which you can define the upper BW limit the vnic can have.  Might be useful for chatty protocols or greedy workloads like VMotion.

CoS:

Ensure that the workloads are treated to the proper QOS SLA within the network.  This is an outbound marking, from server to network.  This is only useful when the QOS is properly set from source to destination.

Setup vNICs:

By default there will only be 2 vnics:

C220-FCH1731V2GJ /chassis # top

C220-FCH1731V2GJ /chassis # scope adapter 1

C220-FCH1731V2GJ /chassis/adapter # show host-eth-if

Name             MTU  Uplink Port MAC Address       CoS VLAN PXE Boot iSCSI Boot

---------------- ---- ----------- ----------------- --- ---- -------- ----------

eth0             1500 0           C0:67:AF:31:A3:31 0   NONE disabled disabled

eth1             1500 1           C0:67:AF:31:A3:32 0   NONE disabled disabled

There are 2 steps -

  1. Configure existing NICS
  2. Add new NICS

Modify existing NICS:

C220-FCH1731V2GJ /chassis/adapter # scope host-eth-if eth0

C220-FCH1731V2GJ /chassis/adapter/host-eth-if # set order 0

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan 2

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan-mode trunk

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# commit

C220-FCH1731V2GJ /chassis/adapter/host-eth-if # exit

C220-FCH1731V2GJ /chassis/adapter # scope host-eth-if eth1

C220-FCH1731V2GJ /chassis/adapter/host-eth-if # set order 1

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan 2

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan-mode trunk

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# commit

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# exit

C220-FCH1731V2GJ /chassis/adapter/ *#


Add and modify new NICS:

Add:

C220-FCH1731V2GJ /chassis/adapter *# create host-eth-if eth2

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# exit

C220-FCH1731V2GJ /chassis/adapter *# create host-eth-if eth3

C220-FCH1731V2GJ /chassis/adapter/host-eth-if # exit

C220-FCH1731V2GJ /chassis/adapter *# create host-eth-if eth4

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# exit

C220-FCH1731V2GJ /chassis/adapter *# create host-eth-if eth5

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# exit

C220-FCH1731V2GJ /chassis/adapter *#

Modify:

C220-FCH1731V2GJ /chassis/adapter *# scope host-eth-if eth2

C220-FCH1731V2GJ /chassis/adapter/host-eth-if # set uplink 0

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set order 2

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan 3003

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan-mode trunk

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set boot disabled

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set mtu 9000

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# exit

C220-FCH1731V2GJ /chassis/adapter *# scope host-eth-if eth3

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set uplink 1

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set order 3

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan 3003

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan-mode trunk

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set boot disabled

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set mtu 9000

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# exit

C220-FCH1731V2GJ /chassis/adapter # scope host-eth-if eth4

C220-FCH1731V2GJ /chassis/adapter/host-eth-if # set uplink 0

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set order 4

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan-mode trunk

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set boot disabled

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# exit

C220-FCH1731V2GJ /chassis/adapter *# scope host-eth-if eth5

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set uplink 1

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set order 5

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set vlan-mode trunk

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# set boot disabled

C220-FCH1731V2GJ /chassis/adapter/host-eth-if *# exit

C220-FCH1731V2GJ /chassis/adapter/ *# commit

Committed host-eth-if eth0 settings will take effect upon the next server reset

Committed host-eth-if eth1 settings will take effect upon the next server reset

Committed host-eth-if eth2 settings will take effect upon the next server reset

Committed host-eth-if eth3 settings will take effect upon the next server reset

Committed host-eth-if eth4 settings will take effect upon the next server reset

Committed host-eth-if eth5 settings will take effect upon the next server reset

C220-FCH1731V2GJ /chassis/adapter # exit

C220-FCH1731V2GJ /chassis/ # power cycle

This operation will change the server's power state.

Do you want to continue?[y|N]y


C220-FCH1731V2GJ /chassis/ #

- CIMC vNIC Configuration is complete -


C220-FCH1731V2GJ /chassis # scope adapter 1

C220-FCH1731V2GJ /chassis/adapter # show host-eth-if

Name             MTU  Uplink Port MAC Address       CoS VLAN PXE Boot iSCSI Boot

---------------- ---- ----------- ----------------- --- ---- -------- ----------

eth0             1500 0           C0:67:AF:31:A3:31 0   2    disabled disabled

eth1             1500 1           C0:67:AF:31:A3:32 0   2    disabled disabled

eth5             1500 1           C0:67:AF:31:A3:35 0   NONE disabled disabled

eth4             1500 0           C0:67:AF:31:A3:36 0   NONE disabled disabled

eth3             9000 1           C0:67:AF:31:A3:37 0   3003 disabled disabled

eth2             9000 0           C0:67:AF:31:A3:38 0   3003 disabled disabled

Note - if you have Serial Over LAN enabled you could at this point type "con host" and watch the reboot without needing the java KVM.



Setting up interfaces in host OS:


Basic bonding configuration is covered in this article:

We won't repeat the basics in this document but cover an important aspect of bonding with nested VNICS on the Cisco VIC.  When using LACP bonding the first bonded NICS use mode 4/802.3ad, all others use "balance-xor".

Check to see if the devices are present:

[root@osp-rhev02 net]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

2: enp8s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether c0:67:af:31:a3:31 brd ff:ff:ff:ff:ff:ff

3: enp9s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether c0:67:af:31:a3:32 brd ff:ff:ff:ff:ff:ff

4: enp10s0: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN qlen 1000

    link/ether c0:67:af:31:a3:38 brd ff:ff:ff:ff:ff:ff

5: enp11s0: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN qlen 1000

    link/ether c0:67:af:31:a3:37 brd ff:ff:ff:ff:ff:ff

6: enp12s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether c0:67:af:31:a3:36 brd ff:ff:ff:ff:ff:ff

7: enp13s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether c0:67:af:31:a3:35 brd ff:ff:ff:ff:ff:ff

8: enp1s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether c0:8c:60:8b:99:42 brd ff:ff:ff:ff:ff:ff

9: enp1s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether c0:8c:60:8b:99:43 brd ff:ff:ff:ff:ff:ff


Bonding Mode Configuration Files


Bond0 Configuration (ovirtmgmt network):

DEVICE=bond0

NAME=bond0

TYPE=bond

BONDING_MASTER=yes

BONDING_OPTS="mode=802.3ad miimon=10 lacp_rate=1"

USERCTL=no

NM_CONTROLLED=no

BOOTPROTO=none

ONBOOT=yes

IPADDR=172.19.17.12

NETMASK=255.255.255.192

GATEWAY=172.19.17.129

DEFROUTE=yes

DNS1=172.00.74.154

DNS2=172.00.74.100

DOMAIN=foo.bar.com

IPV4_FAILURE_FATAL=no

Bond1 Configuration:

DEVICE=bond1

NAME=bond1

TYPE=bond

BONDING_MASTER=yes

BONDING_OPTS="mode=balance-xor miimon=1 xmit_hash_policy=1"

BOOTPROTO=none

USERCTL=no

NM_CONTROLLED=no

MTU=9000

IPADDR=192.168.100.

NETMASK=255.255.255.0

IPV4_FAILURE_FATAL="no"

ONBOOT=yes

Bond2 Configuration (no IP information - used for bridge):

DEVICE=bond2

NAME=bond2

TYPE=bond

BONDING_MASTER=yes

BONDING_OPTS="mode=balance-xor miimon=1 xmit_hash_policy=1"

BOOTPROTO=none

USERCTL=no

NM_CONTROLLED=no

IPV4_FAILURE_FATAL="no"

ONBOOT=yes

CreatePlease to create content