Determine NIF port used by VIF when configured in a port-channel

[matthewcrocker]

I have recently just watched an excellent Cisco Live video on UCS performance troubleshooting which demonstrated how to trace network traffic within Cisco UCS. The speaker made a comment though, in order to determine which NIF is used by a VIF when port-channels are used between the FEX and FI there are different commands to run. Need to determine the load-balancing hash result. Unfortunately he never went into what these commands were.

So when we have veths pinned to port-channels instead of HIFs and NIFs, what commands will show which path is being used?

[Keny Perez]

Matt,

You normally use these commands:

1-A(nxos)# sh mac address-table address XXXX.YYYY.ZZZZ <<< the vNIC you want to track down, the output will give you the vEthernet

2-A(nxos)# sh inter veth XX <<< vEth you got from last command, and you will see a "Bound Interface", something like 1/1/1 (first and last number is the chassis and server ID)

3-A(nxos)# sh pinning interface vethernet XX  < this is probably the command you are looking for.

A(nxos)# sh pinning interface vethernet XX

---------------+----------------------+------------------------------+-----------------

SIF Interface   Sticky               Pinned Border Interface  Pinned Duration

---------------+----------------------+-----------------------------+-----------------

VethXX          Yes(hard-pinned)      Po1     <<<<                             2d 42:33:35

Let me know if that helps.

-Kenny

[matthewcrocker]

Hey Kenny,

As we are using port-channels everywhere, the commands you mention will always show the bound interface as a port-channel. I will start with this example and we can see if we can go up the chain. Here is the output that displays the port-channel for the blades. We have 20Gbps per fabric. Which HIF is my veth using? Port-channel 1282 contains 2 x HIFs. I am trying to trace the exact physical path taken of the traffic, down to individual ports.

ucs-A(nxos)# show interface veth713
Vethernet713 is up
    Bound Interface is port-channel1282
 

I then go check port-channel 1282...as you can see I have 2 port-channel members, the blades HIFs. 1/1/1 & 1/1/3 I believe? Correct me if I am getting any of this wrong.

ucs-A(nxos)# show int port-channel 1282
port-channel1282 is up
  Hardware: Port-Channel, address: 18e7.28b4.34d2 (bia 18e7.28b4.34d2)
  MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is vntag
  full-duplex, 10 Gb/s
  Input flow-control is off, output flow-control is on
  Switchport monitor is off
  EtherType is 0x8100
  Members in this channel: Eth1/1/1, Eth1/1/3
 

The next step, how can I determine which port-channel member my veth is using? This question I would like to answer at all points in UCS, the blade, FEX uplinks and LAN uplinks.

Thanks,

Matt

[Keny Perez · Accepted solution]

Matt,

you may use this commands:

B(nxos)# sh port-channel load-balance << this will tell you the load balance method used

If you are using source-dest-ip like in my case, you may use this command:

B(nxos)# sh port-channel load-balance forwarding-path interface port-channel ID vlan ID src-ip x.x.x.x dst-ip y.y.y.y  and it will show you something like this:

Missing params will be substituted by 0's.

Load-balance Algorithm on switch: source-dest-ip
crc8_hash: 109  Outgoing port id: EthernetX/Y  <<< this is what you are looking for now
Param(s) used to calculate load-balance:
        dst-ip:   y.y.y.y
        src-ip:   x.x.x.x
        dst-mac:  0000.0000.0000
        src-mac:  0000.0000.0000

For the blade, it depends on which the active vNIC is, for the FEX it depends on the pinning based on the blade slot the server is in.  Odd servers go through odd links  and even servers go through even ports.

Don't forget to rate helpful answers.

-Kenny

[matthewcrocker]

This is exactly what I was seeking!

One more question if I may, the output of that first command you mentioned. in what context is the latter used? As it differs to the system default of source-dest-ip. Do I need to worry about that when running the second command you provided? Or just know that the system LB configuration will be used.

ucs-A(nxos)# show port-channel load-balance

Port Channel Load-Balancing Configuration:
System: source-dest-ip

Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: source-dest-mac
IP: source-dest-ip source-dest-mac

And you said for the FEX it depends on the pinning based on the blade slot. What do you mean exactly in situations where FEX port-channels are used? If we are using a 2208XP with 4-links in a port-channel. I assume the port-channels of the blades get bound to the port-channel of the FEX? Trying to see the next relationship, port-channels with HIF members to the NIFs. Does that make sense?

[Keny Perez · Accepted solution]

Just care about the first part :

ucs-A(nxos)# show port-channel load-balance

Port Channel Load-Balancing Configuration:
System: source-dest-ip

With the command " A(nxos)# sh inter vethernet XXX "  you can see the "Bound Interface"  and after that you may run:

A(nxos)# show  system internal fex info satport ethernet 1/1/1 <<< this will show you something like this:

FABRIC-A(nxos)# show  system internal fex info satport ethernet 1/1/1

  Interface-Name    ifindex         State   Fabric-if    Pri-fabric     Expl-Pinned

        Eth1/1/1      0x1f000000   Up        Eth1/1      Eth1/4         Eth1/1

     Dev: 0 Nif3  <<<<    Hif7   <<<<<<  (Nif:0x20000000 Hif:0x1f000000)

NIF = interface from the IOM to the FI

HIF = interface from the blade's adapter to the IOM's backplane

Let me know if you have more questions :)

-Kenny

[matthewcrocker]

You have successfully removed the cloud of mystery for me when using port-channels inside of the UCS chassis!

Thanks for your time, appreciate it.

[Keny Perez]

Glad to read that :)

-Kenny