UCS FEX Question

[visitor68]

Im wondering if someone can clear up one thing for me….

What exactly does a UCS FEX do besides aggregate the server traffic  and push it up to the Fabric Interconnects? What I am wondering is what  functionality/technology does it need to possess to do that?

From my understanding, a conventional blade switch needs to be  DCB-capable (support at least PFC and ETS) and do FIP snooping to be  used as an FCoE pass-through. IS that the case with a UCS FEX?? I dont  think so…it seems like nothing more than a “dumb” MUX that has no  intelligence, no code to upgrade and simply passes DCB traffic between  CNA and FI…

What am I missing, if anything?

[Eric Rose]

If it helps it is similiar to the Nexus 2k that connects to the parent switch - no local switching.

Basically providing 4 - 10G connections to the unified fabric.

[visitor68]

Hi, Eric. That much I understand.

What I dont understand is the disparity in requirements from a blade switch and a UCS FEX when it comes to being capable of passing FCoE traffic.

A blade switch has to do FIP snooping, support PFC, ETS and DCBx....a UCS FEX is nothing but a "bump in the wire" - so how does it get away with passing FCoE traffic to the FIs from the CNAs and back??

[Eric Rose]

A blade switch provides a local switch as well as additional management point. The FEX is an extended line card for the parent switch. The FEX is a fabric architecture where the blade switch is an isolated switch.

~~~~

Cisco UCS 2100 Series Fabric Extenders bring the unified fabric into the blade server enclosure, providing 10 Gigabit Ethernet connections between blade servers and the fabric interconnect, simplifying diagnostics, cabling, and management.

The Cisco UCS 2100 Series extends the I/O fabric between the Cisco UCS 6100 Series Fabric Interconnects and the Cisco UCS 5100 Series Blade Server Chassis, enabling a lossless and deterministic Fibre Channel over Ethernet (FCoE) fabric to connect all blades and chassis together. Since the fabric extender is similar to a distributed line card, it does not do any switching and is managed as an extension of the fabric interconnects. This approach removes switching from the chassis, reducing overall infrastructure complexity and enabling the Cisco Unified Computing System to scale to many chassis without multiplying the number of switches needed, reducing TCO and allowing all chassis to be managed as a single, highly available management domain.

The Cisco 2100 Series also manages the chassis environment (the power supply and fans as well as the blades) in conjunction with the fabric interconnect. Therefore, separate chassis management modules are not required.

[visitor68]

Eric, thanks, but this canned response doesnt answer my question...

I know what a FEX module is...please re-read my question because I dont know how many other ways I can rephrase it. I need a deep-dive technical answer.

[Eric Rose]

Didn't know that you needed such a detailed answer. Sorry. I will have to

let somebody else closer to bu engineering to comment.

[Manish Tandon]

Hi

Attached is a picture which shows the internal architecture of the IOM.

Will not delve into what has been already discussed - chassis mgmt, cabling, mgmt ease etc.

Yes, one of the key functions is mux traffic from HIF's to NIF's and vice versa.

Traffic from HIF's is unconditionally sent up to the FI for switching.

The ASIC is programmed via the CMC . It has the ability to replicate frames as is done with multicast traffic.

If you have2 recievers connected to the same IOM interested in the same multicats group, there is 1 packet which is sent by the FI and the replication

is handled by the ASIC. The ASIC also does tag imposition/removal etc (part of MUX functionality). The ASIC also provides QoS/buffering functionality.

So yes, packet forwarding to the SUP following a line card paradigm is its main purpose in life.

Thanks

--Manish

[visitor68]

Manish, as usual, great stuff. Thank you..

So, what you are telling me is inline with my thoughts about the UCS FEX. There is no switching intelligence in it and -- more pertinent to this discussion  -- no Data Center Bridging (CEE) capability to speak of. Its just a traffic MUX/DeMUX.

That means the UCS FEX HIF ports will NOT act as an 802.1D switch port does when it is DCB-enabled and generate a PFC PAUSE frame and send it back to the server CNA if its input buffers get overwhelmed. I Iimagine then that the UCS FEX leaves it up to the Fabric Interconnect's ingress port to generate the PUASE and the FEX will simply pass it on to the CNA. Correct? That would mean the FEX has no input buffers to speak of. If it did, it would have to be able to PAUSE traffic that is overrrunning them - or so I would think.

The same then can be said for the FEX and ETS. The FEX does not contain the hardware to participate in ETS. There is no egress scheduling algorithm and no recognition of packet prioritization or a mechanism to honor any prioritization that can be leveraged to dynamically assign bandwidth to the confgiured Traffic Class that would be serviced acording to the ETS Transmission Scheduling Algorithm (Algorithim 2, according to IEEE 802.1qaz v2.4).

Lastly, the FEX does not participate in DCBx either - the link initialization semantics occur between the CNA and the Fabric Interconnect port.

Is all this correct, Manish?

I really appreciate your time and help. Thank you!

[Manish Tandon]

The IOM is a pass through for switching but it does have buffers and queues.

The DCBX packets between the FI and the adapter are passed as is like a line card would. Same goes for other things like CDP packets etc.

But the FI communicates with the CMC using a diff software path and the ASIC is programmed on the IOM for CoS and bandwidth values.

Think of it as a "qosmgr" process in the system.

So its not DCBX between the FI and IOM but some other way as the IOM is a part of the FI.

The IOM does send PFC when congestion hits to either the adapter or the FI.

When congestion between IOM and FI hits, its too late or away in path for the FI to send PFC to adapter. The IOM has to.

This is how lossless is achieved.

Thanks

--Manish