L2 connection between 2 FI-6248UP's ?

[Thijs Van Severen]

Hi all,

I wonder if i can directly connect 2 6248UP's without going over an upstream nexus

What i want to accomplish is this :

I have a number of racks all equipped with 6248UP's (main&backup).  In 1 rack i have storage (NAS&SAN) that is only used by the UCS blades in that same rack, so the vlans used for that storage are only relevant in that specific rack.

Therefore i see no need to go over the Nexus.

The reason i'm asking is because i want to keep the storage path a simple and short as possible.  Since this is a lab environment and we are experimenting a lot with storage i dont want the extra routing stage to 'over complicate' things

Is it possible to configure a uplink ports on both main and backup FI of that rack, associate the relevant storage vlans with that uplink port and then directly connect the 2 FI's ?

all help is welcome !

grtz

thijs

[Kirk J]

Greetings.

Connecting FI data planes directly in end host Ethernet mode is definitely not supported.

If the FIs are in Ethernet switching mode this might be permitted, but I haven't found any white papers or examples yet of this kind of config.

I think in your scenario you just need to design your vnics, and storage/appliance IPs such that vnics on A side are only trying to communicate with appliance port resources on the A side, etc.

The UCSM configuration is going to expect that you will allow pinning of your appliance vlans to an uplink port, but the traffic between the vnics and appliance ports can stay local to the individual FIs with the correct design.

Considerations for avoiding storage appliance traffic traversing the uplinks:

• If storage must be accessed outside of the UCS domain.
• If storage targets and initiators/clients are in different subnets.
• If storage is configured in Active/Passive mode and both fabric interconnects require communication to the same controller.
• In certain failover scenarios. (Don't setup vnic fabric fail over for vnics using vlans that only connect to appliance ports

Thanks,

Kirk

[Thijs Van Severen]

hi Kirk,

Thanks for the input !

To clarify: the specific issue we had was with an NFS server running on a dedicated C240.  There are no appliance ports configured for this server, instead the server connects to 2 'server ports' on FI A and B

I understand that i can force traffic to stay on the 'A-side' or 'B-side' but this also means that i lose redundancy since i'm using a active-backup bonding config on the A and B vnics of the NFS server

A FI allows you to connect to multiple upstream routers/switches, so i dont really see why creating a dedicated FI uplink port (for just this 1 vlan) on both FI's that i connect directly wouldnt work ?

I have done this before, the only difference is that the FI uplink ports were connected to 2 switchports on a 4900 that were in the same vlan

It just feels crazy to use an extra switch just to create a L2 connection between the FI's, unless of course i'm missing something (unfortunately that happens frequently lately ;-)

grtz

Thijs

[Walter Dey]

Hi Thijs

It just feels crazy to use an extra switch just to create a L2 connection between the FI'

Just be aware of the following situation:

2 service profiles associated with 2 vnics each (on same or different blade), vnic-a connecting to fabric A, vnic-b connecting to fabric B. Both are in the same vlan, e.g. 20.

If host-A talks over vnic-a to host-B over vnic-a, FI-A does L2 switching !

However, if host-A talks over vnic-a to host-B over vnic-b, then FI-A has to send the frame north bound, because it cannot switch L2 between fabric A and B.

Thats why you need the external L2 links between the 2 switches.

Walter.

[Thijs Van Severen]

Hi Walter

I completely agree with what you are saying and this is what we have right now

This (standard config) works very well, but we had some issues with the northbound switches causing all northbound connections to drop completely.

Therefore it would have been nice to take out that part of the config completely and just replace it with a wire :-)

I have just tried to link to FI's directly using a 'network' interface, but as soon as i do this the port shuts down.  It's clearly not meant to be used like this.

An other option would be to (ab)use the FI's and configure 2 appliance ports and see if i can interconnect the FI's that way, but it feels 'ugly' so i guess i'll have to go with the extra switch  :-(

grtz

Thijs