L2 on a UCS6454 with no upstream N9k

[MikeErnst]

Hello,

I'm trying to figure out if and how to configure 2 vNIC's on 2 different b200 blades to communicate without having to use a northbound ACI switch.

Each b200 has a vNIC-a and vNIC-b

Both vNIC's have been given the same random vlan 1234

Both vNIC's have vlan 1234 as default vlan

Both vNIC's are derived from the same template.

I have several other NIC's configured with an ACI switch with no issues and the OS can see them.

LAN Uplinks manager has all VLANS that need to go to a NB switch as added to the vPC

Vlan 1234 is not pinned to anything 

The NIC's that were configured without an ACI connection are showing as down. OS is linux.

Thanks!

[Sergiu.Daniluk]

Hi @MikeErnst 

How did you configured the IP addresses on the linux machines if both vNICs are in the same vlan?

Cheers,

Sergiu

[MikeErnst]

In the OS we have the 2 vNIC's bonded with an IP/subnet and just no gateway.

[MikeErnst]

Think I figured this out.

FI's in end host making this not possible, not going to switching mode either so ill have to define a vlan on a NB switch regardless.

[RedNectar · Accepted solution]

hi @MikeErnst ,

Yep. You've nailed it. Without VLAN 1234 being able to communicate across the Ethernet uplink you are dead in the water. I've even tried a patch cable between two FIs on ports configured as uplinks - still didn't work.

So to your later question

if we have a bonded pair in the OS with 1 NIC pinned to FI-A and another nic pinned to FI-B that the bond wont work?

you are correct. The bond won't work (I'll qualify that as 95% sure - @Wes Austin would probably know better than me)

[Wes Austin]

The vNIC are not pinning likely because VLAN 1234 is not allowed on the uplink and/or the uplink is not online that allows VLAN 1234. As long as VLAN 1234 is not allowed on the upstream ACI fabric, and vNIC are learned on the same FI in the same VLAN 1234 they should be able to communicate. If one vNIC is pinned to FI-A and the other is pinned to FI-B, they will not communicate.

Any screen shots or outputs you can share from your VLAN configurations in the GUI / LAN uplinks manager or NXOS CLI output for the running configuration of the network uplinks would be helpful in diagnosing the problem further.

[MikeErnst]

Thanks for the reply

From what you said, I think a good test is to unbond the NIC's in the OS and try to configure one of them by itself. Which I'm trying now.

Correct me if I'm wrong.

[MikeErnst]

To clarify

Do you mean if we have a bonded pair in the OS with 1 NIC pinned to FI-A and another nic pinned to FI-B that the bond wont work OR are you stating that between 1 blade a NIC from FI-A and FI-B will not communicate?

[Wes Austin]

I am saying that if you want two vNIC (regardless of bonding) to communicate L2 without allowing the VLAN across the upstream switches, the vNIC have to be pinned to the same FI. There is no inter-switch data plane communication between the fabric interconnects A and B. 

[MikeErnst]

OK so if I had vNIC-A pinned to FI-A defined on B200-1 and vNIC-A pinned to FI-A on B200-2 I should in theory see connectivity.

Which is still not working.

I think this is due to the fact that the host isn't in switching mode and requires a NB switch regardless.

[Wes Austin]

Correct. If its not working, please provide the screen captures and NXOS outputs so we can check it out further and confirm your mac address table on FI-A showing that both MAC addresses are learned in the same VLAN

"host isnt in switching mode"...If you are talking about the FI, you typically do not want FI in switching mode unless you have a very specific use case. Most use end host mode. 

[Steven Tardy · Accepted solution]

You shouldn't need "switching" mode. End host mode should work just fine. If the VLAN is not pinned, that's the issue.

What is the "Action on Uplink Fail" policy?

See:

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Network-Mgmt/4-2/b_UCSM_Network_Mgmt_Guide_4_2/b_UCSM_Network_Mgmt_Guide_chapter_01010.html#concept_80DD753607A0427A9874F278E172F2FA__section_DB2F3D286BB342929E44...

Ever had a servers switch loose uplink connectivity and never failed over to the "other" NIC/switch because the server didn't know that connection was isolated?
Well UCS defaults to "Action on Uplink Fail" as "disable" to prevent this.

Since you have no uplinks (for this VLAN), then change this to "warning" and it should allow the servers to communicate without upstream connectivity.

Also check from the FI SSH console: connect nxos a; show mac address table vlan 1234

[MikeErnst]

Thanks for the reply,

Due to the fact this was a time sensitive issue I went and had the vlan added to one of the upstream switches regardless which resolved the issue.  This is interesting and I wouldn't mind testing it in a lab when I get ahold of one again.