01-04-2023 10:32 AM
Hello,
I'm trying to figure out if and how to configure 2 vNIC's on 2 different b200 blades to communicate without having to use a northbound ACI switch.
Each b200 has a vNIC-a and vNIC-b
Both vNIC's have been given the same random vlan 1234
Both vNIC's have vlan 1234 as default vlan
Both vNIC's are derived from the same template.
I have several other NIC's configured with an ACI switch with no issues and the OS can see them.
LAN Uplinks manager has all VLANS that need to go to a NB switch as added to the vPC
Vlan 1234 is not pinned to anything
The NIC's that were configured without an ACI connection are showing as down. OS is linux.
Thanks!
Solved! Go to Solution.
01-04-2023 12:01 PM
hi @MikeErnst ,
Yep. You've nailed it. Without VLAN 1234 being able to communicate across the Ethernet uplink you are dead in the water. I've even tried a patch cable between two FIs on ports configured as uplinks - still didn't work.
So to your later question
if we have a bonded pair in the OS with 1 NIC pinned to FI-A and another nic pinned to FI-B that the bond wont work?
you are correct. The bond won't work (I'll qualify that as 95% sure - @Wes Austin would probably know better than me)
01-06-2023 11:42 AM
You shouldn't need "switching" mode. End host mode should work just fine. If the VLAN is not pinned, that's the issue.
What is the "Action on Uplink Fail" policy?
See:
Ever had a servers switch loose uplink connectivity and never failed over to the "other" NIC/switch because the server didn't know that connection was isolated?
Well UCS defaults to "Action on Uplink Fail" as "disable" to prevent this.
Since you have no uplinks (for this VLAN), then change this to "warning" and it should allow the servers to communicate without upstream connectivity.
Also check from the FI SSH console: connect nxos a; show mac address table vlan 1234
01-04-2023 10:43 AM
Hi @MikeErnst
How did you configured the IP addresses on the linux machines if both vNICs are in the same vlan?
Cheers,
Sergiu
01-04-2023 10:49 AM
In the OS we have the 2 vNIC's bonded with an IP/subnet and just no gateway.
01-04-2023 11:12 AM
Think I figured this out.
FI's in end host making this not possible, not going to switching mode either so ill have to define a vlan on a NB switch regardless.
01-04-2023 12:01 PM
hi @MikeErnst ,
Yep. You've nailed it. Without VLAN 1234 being able to communicate across the Ethernet uplink you are dead in the water. I've even tried a patch cable between two FIs on ports configured as uplinks - still didn't work.
So to your later question
if we have a bonded pair in the OS with 1 NIC pinned to FI-A and another nic pinned to FI-B that the bond wont work?
you are correct. The bond won't work (I'll qualify that as 95% sure - @Wes Austin would probably know better than me)
01-04-2023 11:13 AM
The vNIC are not pinning likely because VLAN 1234 is not allowed on the uplink and/or the uplink is not online that allows VLAN 1234. As long as VLAN 1234 is not allowed on the upstream ACI fabric, and vNIC are learned on the same FI in the same VLAN 1234 they should be able to communicate. If one vNIC is pinned to FI-A and the other is pinned to FI-B, they will not communicate.
Any screen shots or outputs you can share from your VLAN configurations in the GUI / LAN uplinks manager or NXOS CLI output for the running configuration of the network uplinks would be helpful in diagnosing the problem further.
01-04-2023 11:29 AM
Thanks for the reply
From what you said, I think a good test is to unbond the NIC's in the OS and try to configure one of them by itself. Which I'm trying now.
Correct me if I'm wrong.
01-04-2023 11:45 AM
To clarify
Do you mean if we have a bonded pair in the OS with 1 NIC pinned to FI-A and another nic pinned to FI-B that the bond wont work OR are you stating that between 1 blade a NIC from FI-A and FI-B will not communicate?
01-04-2023 12:04 PM
I am saying that if you want two vNIC (regardless of bonding) to communicate L2 without allowing the VLAN across the upstream switches, the vNIC have to be pinned to the same FI. There is no inter-switch data plane communication between the fabric interconnects A and B.
01-04-2023 12:06 PM
OK so if I had vNIC-A pinned to FI-A defined on B200-1 and vNIC-A pinned to FI-A on B200-2 I should in theory see connectivity.
Which is still not working.
I think this is due to the fact that the host isn't in switching mode and requires a NB switch regardless.
01-04-2023 12:29 PM - edited 01-04-2023 12:32 PM
Correct. If its not working, please provide the screen captures and NXOS outputs so we can check it out further and confirm your mac address table on FI-A showing that both MAC addresses are learned in the same VLAN
"host isnt in switching mode"...If you are talking about the FI, you typically do not want FI in switching mode unless you have a very specific use case. Most use end host mode.
01-06-2023 11:42 AM
You shouldn't need "switching" mode. End host mode should work just fine. If the VLAN is not pinned, that's the issue.
What is the "Action on Uplink Fail" policy?
See:
Ever had a servers switch loose uplink connectivity and never failed over to the "other" NIC/switch because the server didn't know that connection was isolated?
Well UCS defaults to "Action on Uplink Fail" as "disable" to prevent this.
Since you have no uplinks (for this VLAN), then change this to "warning" and it should allow the servers to communicate without upstream connectivity.
Also check from the FI SSH console: connect nxos a; show mac address table vlan 1234
01-06-2023 02:54 PM
Thanks for the reply,
Due to the fact this was a time sensitive issue I went and had the vlan added to one of the upstream switches regardless which resolved the issue. This is interesting and I wouldn't mind testing it in a lab when I get ahold of one again.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide