UCS Inter-fabric Networking Issue

[robert.moon]

Hi,

I have recently implemented UCS with dual fabric and am running in to a network issue. Basically VMs with vNIC on Fabric A cannot communicate on VMs with vNIC on Fabric B (same VLAN). I will try to describe the environment and testing performed as below.

• Fabrics are configured in EHM and are pinned to Nexus 3Ks. Nexus' have been configured with the correct VLANs. 
• N3Ks are in VPC with N5K core where each gateway resides.
• From VM vNIC FI-A I can ping the gateway and IP addresses connected on the same switches ( NAS connected to N3Ks) but cannot connect or ping any VM vNIC FI-B
•  From VM vNIC FI-B I can ping the gateway and IP addresses connected on the same switches ( NAS connected to N3Ks) but cannot connect or ping any VM vNIC FI-A

It appears that network is functioning correctly on both FI-A and FI-B but not between servers on FI-A and FI-B.

I have checked the upstream switch's MAC table and it has the MAC addresses of all servers, same with the FIs, but for some reason it's not working.

I believe the issue lies at the UCS layer which is preventing the traffic from exiting the FI-A uplink and entering the FI-B.

Any ideas would be greatly appreciated.

[Walter Dey]

It seems to me that the problem is outside of UCS;

Basically VMs with vNIC on Fabric A cannot communicate on VMs with vNIC on Fabric B (same VLAN).

Because the FI-A cannot L2 switch to fabric B, it has to send the frame Northbound !

Are you sure that this particular VLAN is setup between the A and B Northbound switch, and therefore L2 switch from fabric A to fabric B ? This is mandatory, otherwise it won't work.

[Kirk J]

Greetings.

Your tests seem to indicate the various guestVM MACs are being correctly learned upstream if you can reach the DG and the ping responses make it back:

• From VM vNIC FI-A I can ping the gateway and IP addresses connected on the same switches ( NAS connected to N3Ks) but cannot connect or ping any VM vNIC FI-B
•  From VM vNIC FI-B I can ping the gateway and IP addresses connected on the same switches ( NAS connected to N3Ks) but cannot connect or ping any VM vNIC FI-A

Some additional things to check:

• Is subnet mask right?  Are VMNIC A/B in same subnet?
• Is each N3k connected to both FIs?
• Do you have disjoint layer2 config where you expect different vlans to go out separate uplinks on each FI?
• Can you post a general topology of your FI>N3k>N5K port connectivity?
• Make sure your vswitch teaming settings are set to 'originating port id', and not IP hash.
• Confirm your VMNIC mac addresses match the VNIC mac addresses in UCSM, so the vnic you think the VMNIC number matches is actually true.
• Can you ping both A and B VMNIC IPs from both N5K SVIs?
• Are you running HSRP on your N5ks?
• What UCSM firmware version are you running?

If each FI is connected to both N3k, assuming your vnic A <> vnic B communication is in same vlan, then your switching should occur on a single N3k...

I'm not sure your problem resides in the UCSM.

Thanks,

Kirk

[robert.moon]

Thanks for the replies.

• Subnet is correct, both vNICs in same VLAN, same subnet.
• Each 3k connected to both FIs
• No disjoint layer 2, same VLANs for all uplinks
• Basic topology attached.
• Confirmed set to 'originating port ID'
• VMNIC MAC address match vNIC in UCSM
• Can ping both IPs from 5Ks
• Not running HSRP on 5Ks
• Firmware 3.1 (updated in November)

Tested against IP on same VLAN that is not either Chassis (a NAS server connected to the 3Ks). Both VMs on each fabric can reach it, VLAN appears to be configured correctly based on this test. Traffic on this VLAN works fine on the 3Ks from both FI-A and FI-B. Traffic only fails from FI-A > 3Ks > FI-B and vice versa. 

[Huyen Duong]

Hi Robert

Is there any update from you, how did you get it resolved?

Thanks much

[Kirk J]

Seems like you may have some kind of issue with the VPC and the PCs on the N3Ks facing the FIs.

Please list the following for each n3k:

#show run int ethx/y (for any FI connected ports)

#show vpc brief

#show vpc

#show port-channel summary

#show vpc consistency-parameters

It would be helpful if you could update your topology diagram to include the N3k ports that are connected to FIs and indicate which FI and FI port the links are connected to.  Also the VLAN for the guestVMs.

Then during a continuous pingtest from guestVM pinned to FI-A to guestVM pinned to FI-B test, list the following:

• FI-A, #show mac address-table | in guestVM-A'sMAC
• FI-A, #show mac address-table | in guestVM-B'sMAC
• FI-B, #show mac address-table | in guestVM-A'sMAC
• FI-B, #show mac address-table | in guestVM-B'sMAC
• N3K-A,#show mac address-table | in guestVM-A'sMAC
• N3K-A,#show mac address-table | in guestVM-B'sMAC
• N3K-B,#show mac address-table | in guestVM-A'sMAC
• N3K-B,#show mac address-table | in guestVM-B'sMAC

Thanks,

Kirk

[Huyen Duong]

Hi Kirk

I double check the issue in my case is our engineer forget to config disjoint Layer 2

It works after we config correct disjoin layer 2.

We use physical server (bare metal) not VM environment, so I am not sure about your case.

But normally, in my experience with vpc and vmware we can use IP hash on vswitch

Thanks

[Kirk J]

Greetings.

Robert has confirmed that they don't have a disjoint L2 config or requirement.

UCSM in EHM does not work with IPhash on the blade hosts because the FIs are not directly connected to each other, and wouldn't be able to bring up the other end of the port-channel.

Outside of the UCSM, the UCS rack servers can accommodate IP-hash with a correctly configured switch.

Thanks,

Kirk...

[robert.moon]

Hi all,

I got a TAC case going with the Cisco team and identified the issue.

For those interested, this is a bug in the Fabric Interconnects running release 3.1(1e) and 3.1(1g).

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz46574/?reffering_site=dumpcr

Thanks to all who assisted.

-Robert.