05-23-2012 11:31 AM - edited 03-01-2019 10:25 AM
FI : 6248
Chassis : 5108
Blades : B200 M2 x 2
Uplinks : 5548UP
I'm setting up a pretty standard UCS deployment and have VPC uplinks from the 6248 fab interconnects to Nexus 5548s. The uplinks appear to be working correctly as I can manage UCS without any issues and can KVM to each blade. I've set up the mgmt ext pool and each of the blades gets an address correctly.
I initially setup a Xenserver blade and was able to connect to it on a server vlan that I created. However, since then I have reloaded the server and can not get it to work again no matter what I try. I've also tried load a Windows 2008 x64 server and while I am able to load the OS and configure the NICs within Windows (after installing the B200 drivers), there is no network connectivity .. it's as if the VLAN tagged traffic is not being forwarded upstream.
I have verified that all of our vlans are being passed on the uplinks between the FIs and the Nexus 5548s and I know it worked at one point so this has me puzzled.
I must have since misconfigured something but I can not for the life of me figure out what that was. I have recreated the VLANs, Service Templates, Services Profiles, etc. Each time I reload a blade, whether it be with Xenserver or with standard windows, I am able to see and configure the vNICS but I can pass no traffic. I can't do a ping from within the 6248s either since that cmd is not support, nor can I see any way of troubleshooting the traffic being passed.
Can someone please pass along some troubleshooting tips and/or places I can look to troubleshoot where the traffic is dying? I'm pulling my hair out!!
TIA!
Solved! Go to Solution.
05-24-2012 11:28 AM
Just to clarify,
Native VLAN and Default VLAN are two different things. Native simply refers to the VLAN on a trunked 802.1q link that will not be appended a VLAN tag. Default VLAN is the "automatically assigned" VLAN to all vNIC interfaces. By default the "default vlan" is vlan 1, but this can be changed.
When creating a Service Profile (or Template for that matter) you can use the Simple, or Expert wizard. The Simple wizard assumes each vNIC will operate as an "access port" by only being assigned to one vlan. In expert mode you can select either the port to operate as an Access Port (single VLAN) or Trunk (multiple VLANs). It's only with the Trunk vNIC configuration where you can select the "Native" VLAN radio button.
So let's say you have a Windows host that needs to be assigned to VLAN 10.
You could create a simple vNIC that acccess VLAN 10 only, or you could create a Trunk Interface, but set VLAN 10 as the native. Either method will return the same result.
For OSes like ESX & Hyper-V you will indeed want Trunking and therefore allowing multiple VLANs.
Now to top it all off, though we present what looks to be "Access" and "Trunk" ports, there's actually no access ports. What UCS does when you select a vNIC to be an Access port is just make a Trunk interface with the sole VLAN as the native.
You can see this from NXOS if you look at "show int trunk".
Regards,
Robert
05-23-2012 06:34 PM
Hi,
First thing to note, the way you manage the UCS and connect to it is via the management ports on the Fabric Interconnects. This includes all the KVM access (hence the reason why the IP management pool for the KVMs need to be on the same IP subnet as the Fabric interconnect management interfaces). Therefore, the fact that you can access the UCS to manage it is not indicative that the upstream network (N5K to UCS with vPC) is setup correctly.
Some things that you need to verify:
* The links and vPC between N5K and UCS are correct and in working order
* You are trunkning the correct VLANs on the links between the N5K and UCS
* You do not have a native vlan mismatch between UCS and N5K
* You are passing the correct vlans down to the server via the vNICs
* You have correctly defined the right vlan as native for the server vNIC
* If you are trunking all the way to the server, that the server NIC is setup to use the right vlan tagging
As there are numerous things that could be wrong, this will be easier to troubleshoot during a Live WebEx session so you should consider opening a case with the Cisco TAC.
Thanks,
Michael
05-24-2012 04:22 AM
Thanks Michael - that is all useful information and you are correct re: the mgmt traffic not validating the uplinks.
The odd part, however, is that when I initially set everything up one of the blades was working on one of our server vlans, so I know I had the uplinks, vlans on trunked uplinks, vlans to the server chassis and vnic vlans all correct - I was able to route that traffic.
Since then I removed the vlans, setup new vlans, vnic templates, service templates, service profiles and assigned back to the same blade and I can not route to the installed blade(s) anymore. The first blade (currently running xenserver) sees all of the NICs, so I know that's not an issue. I installed Windows 2008 on the second blade and it sees the same behavior. It really seems like an upstream routing issue but nothing has changed upstream since I originally installed UCS and configured the Uplinks to the Nexus 5548s.
At the 5548s I can see that all of the appropriate vlans are trunked on the Port-channel (VPC). In addition, I see all of the vlans within the 6248s and also see that the vethernet interfaces for the vNICs to the chassis (ie. Ve870, Ve871, etc) are all configured as trunks.
Here is an output of the port configs if that helps at all - it appears to all be correct to me (it was all configured from the GUI). Again, any help or suggestions is much appreciated!!
Eth1/1 S: Server, Port-ch connected 1 full 10G 1/10g
Eth1/2 S: Server, Port-ch connected 1 full 10G 1/10g
Eth1/5 U: Uplink connected trunk full 10G 1/10g
Eth1/6 U: Uplink connected trunk full 10G 1/10g
Po1025 S: Server connected 1 full 10G --
Veth870 server 1/1, VNIC X connected trunk auto auto --
Veth871 server 1/2, VNIC S connected trunk auto auto --
Veth872 server 1/1, VNIC X connected trunk auto auto --
Eth1/1/1 -- connected 1 full 10G --
Eth1/1/5 -- connected 1 full 10G --
Eth1/1/33 -- connected trunk full 1000 --
interface Ethernet1/5
description U: Uplink
pinning border
switchport mode trunk
switchport trunk allowed vlan 1,83,93,998
no shutdown
interface Ethernet1/1
description S: Server, Port-channel 1025
pinning server
switchport mode fex-fabric
fex associate 1
channel-group 1025
no shutdown
interface port-channel1025
description S: Server
switchport mode fex-fabric
pinning server
fex associate 1 chassis-serial FOX1606H4VB module-serial FCH161272GG
interface Vethernet870
description server 1/1, VNIC Xen_Mgmt_eth0_A
switchport mode trunk
no pinning server sticky
pinning server pinning-failure link-down
no cdp enable
switchport trunk allowed vlan 998
bind interface Ethernet1/1/1 channel 870
service-policy type queuing input default-in-policy
no shutdown
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1025 Po1025(SU) Eth NONE Eth1/1(P) Eth1/2(P)
05-24-2012 04:47 AM
wat does show port-channel and show interfaces trunk show on the nexus devices?
I assume you created a vPC between each FI and the nexus 5596's
05-24-2012 05:24 AM
Those both look correct to me (we have several other VPCs running on the 5548s as well). I've included some output from both the 5548 and 6248 below. Something else occurred to me .. as the vethernet interfaces are configured as trunks, the traffic at the blades is actually tagged, yes?
I'm wondering then if the the configuration is correct - I changed one of the vlans to be the "native" vlan and some traffic started passing, which makes me think I'm just not doing the native/default vlans correctly. I wouldn't have thought I'd have to configured the default and/or native vlan as I'm not allowing the default vlan on any of my vNICs, nor is the default changed anywhere else in our network (it's always 1)
5548-A
Port channels to each 6248
17 Po17(SU) Eth NONE Eth1/17(P)
18 Po18(SU) Eth NONE Eth1/18(P)
sh int trunk
Po17 1,81,83,91-97,99,222-223,998-999,1002-1005
Po18 1,81,83,91-97,99,222-223,998-999,1002-1005
vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
17 Po17 up success success 1,81,83,91-
97,99,222-2
23,998-999
6248-A
--------------------------------------------------------------------------------
Port Native Status Port
Vlan Channel
--------------------------------------------------------------------------------
Eth1/5 1 trunking --
Eth1/6 1 trunking --
Veth868 1 trunk-down --
Veth871 1 trunking --
Veth872 1 trunking --
Veth873 1 trunking --
Eth1/1/33 4044 trunking --
--------------------------------------------------------------------------------
Port Vlans Allowed on Trunk
--------------------------------------------------------------------------------
Eth1/5 1,83,93,998
Eth1/6 1,83,93,998
Veth868 93
Veth871 83
Veth872 83
Veth873 998
Eth1/1/33 4044
interface Vethernet873
description server 1/1, VNIC Xen_Mgmt_eth0_A
switchport mode trunk
no pinning server sticky
pinning server pinning-failure link-down
no cdp enable
switchport trunk allowed vlan 998
bind interface Ethernet1/1/1 channel 873
service-policy type queuing input default-in-policy
no shutdown
When I changed this vnic so that vlan 998 was the native vlan, some traffic started passing (when I say 'some', I mean that I was able to ping the default gateway of that vlan from the blade itself, but I could not ping from the gateway back to the blade, presumably because of the default and/or native vlan mismatches). I don't want to change any of the default or native vlan information and I was under the impression I didn't have to modify that on the UCS in any way.
05-24-2012 05:54 AM
the trunk allowed list on the veth seems to imply that only vlan 998 is allowed I assume you want more. How are the nic profiles/templates configured.
05-24-2012 06:50 AM
That particular veth/vnic only needs vlan 998 on it - I've created the vnic templates to only allow one vlan for the time being.
The nic profiles/templates are as simple as I could make them. Each one on only has a single vlan. Here is a picture of the VLAN config, vnic templates and the service templates and profiles. Notice that I'm not using the default or native vlan anywhere as I want all traffic to travel on the configured vlans (ie. no untagged traffic)
05-24-2012 11:08 AM
Ok, so I came to a few realizations. Each of my vNIC templates only had a single VLAN on it, hence there was a one to one relationship between the vnic template, the vnics in the service profile template and the service profile vnics.
I decided to experiment a little bit with the vlans in the vNIC template and got the traffic to pass when I assigned the single VLAN in the vNIC to be the "native" vlan. This doesn't make a lot of sense to me until the vNIC itself is actually acting as a virtual 'switch', which I assume it is because you can have multiple vlans on each vNIC. After setting each of the single vlans in my vnic templates to be 'native', I can pass all of my vlan traffic now.
So my question then is .. do you always have to define a native vlan in each of the vnics unless your blades are tagging traffic (ie. vlan aware).
To me, the term 'native' vlan is a little bit misleading - I would expect it to be called 'default' vlan because the traffic on that vlan is not being tagged at or by the host.
Anyway, thanks for the help everyone. If someone can just quickly answer my question about the native vlan question that would be great. I expect that if you are using a hypervisor that you wouldn't need to define a native vlan because you're creating vlan aware virtual nics there.
05-24-2012 11:28 AM
Just to clarify,
Native VLAN and Default VLAN are two different things. Native simply refers to the VLAN on a trunked 802.1q link that will not be appended a VLAN tag. Default VLAN is the "automatically assigned" VLAN to all vNIC interfaces. By default the "default vlan" is vlan 1, but this can be changed.
When creating a Service Profile (or Template for that matter) you can use the Simple, or Expert wizard. The Simple wizard assumes each vNIC will operate as an "access port" by only being assigned to one vlan. In expert mode you can select either the port to operate as an Access Port (single VLAN) or Trunk (multiple VLANs). It's only with the Trunk vNIC configuration where you can select the "Native" VLAN radio button.
So let's say you have a Windows host that needs to be assigned to VLAN 10.
You could create a simple vNIC that acccess VLAN 10 only, or you could create a Trunk Interface, but set VLAN 10 as the native. Either method will return the same result.
For OSes like ESX & Hyper-V you will indeed want Trunking and therefore allowing multiple VLANs.
Now to top it all off, though we present what looks to be "Access" and "Trunk" ports, there's actually no access ports. What UCS does when you select a vNIC to be an Access port is just make a Trunk interface with the sole VLAN as the native.
You can see this from NXOS if you look at "show int trunk".
Regards,
Robert
05-24-2012 12:08 PM
Thanks Robert - that helps to clear things up. When I first set the FIs and loaded a server it was working fine and I see now that it was because I initially selected the simple wizard - even though I assigned a specific vlan (lets say vlan10, as in your example) it was treating the vnic (vethernet interface) like an 'access' port as you say - I just didn't see at the time that it was assigning the native vlan to the sole vlan on the vnic.
The next time I setup the server I was using the expert mode and/or creating the vnics manually and not assigning the native vlan. I was afraid the 'native' term was pervasive throughout UCS and it would affect the whole FI, however, as per your explanation (and what I saw from my testing) it only applies to that particular vNIC, which in essence is a like a switch in and of itself.
Phew, I thought the vlans would be the easy part in this whole setup - the configuration guides don't really explain this in good detail I suppose because a lot of installs will use the simple wizard. The uplinks, VPCs, server ports and template and profile creation turned out to be the easy part!
Thanks again!!
05-24-2012 01:17 PM
Happy to Help.
I know our documentaiton can be daunting & limited in many aspects but that's what we're here for. We can answer your questions with a proper explanation
Let us know if you have any other issues or questions.
Cheers,
Robert
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide