Pinned interface up, but Veth down?

David Grocke · ‎10-11-2012

Hey Guys

What reasons would a Veth be in nonParticipating mode and my northbound interfaces be in an up state?

The whole story behind this is every one of my Veths are down, including my FC VFCs (typically within ErrDisable - reason is pinned uplink is down)

everything is v(2.04a) / my uplinks are trunking appear fine, inluding eth and FC / it has been working fine (don't have a clue what I have changed to break it )

IE:

Major F0283 2012-10-12T11:55:02.574 241330 ether VIF 1 / 2 A-931 down, reason: Non participating

Po2 900 eth trunk up none a-10G(D) lacp

Veth931 900 eth trunk down nonPartcipating auto

Eth1/1/2 1 eth vntag up none 10G(D) --

//show

Po2 900 trunking --

interface port-channel2

description U: Uplink

switchport mode trunk

pinning border

switchport trunk native vlan 900

switchport trunk allowed vlan 1,55,900

speed 10000

interface Vethernet931

description server 1/2, VNIC DMZ-e-x-BE-FIA

switchport mode trunk

pinning server sticky border-interface port-channel2

pinning server pinning-failure link-down

no cdp enable

switchport trunk native vlan 900

switchport trunk allowed vlan 55,900

bind interface Ethernet1/1/2 channel 931

service-policy type queuing input default-in-policy

no shutdown

interface Ethernet1/1/2

switchport vntag max-vifs 118

no pinning server sticky

switchport mode vntag

fabric-interface Eth1/1

no shutdown

Thoughts?

Cheers

Me

David Grocke · ‎10-12-2012

Realised I should have posted this within UCompute, rather than in here

Time for me to call on the beautiful TAC, as I'm seeing a few other strange things... such as unsucessful pinning on on FI but not the other.... I'm seeing my FIs Flogi, but not my vHBAs, etc. etc.

"show pinning server" FIA

Veth929 Yes(hard-pinned) Po1 4:12:23

Veth931 Yes(hard-pinned) - -

Veth933 Yes(hard-pinned) Po3 4:12:23

Veth935 Yes(hard-pinned) Po4 4:12:23

"show pinning server" FIB

Veth930 Yes(hard-pinned) Po1 3:57:15

Veth932 Yes(hard-pinned) Po2 3:57:15

Veth934 Yes(hard-pinned) Po3 3:57:15

Veth936 Yes(hard-pinned) Po4 3:57:15

Po1 Active Veth929

Po2 Active

Po3 Active Veth933

Po4 Active Veth935

Po1 Active Veth930

Po2 Active Veth932

Po3 Active Veth934

Po4 Active Veth936

Cheers

Me

padramas · ‎10-12-2012

Hello David,

What is the status of the IOM back plane ports ?

Do you have hard pinning configruation ?

Padma

David Grocke · ‎10-12-2012

Good question, the Backplane ports are all up and I have 'reset' connectivity on those mupltiple times.

Yes, I have hard pinning configuration (reasons are for security within multiple DMZ environments / dis-joint L2 northbound out of my FIs)

I have noticed that if I break my hard pinning (by unlugging or shutting down either end of the uplink) it never comes back until I unassociate the service profile and re-associate the profile (essentially creating entirely new veths and vfcs to play with...).

Because the "show pinning" never resolves even after my uplink ports regain connectivity, to me it smells a little buggy...

Dave

padramas · ‎10-12-2012

Dave,

If you are connecting FI to separate L2 networks, please make sure you have correct disjoint L2 configuration on the FI.

Please open a TAC SR so that we can understand your network configuration.

Padma

David Grocke · ‎10-12-2012

Thanks; I certainly will log a TAC for this. It's a little late on a Friday night where I live so I might wait until next week

I've knocked off work for the week and need some rest

Imagine if I'm not hard pinning or disjoined layer 2 (like my vHBAs and FC ports), what else would cause a Veth to be within a 'nonParticipating' state? If there are still uplinks that could be used I would have expected my Veth to be re-pinned to one of these available uplinks. Are there any other causes that would stop a Vethernet being dynamically pinned outbound?

Thanks

Dave

padramas · ‎10-15-2012

Dave,

Nonparticipating status generally refers to physical link being down. We need to trace teh path veth upto to uplink interface and verify port status along the path.

You can start with following UCSM command

show service-profile circuit server X/Y ( X - chassis and y - Blade )

Then check IOM ports by

connect iom X ( X - chassis ID from each FI )

show platform software {redwood | woodise} sts

## Few show commands on NXOS

show int {veth | eth } ( VIF ID maps to veth number, IOM eth port, uplink Eth port status )

show int bri

-------------------------

Also check the upstream switch port configuration ( i.e verify if they are configured as trunk edge / trunk portfast when FI is operating in end host mode )

Padma

David Grocke · ‎10-15-2012

Thanks Padma

The service-profile is what is erroring (Link state and overall state), and the rest of the information confirms the information that I see on 'show pinning border-interfaces' and 'show pinning server-interfaces'

My IOM confirms the same information as a 'show int brief' within the nxos, which seems correct. All my nexus configuration seems correct and both the IOM ports are saying up and the FI ports are saying up. Just my Veths and VFCs that are down.

Yesterday I decided to delete all the configuration and start again, however, that didn't help . Removing all disjoint layer 2 configuration and all port channels.

My uplinks appear fine (Flogis of the switches received from the FIs, etc.).

I had thought it was because I was using a shared Native VLAN across multiple disjoint layer 2 uplinks, but after removing all configuration that didn't fix it, and I think I would see other symptoms.

Thanks for your help and from here I'll just rely on the wisdom of the TAC. I'll report back once I find out what I've done wrong or if there is something more sinister.

Thanks again

David Grocke

David Grocke · ‎10-15-2012

Oh, I'm an absolute idiot.

Unless I have a working OS my vNICs will not come up, and unless I have san boot targets my vHBAs will never try to flogi.

Absolutely embarrassing... maybe I should have made up the TAC said that it was a software bug....

Thanks for your help Padma; I certainly have learnt a few diagnostic commands that I can run out of the whole experience.

Cheers

Me

CSCO11859847 · ‎03-08-2013

Hi David

I am having the same problem in our new environment, how exactly did you get it to work?

I have the following output on the UCSM:

David Grocke · ‎03-12-2013

Hey Nkateko

My issue was I hadn't put a working OS on my blades or started to install an OS on my blades.

I had expected that the vNICs would come up just by powering on the blade, regardless what OS is installed, but I was wrong.

I hope that helps... Otherwise, it's typically your pinned uplink that is down (ie: the port channel or uplink ports)

Cheers

Dave