N5K Trunk VLAN Forwarding State - none

[kstamandk]

Nexus 5548 vPC pair with down stream N2K in "cut through" or box design - N2K single connect connection to each N5K.

Each N5K Singly connected upstream to a pair of 6500 non-vPC.

Spanning-tree priority set with Odd number VLANs STP root set on "left hand" 6500 and Even numbered VLANs on "right hand" 6500.

All VLANs are Root Bridged to expected 6500, downstream Nexus learn all VLANs properly along with expected STP priority.

Servers(IBM AIX) are not Etherchanneled connected to N2K, but Redundant VIO Servers are setup Active / Standby with connections into each N2K (working well).

Problem: unable to ping Default Gateway from Client LPARs having multiple Virtual Ethernet Adapters

IBM AIX Client LPARS, with multiple Virtual Ethernet Adapters will only have one Virtual Ethernet Adapter configured with a default gateway. Other Virtual Ethernet Adapters will not have a default gateway set (IBM Recommendation). Client LPAR are unable to ping the gateway for those interfaces that do not have a default gateway set.

- from Nexus switch, we can ping all interfaces of the Client LPAR - MAC Address-table is updated as expected

- on Aggregation 6500 (VLAN Gateway), unable to ping those Client LPAR Virutal Ethernet Adapters that do not have the default gateway set.

- ARP table INCOMPLETE on 6500 for corresponding IP

- Checking Spanning-Tree Forwarding State of Trunk interface to Nexus from the 6500 that is Root Bridge for associated VLAN - the Forwarding State to the Nexus for that VLAN is None.

Challenge is, we only have a limited weekend window to identify / correct this issue. Below are planned approaches;

- investigate / correct why Trunk Forwarding state is NONE on the Trunk Interface to the Nexus

>> vPC Spanning-Tree Type on the vPC Peer Link is currently "Normal" - we plan to set that to "Network"

- investigate setting Spanning-Tree pathcost method to LONG on both the N5K and the 6500

>> concern is potential impact on legacy switches connected into the 6500

Thoughts / Suggestions / Feedback would be appreciated.

[jdewberr]

Hi,

Could you possibly attach a diagram/jpg of what you have connected, jpg preferred :-)

To look for the ARP on the Nexus 55xx here's a helpful tool

1. setup an interface vlan/SVI on that particular subnet example for vlan 10

interface vlan 10

no shutdown

ip add 172.16.10.100/24

you don't need a route in the default VRF, all we're doing is using this as a means with which to see ARP via ethanalyzer

2. ethanalyzer

ethanalyzer local interface inbound-low display-filter "arp" limit-captured-frames 100 write bootflash:arp-cap1.pcap

ftp the capture out and figure out if

1. arp request is getting to the Nexus55xx

2. arp reply is getting to the Nexus55xx

do on both vPC members.

[kstamandk]

Attached is a drawing.

I have setup the SVI and tried running Ethanalyzer, but not quite sure it's working - session hangs and I have to Clear it after 15 minutes, nothing written to the file.

[kstamandk]

Found out a couple of things since posting this;

1) vlan forwarding state of none appears to be a 6500 code issue with Nexus - TAC investigating

2) IBM Client LPARS with Virtual Ethernet Adapters without a default gateway will not work without the following defined

>> feature interface-vlan

>> interface vlan xxx

**** I do not understand why the VLAN Interface was requred. Anyone able to explain and help us understand??

(The whole discussion regarding not having vs having a default gateway interface is still on going with Unix / IBM teams. From a networking perspective - we don't understand why you would not have it.)

3) Ethanalyzer trick described by jdewberr  is a very handy tool. Although documentation and other posts indicate Data Plane traffic cannot be captured by Ethanalyzer on the N5K, this works well for us