Intra-VLAN traffic not passing back to Nexus 5k

rsjordan00 · ‎03-29-2013

We recently extended our access layer using a pair of 5ks with extenders. We have a pair of 6509s at our core and they handle the intra-VLAN routing with SVIs. I recently noticed that access hosts connected to the extenders cannot pass traffic between each other if they are in different VLANs.

The strange thing is these same hosts can ping devices in other VLANs as long as the other devices are not connected to the 5k environment.

For example, consider the following hosts. Each host has their gateway set to the appropriate SVI on our core.

HostA - VLAN100 - connected to 5k extender

HostB - VLAN200 - connected to 5k extender

HostC - VLAN100 - connected to 2960 off our core

HostD - VLAN200 - connected to 2960 off our core

Each host can ping each other with the exception of HostA and HostB.

As for specifics, we use HSRP (no VSS) between our cores.

Core1 interface

interface Port-channel31

description 5k-vPC-101

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 50

switchport mode trunk

no ip address

5k interface to Core1:

interface port-channel101

description vPC-Uplink-Core1

switchport mode trunk

switchport trunk native vlan 50

spanning-tree cost 24000

speed 10000

vpc 101

Core2 interface:

interface Port-channel31

description 5k-vPC-102

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 50

switchport mode trunk

no ip address

5k interface to Core2:

interface port-channel102

description vPC-Uplink-Core2

switchport mode trunk

switchport trunk native vlan 50

spanning-tree cost 28000

speed 10000

vpc 102

I setup a span with the following sources:

5k1: po101

5k2: po102

Core1: po31

When I ping between hostA and hostB, I see the egress packets on either 5k1 or 5k2. I then see ingress AND egress on Core1. There are no ingress packets on 5k1 or 5k2.

The egress packets from Core1 show the correct destination MAC address of the target host. The mac address table shows the mac address on po31.

At this point, I'm totally stumped. Is anyone aware of known issues or additional troubleshooting steps I can take?

Reza Sharifi · ‎03-29-2013

Is vPC running between the 5ks?

Is vPC running between the 5ks and extenders?

What is output of "sh vpc"

What device is the root and backup root?

sh spann root

HTH

rsjordan00 · ‎03-29-2013

We are running vPC between the 5ks (vpc50).

vPC is running between the extenders in a dual-homed method where each extender connects to both 5ks.

Below is sh vpc on our primary 5k. The output on our secondary is identical besides the role.

5k-primary# sh vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

Configuration consistency status : success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : primary, operational secondary

Number of vPCs configured : 394

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

Auto-recovery status : Disabled

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po50 up 1,3,5,9-10,12,14-15,50,99-100,102,230,302,318,337,

346-348,394-395,400-600

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

101 Po101 up success success 1,3,5,9-10,

12,14-15,50

,99-100,102

,230,302,31

8,337,34....

102 Po102 up success success 1,3,5,9-10,

12,14-15,50

,99-100,102

,230,302,31

8,337,34....

The show span root is below. I abreviated it because of the amount of VLANs:

5k-primary# sh spanning-tree root

Root Hello Max Fwd

Vlan Root ID Cost Time Age Dly Root Port

---------------- -------------------- ------- ----- --- --- ----------------

VLAN0001 8200 001f.0000.0c01 24001 2 14 10 port-channel50

VLAN0100 8200 001f.0000.0c03 24001 2 14 10 port-channel50

VLAN0200 8200 001f.0000.0c05 24001 2 14 10 port-channel50

5k-secondary# sh spanning-tree root

Root Hello Max Fwd

Vlan Root ID Cost Time Age Dly Root Port

---------------- -------------------- ------- ----- --- --- ----------------

VLAN0001 8200 001f.0000.0c01 24000 2 14 10 port-channel101

VLAN0100 8200 001f.0000.0c03 24000 2 14 10 port-channel101

VLAN0200 8200 001f.0000.0c05 24000 2 14 10 port-channel101

I'm a little confused on why my primary 5k is using the peer link (po50), instead of po101. Would this cause the 5k to drop the packet to avoid some kind of loop?

Masked the actual switch name in the output

Guido Arturo Catalano · ‎03-29-2013

Can you post the vpc domain setup?

is there the command peer-gateway ?

rsjordan00 · ‎03-29-2013

My previous post has the output of sh vpc. You can see that peer-gateway is disabled. I did a little research and it seems this is only needed when you have certain storage systems that don't follow Ethernet standards.

The only config specific to the vpc domain is as follows:

5k-primary#

vpc domain 1

role priority 2000

peer-keepalive destination 172.31.255.242

5k-secondary#

vpc domain 1

peer-keepalive destination 172.31.255.241

Reza Sharifi · ‎03-29-2013

Guido,

Peer-gateway is used when the device is running layer-3, not just layer-2, which is the case here.

HTH

pille1234 · ‎03-29-2013

deleted

EDIT:

Ups, after reading again I think this was a misunderstanding. never mind...

Reza Sharifi · ‎03-29-2013

I'm a little confused on why my primary 5k is using the peer link (po50), instead of po101. Would this cause the 5k to drop the packet to avoid some kind of loop?

I have the same scenario except I use 7ks instead of 6500 for routing and was confused too until I saw this in the vPC design document:

For vPC ports only, the operational primary switch generates and processes BPDUs. The operational

secondary switch forwards BPDUs to the primary switch.

So, it is also recommended to match vPC primary with the root switch and vPC secondary with the backup root switch if the same device plays both roles.

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572834-00_STDG_NX-OS_vPC_DG.pdf

HTH

pille1234 · ‎03-29-2013

rsjordan00 schrieb:

I'm a little confused on why my primary 5k is using the peer link (po50), instead of po101. Would this cause the 5k to drop the packet to avoid some kind of loop?

Masked the actual switch name in the output

That is exactly your problem: vPC loop-prevention drops your packets. Question is, why isn't port-channel 101 the root port for 5k-primary? Do a 'show spanning-tree vlan 100', a 'show running-config int po101' and a 'show port-channel summary' on both N5k please.

Regards.

rsjordan00 · ‎03-29-2013

Well, the output of sh vpc on my 5k-primary says "operational secondary". Would this explain why the vPC is the root port on this 5k? Here is the output from the requested commands:

5k-primary# sh spanning-tree vlan 100

VLAN0100

Spanning tree enabled protocol rstp

Root ID Priority 8200

Address 001f.0000.0c64

Cost 24001

Port 4145 (port-channel50)

Hello Time 2 sec Max Age 14 sec Forward Delay 10 sec

Bridge ID Priority 32868 (priority 32768 sys-id-ext 100)

Address 547f.0000.4efc

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po50 Root FWD 1 128.4145 (vPC peer-link) Network P2p

Po101 Root FWD 24000 128.4196 (vPC) P2p Peer(STP)

Po102 Altn BLK 28000 128.4197 (vPC) P2p Peer(STP)

Eth113/1/13 Desg FWD 1 128.2701 (vPC) Edge P2p

Eth114/1/5 Desg FWD 1 128.2821 (vPC) Edge P2p

Eth114/1/6 Desg FWD 1 128.2822 (vPC) Edge P2p

Eth114/1/7 Desg FWD 1 128.2823 (vPC) Edge P2p

Eth114/1/8 Desg FWD 1 128.2824 (vPC) Edge P2p

5k-primary# sh run int po101

!Command: show running-config interface port-channel101

!Time: Fri Mar 29 18:40:03 2013

version 5.2(1)N1(3)

interface port-channel101

description vPC-Uplink-Core1

switchport mode trunk

switchport trunk native vlan 50

spanning-tree cost 24000

speed 10000

vpc 101

5k-primary#

5k-primary# sh port-channel summary

Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)

s - Suspended r - Module-removed

S - Switched R - Routed

U - Up (port-channel)

M - Not in use. Min-links not met

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

50 Po50(SU) Eth LACP Eth1/27(P) Eth1/28(P) Eth1/29(P)

Eth1/30(P)

101 Po101(SU) Eth NONE Eth1/31(P)

102 Po102(SU) Eth NONE Eth1/32(P)

111 Po111(SU) Eth NONE Eth1/1(P)

112 Po112(SU) Eth NONE Eth1/2(P)

113 Po113(SU) Eth NONE Eth1/3(P)

114 Po114(SU) Eth NONE Eth1/4(P)

121 Po121(SU) Eth NONE Eth1/5(P)

122 Po122(SU) Eth NONE Eth1/6(P)

123 Po123(SU) Eth NONE Eth1/7(P)

124 Po124(SU) Eth NONE Eth1/8(P)

5k-secondary# sh spanning-tree vlan 100

VLAN0100

Spanning tree enabled protocol rstp

Root ID Priority 8200

Address 001f.0000.0c64

Cost 24000

Port 4196 (port-channel101)

Hello Time 2 sec Max Age 14 sec Forward Delay 10 sec

Bridge ID Priority 32868 (priority 32768 sys-id-ext 100)

Address 547f.0000.2f01

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po50 Desg FWD 1 128.4145 (vPC peer-link) Network P2p

Po101 Root FWD 24000 128.4196 (vPC) P2p Peer(STP)

Po102 Altn BLK 28000 128.4197 (vPC) P2p Peer(STP)

Eth113/1/13 Desg FWD 1 128.2701 (vPC) Edge P2p

Eth114/1/5 Desg FWD 1 128.2821 (vPC) Edge P2p

Eth114/1/6 Desg FWD 1 128.2822 (vPC) Edge P2p

Eth114/1/7 Desg FWD 1 128.2823 (vPC) Edge P2p

Eth114/1/8 Desg FWD 1 128.2824 (vPC) Edge P2p

5k-secondary# sh running-config int po101

!Command: show running-config interface port-channel101

!Time: Fri Mar 29 18:42:49 2013

version 5.2(1)N1(3)

interface port-channel101

description vPC-Uplink-Core1

switchport mode trunk

switchport trunk native vlan 50

spanning-tree cost 24000

speed 10000

vpc 101

5k-secondary# sh port-channel summary

Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)

s - Suspended r - Module-removed

S - Switched R - Routed

U - Up (port-channel)

M - Not in use. Min-links not met

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

50 Po50(SU) Eth LACP Eth1/27(P) Eth1/28(P) Eth1/29(P)

Eth1/30(P)

101 Po101(SU) Eth NONE Eth1/31(P)

102 Po102(SU) Eth NONE Eth1/32(P)

111 Po111(SU) Eth NONE Eth1/1(P)

112 Po112(SU) Eth NONE Eth1/2(P)

113 Po113(SU) Eth NONE Eth1/3(P)

114 Po114(SU) Eth NONE Eth1/4(P)

121 Po121(SU) Eth NONE Eth1/5(P)

122 Po122(SU) Eth NONE Eth1/6(P)

123 Po123(SU) Eth NONE Eth1/7(P)

124 Po124(SU) Eth NONE Eth1/8(P)

pille1234 · ‎03-29-2013

Well, the output of sh vpc on my 5k-primary says "operational secondary". Would this explain why the vPC is the root port on this 5k? Here is the output from the requested commands:

5k-primary# sh spanning-tree vlan 100

VLAN0100

Spanning tree enabled protocol rstp

Root ID Priority 8200

Address 001f.0000.0c64

Cost 24001

Port 4145 (port-channel50)

Hello Time 2 sec Max Age 14 sec Forward Delay 10 sec

Bridge ID Priority 32868 (priority 32768 sys-id-ext 100)

Address 547f.0000.4efc

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po50 Root FWD 1 128.4145 (vPC peer-link) Network P2p

Po101 Root FWD 24000 128.4196 (vPC) P2p Peer(STP)

Po102 Altn BLK 28000 128.4197 (vPC) P2p Peer(STP)

Eth113/1/13 Desg FWD 1 128.2701 (vPC) Edge P2p

Eth114/1/5 Desg FWD 1 128.2821 (vPC) Edge P2p

Eth114/1/6 Desg FWD 1 128.2822 (vPC) Edge P2p

Eth114/1/7 Desg FWD 1 128.2823 (vPC) Edge P2p

Eth114/1/8 Desg FWD 1 128.2824 (vPC) Edge P2p

Alright, I misinterpreted the output of 'sh spanning-tree root' that you showed earlier. What we see here is exactly as it is supposed to be: peerlink and vPC to Core are root ports. This is the correct output for vPC secondary devices.

However the Type "Peer(STP)" is confusing. I don't know if it is related to your problem, but from the look of it I believe you are using a different type of STP on your Core switches. Is this done with intention?

rsjordan00 · ‎04-01-2013

Our access switches are set to rpvst but our core switches run pvst. Not for any particular reason, we just don't want to take the massive hit (~100 switches) to configure our core to rpvst.

I opened a TAC with Cisco and they believe it could be related to the load balancing algorithm on the core's etherchannel. On both sides of the etherchannel we are using the default src-dst-ip. They want to change it to src-dst-mac, but I need more justification before I make a change that will disrupt all my etherchannels. I'm waiting to hear back from them.

pille1234 · ‎04-01-2013

Our access switches are set to rpvst but our core switches run pvst. Not for any particular reason, we just don't want to take the massive hit (~100 switches) to configure our core to rpvst.

That's certainly another topic, but doesn't that drastically increase convergence times?

I opened a TAC with Cisco and they believe it could be related to the load balancing algorithm on the core's etherchannel. On both sides of the etherchannel we are using the default src-dst-ip. They want to change it to src-dst-mac, but I need more justification before I make a change that will disrupt all my etherchannels. I'm waiting to hear back from them.

While I don't believe that changing the lb methods causes disruption of traffic, let alone the disruption of your port channels, I agree that this 'solution' does not sound very reasonable, except Cisco has knowledge about a specific software bug that we don't know about.

Do you have a specific reason for configuring the spanning tree cost on the uplink ports?

You don't use LACP for your uplinks, but can we rule out a simple wiring error anyway? Have you checked with cdp neighbors? I'm just guessing a little...

Regards

Pille

rsjordan00 · ‎04-01-2013

I leaned back on them about the LB algorithm and they agreed there is not enough evidence to warrant this change. The engineer told me there would be a disruption if I were to change it.

We configured the spanning tree cost because both 5ks are uplinked to our 6500 using 10g. The interswitch link between our cores is a 4 member etherchannel on 1g. We set a higher cost on the uplinks otherwise traffic between our cores would go through the 5ks.

Tonight, I'll be shutting the interface from core1 to 5k2-secondary. If my pings work then it at least narrows the problem a little further.

Abed Alrahim Alakhras · ‎03-30-2013

You cant route to VPC peers as long as you're using a VPC port-channel, just use a plain p2p link or a L3 port-channel to a each Nexus.This should resolve ur issue.