Solved: Issues with port-channel not passing traffic in a L2 ring

chad-young · ‎03-18-2011

Hi all! First time poster, so please be gentle. My thanks in advance to anyone who can assist.

Anyway, here goes. I have three sites, all of which hosts various VLANs for multiple kinds of hosts. A previous tech had set up a trio of L2 switches in a ring and trunked all the active VLANs on the three switches to each other. The idea was to be able to connect to the VLANs at any of the three sites and have a redundant L2 path between all the sites. I upgraded one leg to an etherchannel combining two ports into a single 2Gbps pipeline thhrough a Canary CCN-2000 CWDM multiplexer. All links are up, all trunks are up, and the port-channel is up.

Here is the issue - that I can tell, traffic from Site A to Site C is almost entireley *not* going over the faster etherchannel. It is choosing the slower path through Site B. AFAIK, spanning tree seems to see the A-C trunk and assigns it a lower cost, but, as far as I can tell, nearly all the traffic is all going taking the other connection. I admit, though, that my STP-fu is not strong.

I suspect traffic taking the A->B->C path may been the case for a while. I only noticed it today when adding VLAN 3012 to the trunk. I made a typo and brought the A->C port-channel down briefly and nothing lost connectivity. The previous tech noted that when the Site B switch was rebooted last year, traffic did not appear to revert to the A->C direct path.

Am I missing something here? What do you think might be going on?

Site A to site C trunk config:

Side A Switch:

!
interface Port-channel1
switchport trunk allowed vlan 22,27,33,35,46,49,70,91,95,97-99,101,253,400,777
switchport trunk allowed vlan add 1099,1100,2001,2002,3012
switchport mode trunk

!
interface GigabitEthernet0/47
description Ring Trunks CDC
switchport trunk allowed vlan 22,27,33,35,46,49,70,91,95,97-99,101,253,400,777
switchport trunk allowed vlan add 1099,1100,2001,2002,3012
switchport mode trunk
no cdp enable
channel-group 1 mode on
!
interface GigabitEthernet0/48
description Ring Trunks CDC
switchport trunk allowed vlan 22,27,33,35,46,49,70,91,95,97-99,101,253,400,777
switchport trunk allowed vlan add 1099,1100,2001,2002,3012
switchport mode trunk
no cdp enable
channel-group 1 mode on
!

Site C Switch:

!

interface Port-channel1
switchport trunk allowed vlan 22,27,33,35,46,49,70,91,95,97-99,101,253,400,777
switchport trunk allowed vlan add 1099,1100,2001,2002
switchport mode trunk
!

interface GigabitEthernet0/47
description Ring Trunks Admin
switchport trunk allowed vlan 22,27,33,35,46,49,70,91,95,97-99,101,253,400,777
switchport trunk allowed vlan add 1099,1100,2001,2002
switchport mode trunk
channel-group 1 mode on
!
interface GigabitEthernet0/48
description Ring Trunks Admin
switchport trunk allowed vlan 22,27,33,35,46,49,70,91,95,97-99,101,253,400,777
switchport trunk allowed vlan add 1099,1100,2001,2002
switchport mode trunk
channel-group 1 mode on
!

From site C Switch:

Cleared counters, waited a few minutes and then:

SiteC#sh int port-channel 1
Port-channel1 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 000a.b841.ddb0 (bia 000a.b841.ddb0)
MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is unknown
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi0/47 Gi0/48
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:01:23
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 58000 bits/sec, 45 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     3724 packets output, 596017 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

SiteC#sh int port-channel 1 trunk

Port Mode Encapsulation Status Native vlan
Po1 on 802.1q trunking 1

Port Vlans allowed on trunk
Po1 22,27,33,35,46,49,70,91,95,97-99,101,253,400,777,1099-1100,2001-2002

Port Vlans allowed and active in management domain
Po1 27,33,35,46,49,70,91,95,97-99,101,400,777,1099-1100,2001-2002

Port Vlans in spanning tree forwarding state and not pruned
Po1 27,33,35,46,49,70,91,95,97-99,101,400,777,1099-1100,2001-2002

Kishore Chennupati · ‎03-19-2011

Hi,

If the MUX's are ok then you might want to check your SFP's as well. The issue could very well be at your end too

Regards,

Kishore

View solution in original post

samavedula_rama · ‎03-18-2011

Hi Chad...Happy Friday.

I know this is kinda dumb but did you already add a test vlan just between A and C [on the port-channel] and try pinging ? Just want to make sure there isn't an issue with port-channel itself.

Also, just pick a vlan in the ring and post "sh spanning-tree vlan " output.

Thanks.

Atif Awan · ‎03-18-2011

You need to understand your Layer-2 topology to be able to get an idea of why traffic is not taking the desired path. A good point to start will be to determine what your root bridge for a VLAN is (assuming you are running a PVST variant). Take a couple of sample VLANs and map their Layer-2 topology using the 'show spanning-tree' commands. Once you have that I am sure you will understand what is going on; may be the root switch needs to be manually enforced. If you still need assistance do share the relevant command outputs with us for one or two VLANs.

Atif

glen.grant · ‎03-18-2011

Was spanning tree roots ever defined for all these vlans ? Also I would look to see which port is blocked on what switch as in this design it has a built in loop for all vlans so it has to block one of the ports in the design so all traffic for a given vlan can only flow one way depending on what port is blocking for that vlan as cisco uses per vlan spanning tree .

chad-young · ‎03-18-2011

Ok, got the spanning-tree infor for two VLANs from the Site A Switch:

SiteA#sh spanning-tree vlan 27

VLAN0027
Spanning tree enabled protocol ieee
Root ID    Priority    32768
             Address     0014.2285.fee3
             Cost        8
             Port        2 (GigabitEthernet0/2)
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority    32795 (priority 32768 sys-id-ext 27)
             Address     000a.b841.e200
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec
             Aging Time 300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi0/2               Root FWD 4         128.2    P2p
Gi0/46              Desg FWD 4         128.46   P2p
Po1                 Desg FWD 3         128.56   P2p

SiteA#sh spanning-tree vlan 35

VLAN0035
Spanning tree enabled protocol ieee
Root ID    Priority    32768
             Address     0012.a9ef.8960
             Cost        16
             Port        46 (GigabitEthernet0/46)
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority    32803 (priority 32768 sys-id-ext 35)
             Address     000a.b841.e200
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec
             Aging Time 300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi0/46              Root FWD 4         128.46   P2p
Po1                 Desg FWD 3         128.56   P2p

When I get back in the office later next week, I will create a test VLAN with IP addresses (none of the VLANs have IP addresses except one mgmt VLAN) and try to ping across the Port-channel link.

Atif Awan · ‎03-18-2011

chad-young wrote:

Ok, got the spanning-tree infor for two VLANs from the Site A Switch:

SiteA#sh spanning-tree vlan 27

VLAN0027
Spanning tree enabled protocol ieee
Root ID    Priority    32768
             Address     0014.2285.fee3
             Cost        8
             Port        2 (GigabitEthernet0/2)
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority    32795 (priority 32768 sys-id-ext 27)
             Address     000a.b841.e200
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec
             Aging Time 300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi0/2               Root FWD 4         128.2    P2p
Gi0/46              Desg FWD 4         128.46   P2p
Po1                 Desg FWD 3         128.56   P2p

SiteA#sh spanning-tree vlan 35

VLAN0035
Spanning tree enabled protocol ieee
Root ID    Priority    32768
             Address     0012.a9ef.8960
             Cost        16
             Port        46 (GigabitEthernet0/46)
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority    32803 (priority 32768 sys-id-ext 35)
             Address     000a.b841.e200
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec
             Aging Time 300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi0/46              Root FWD 4         128.46   P2p
Po1                 Desg FWD 3         128.56   P2p

When I get back in the office later next week, I will create a test VLAN with IP addresses (none of the VLANs have IP addresses except one mgmt VLAN) and try to ping across the Port-channel link.

VLAN 27 seems to be a local Site-A VLAN as the root port is not one of your uplinks. For VLAN 35 my guess is that The switch in Site B is the root switch. If that is the case then Site C will probably have it's port-channel port blocking for this VLAN. Like I said in my previous post you need to map the complete Layer-2 topology for one or two VLANs and only then will you be able to appreciate what is going on. I am leaning towards incorrect root bridge mapping here as the cause of sub-optimal traffic flow but you need to execute the spanning-tree command on each of the three switches to map the Layer-2 topology for the VLAN.

Atif

chad-young · ‎03-18-2011

First off, thanks again.

I think I see where you folks are headed and makes sense. My lack of STP knowledge is hindering me a bit, but I will do some reading over the weekend on the matter.

That said, I tracked one high-use VLAN with hosts at Site A and Site C and mapped out the VLAN and discovered that Site B was claiming root bridge for this VLAN:

Site A:

VLAN0091

Spanning tree enabled protocol ieee

Root ID Priority 32859

Address 000a.b825.a700

Cost 4

Port 46 (GigabitEthernet0/46)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32859 (priority 32768 sys-id-ext 91)

Address 000a.b841.e200

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi0/18 Desg FWD 4 128.18 P2p

Gi0/20 Desg FWD 4 128.20 P2p

Gi0/21 Desg FWD 4 128.21 P2p

Gi0/22 Desg FWD 4 128.22 P2p

Gi0/23 Desg FWD 19 128.23 P2p

Gi0/25 Desg FWD 4 128.25 P2p

Gi0/28 Desg FWD 4 128.28 P2p

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi0/29 Desg FWD 4 128.29 P2p

Gi0/30 Desg FWD 4 128.30 P2p

Gi0/46 Root FWD 4 128.46 P2p

Po1 Desg FWD 3 128.56 P2p

Site B:

VLAN0091

Spanning tree enabled protocol ieee

Root ID Priority 32859

Address 000a.b825.a700

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32859 (priority 32768 sys-id-ext 91)

Address 000a.b825.a700

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi0/23 Desg FWD 4 128.23 P2p

Gi0/24 Desg FWD 4 128.24 P2p

Site C:

VLAN0091

Spanning tree enabled protocol ieee

Root ID Priority 32859

Address 000a.b825.a700

Cost 4

Port 45 (GigabitEthernet0/45)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32859 (priority 32768 sys-id-ext 91)

Address 000a.b841.dd80

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi0/18 Desg FWD 19 128.18 P2p

Gi0/22 Desg FWD 4 128.22 P2p

Gi0/23 Desg FWD 4 128.23 P2p

Gi0/27 Desg FWD 19 128.27 P2p

Gi0/28 Desg FWD 4 128.28 P2p

Gi0/30 Desg FWD 4 128.30 P2p

Gi0/31 Desg FWD 4 128.31 P2p

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi0/32 Desg FWD 4 128.32 P2p

Gi0/37 Desg FWD 4 128.37 P2p

Gi0/45 Root FWD 4 128.45 P2p

Po1 Desg FWD 3 128.56 P2p

So, question is, how do I fix this?

Atif Awan · ‎03-18-2011

Before we fix this I see something strange here. I would have expected either the switch at Site-A or switch at Site-C to put one of it's ports in blocking for this VLAN but this is not the case. Can you please post 'show spanning-tree vlan 91 detail' from both switch-A and switch-C? The output might be long so just attach it as a text file.

Atif

chad-young · ‎03-18-2011

File attached for VLAN 91 on Site C and Site A.

Atif Awan · ‎03-18-2011

Chad,

There appears to be a communicatoin issue over the port-channel interface. Both switches claim not to receive any BPDUs over this interface.

From Switch-A:

Port 56 (Port-channel1) of VLAN0091 is designated forwarding

Port path cost 3, Port priority 128, Port Identifier 128.56.

Designated root has priority 32859, address 000a.b825.a700

Designated bridge has priority 32859, address 000a.b841.e200

Designated port id is 128.56, designated path cost 4

Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1

Link type is point-to-point by default

BPDU: sent 50101, received 0

From Switch-C:

Port 56 (Port-channel1) of VLAN0091 is designated forwarding

Port path cost 3, Port priority 128, Port Identifier 128.56.

Designated root has priority 32859, address 000a.b825.a700

Designated bridge has priority 32859, address 000a.b841.dd80

Designated port id is 128.56, designated path cost 4

Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1

Link type is point-to-point by default

BPDU: sent 10190452, received 0

There should be a non-zero value in the received field on both switches. I would like you to verify forwarding on both member links first before you make any changes to alter the root bridge.

Atif

samavedula_rama · ‎03-18-2011

Looks like an issue with the media converter. If they are fiber ports, try doing a speed nonegotiate.

If they are not, try hard-coding speed & Duplex settings.

chad-young · ‎03-18-2011

Based on some other issues with that MUX, I suspect you may be right and it is the culprit. I will have to schedule some down time and test further. I might have to use another fiber path and bypass that MUX entirely.

Thanks! I will update when I get through with testing. It may be a few weeks.

Kishore Chennupati · ‎03-19-2011

Hi,

If the MUX's are ok then you might want to check your SFP's as well. The issue could very well be at your end too

Regards,

Kishore

chad-young · ‎03-21-2011

Well, the connections from Site A to Site C are regular copper Gbit on the switch, then get converted to fiber in the Mux, then back to Copper on the other end. Ergo, no sfps on that connection.

chad-young · ‎03-25-2011

Hi all! I am going to mark this as resolved now. The Mux is definitely not behaving, so we are going to switch that to a SM fiber pathway and simplify things a bit.

Thank you everyone for your kind assistance.