How to you troubleshoot LACP? No ip communication between switches...

[Brad Hodgins]

This problem has stumped me for days (and nights).

We have a multi-vendor network consisting of Cisco and Avaya. We are now trying to merge 2 Brocade VDX switches as TOR for our VSphere infrastructure.

Both VDXs are in VCS fabric mode. VLANS 186 and 200 are defined in the Cisco infrastructure, as well as the Brocade. Creating Etherchannel was done without incident on a 2960x.

The VLAG was created on the VDXs, no problem, but I cannot communicate across that link no matter what.

Servers A and B connected to VDXs. - Both are 192.168.1.x/22 on VLAN 1 and 10.10.10.x/24 on VLAN 200
Servers C and D connected to Cisco network. Both are 192.168.1.x on VLAN 1 and 10.10.10.x/24 on VLAN 200.

Server A can talk to B, so the Ethernet Fabric is working
Server C can talk to D (production network)

A and B cannot talk to C or D and vice-versa. The are connected via 2960 te1/0/1 and 1/0/2 on port channel 24
This is not a media issue, as remote MACs are populating the port channel tables. On the 2960x when I enter sh mac addr int po 24, I can see the MAC addresses of servers A and B on both VLAN 1 and 200 repopulating within seconds after clearing it. Same with sh lldp nei

Same with the other way on the Brocade port channels.

On the Cisco side the Etherchannel is set to LACP, active.

Here is some of the setup, if someone help me out, maybe I can finally get some sleep.

Thanks !!!

Cisco 2960x port group

2960x-48P#sh run in po 24
Building configuration...

Current configuration : 112 bytes
!
interface Port-channel24
switchport mode trunk
switchport nonegotiate
spanning-tree bpdufilter enable
end

2960x-48P#sh run in te 1/0/1
Building configuration...

Current configuration : 129 bytes
!
interface TenGigabitEthernet1/0/1
switchport mode trunk
switchport nonegotiate
no cdp enable
channel-group 24 mode on
end

2960x-48P#sh run int te 1/0/2
Building configuration...

Current configuration : 129 bytes
!
interface TenGigabitEthernet1/0/2
switchport mode trunk
switchport nonegotiate
no cdp enable
channel-group 24 mode on

2960x-48P#sh int po 24
Port-channel24 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 2c3e.cf9c.59b4 (bia 2c3e.cf9c.59b4)
  MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, link type is auto, media type is unknown
  input flow-control is off, output flow-control is unsupported
  Members in this channel: Te1/0/1 Te1/0/2
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 19:36:27, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 2000 bits/sec, 2 packets/sec
  5 minute output rate 11000 bits/sec, 13 packets/sec

VDX-1# sh run int te 1/0/24
interface TenGigabitEthernet 1/0/24
no fabric isl enable
no fabric trunk enable
channel-group 24 mode active type standard
lacp timeout long
no shutdown

VDX-1# sh run in po 24
interface Port-channel 24
vlag ignore-split
switchport
switchport mode trunk
switchport trunk allowed vlan all
switchport trunk tag native-vlan
no shutdown

VDX-1# sh lldp nei
Local Intf   Dead Interval  Remaining Life  Remote Intf                   Chassis ID        Tx           Rx
Te 1/0/17    180            155             0050.5654.6278                vmnic4            16218        7963
Te 1/0/18    180            148             0050.5654.6c84                vmnic4            16219        7963
Te 1/0/19    180            165             0050.5654.6aa0                vmnic4            16223        7964
Te 1/0/20    180            138             0000.0000.0000                vmnic4            16222        7963
Te 1/0/24    120            115             Te1/0/2                       2c3e.cf9c.5980    1826         2171--------------This is our Cisco 2960x

VDX-2# sh run int te 2/0/24
interface TenGigabitEthernet 2/0/24
no fabric isl enable
no fabric trunk enable
channel-group 24 mode active type standard
lacp timeout long
no shutdown

VDX-2# sh run int po 24
interface Port-channel 24
vlag ignore-split
switchport
switchport mode trunk
switchport trunk allowed vlan all
switchport trunk tag native-vlan
no shutdown

sh lldp nei
Local Intf   Dead Interval  Remaining Life  Remote Intf                   Chassis ID        Tx           Rx
Te 2/0/9     180            166             0050.5652.1a80                vmnic0            16201        2998
Te 2/0/10    180            159             0050.5651.d128                vmnic0            16205        2998
Te 2/0/11    180            177             0050.5652.1af8                vmnic0            16201        2998
Te 2/0/17    180            166             0050.5654.6a7c                vmnic6            16201        7965
Te 2/0/18    180            159             0050.5654.6a9c                vmnic6            16201        7965
Te 2/0/19    180            177             0050.5654.8758                vmnic6            16205        7965
Te 2/0/20    180            150             0000.0000.0000                vmnic6            16205        7965
Te 2/0/24    120            103             Te1/0/1                       2c3e.cf9c.5980    16131        16074

VDX-2# sh fab all
VCS Id: 10
Config Mode: Local-Only

Rbridge-id           WWN             IP Address         Name
----------------------------------------------------------------------------
   1       10:00:00:27:F8:F9:C9:E1   192.168.136.232   >"VDX-1"
   2       10:00:00:27:F8:F9:3C:B9   192.168.136.233    "VDX-2"*

The Fabric has 2 Rbridge(s)

VDX-2# sh fab islports
Name:       VDX-2
Type:       95.2
State:      Online
Role:       Fabric Subordinate
VCS Id:     10
Config Mode:Local-Only
Rbridge-id: 2
WWN:        10:00:00:27:f8:f9:3c:b9
FCF MAC:    00:27:f8:f9:3c:b9

Index   Interface      State   Operational State
===================================================================
  0     Te 2/0/1          Up    ISL (Trunk port, Primary is Te 2/0/2 )
  1     Te 2/0/2          Up    ISL 10:00:00:27:f8:f9:c9:e1 "VDX-1" (upstream)(Trunk Primary)
  2     Te 2/0/3          Down
  3     Te 2/0/4          Down
  4     Te 2/0/5          Down
  5     Te 2/0/6          Down
  6     Te 2/0/7          Down
  7     Te 2/0/8          Down
  8     Te 2/0/9          Down
  9     Te 2/0/10         Down
10     Te 2/0/11         Down
11     Te 2/0/12         Down
12     Te 2/0/13         Down
13     Te 2/0/14         Down
14     Te 2/0/15         Down
15     Te 2/0/16         Down
16     Te 2/0/17         Down
17     Te 2/0/18         Down
18     Te 2/0/19         Down
19     Te 2/0/20         Down
20     Te 2/0/21         Down
21     Te 2/0/22         Down
22     Te 2/0/23         Down
23     Te 2/0/24         Down  Down (Admin)

[Steve Fuller · Accepted solution]

Hi,

I don't know the Brocade switches, but a couple of things look odd here:

1) The Brocade switch has the command switchport trunk tag native-vlan on its port-channel interface, which means that its expecting IEEE 802.1Q tags on frames it receives, and adds tags to those it transmits.

I don't see the switchport trunk native vlan tag command configured on the Catalyst interface so unless you're using the global vlan dot1q tag native, the trunking on the two switches is not compatible and could be part of the problem.

Try adding no switchport trunk tag native-vlan to the Brocade or adding the switchport trunk native vlan tag command on the interface of the Catalyst switch.

2) You say you're using Link Aggregation Control Protocol (LACP), but the Catalyst switch is configured with channel-group 24 mode on. This is not using LACP, but rather it's a static LAG. This means it does not send LACPDU.

By contrast the Brocade is configured with channel-group 24 mode active type standard on its interfaces and so sends LACPDU and also expects to receive them. As it is not going to be receiving them I suspect the port-channel is not formed correctly.

You should change the command on the Cisco switch to channel-group 24 mode active.

Try the above and then using the show etherchannel summary command ensure the port-channel shows the S (Layer-2) and U (in use) flags, and that both interface appear in the ports column with the P (bundled) flag.

You can verify whether the switch is sending and receiving LACPDU with the show lacp counters command, and the show lacp neighbor should provide details of the Brocade switch.

Regards

[Brad Hodgins]

Thank you Steve,

I had since corrected the LACP setting on the 2960x, but still no joy. I am seeing the Brocade port channel MAC address on the sh lacp nei though. Here are my results:

Group  Port-channel  Protocol    Ports
------+-------------+-----------+----------------------------------------------
1      Po1(SU)          -        Gi1/0/47(P) Gi1/0/48(P)
2      Po2(SU)         LACP      Te1/0/1(P)  Te1/0/2(P)

Channel group 2 neighbors

Partner's information:

                  LACP port                        Admin  Oper   Port    Port
Port      Flags   Priority  Dev ID          Age    key    Key    Number  State
Te1/0/1   SA      32768     01e0.5200.0001  23s    0x0    0x18   0x418   0x3D
Te1/0/2   SA      32768     01e0.5200.0001   8s    0x0    0x18   0x218   0x3D
2960x-48P#sh lacp count
             LACPDUs         Marker      Marker Response    LACPDUs
Port       Sent   Recv     Sent   Recv     Sent   Recv      Pkts Err
---------------------------------------------------------------------
Channel group: 2
Te1/0/1     1040   964      0      0        0      0         0
Te1/0/2     1047   967      0      0        0      0         0

Brocade VDX:

sw0# sh lacp sys-id
 Port-channel Po 24 - System ID: 0x8000,01-e0-52-00-00-01 - SID Master: N/A

sw0# sh lacp count
 Traffic statistics
Port            LACPDUs            Marker           Pckt err
             Sent     Recv     Sent     Recv     Sent     Recv
 Aggregator Po 24
Te 1/0/24    2457     1440     0        0        0        0

I'm not sure what to do with the native VLAN tagging on that port channel.

[Steve Fuller]

Hi,

So the port-channel is now up and working correctly now so some progress at least.

As far as tagging of the native VLAN, assuming the Brocade switches also use VLAN 1 as their native VLAN, any issue here would only affect VLAN 1. You said you also had servers in VLAN 200 using 10.10.10.x/24. Are you able to verify whether you have any connectivity between them?

When I look back at you original post I can also see you have spanning-tree bpdufilter enable configured on the port-channel interface. Is there a specific reason for this? It's never a good idea to disable spanning tree if it can be avoided.

Also can you post the output of a show spanning-tree interface port-channel 24 from the Cisco switch? I don't believe this would be a problem as you said you were seeing MAC addresses being learned across the port-channel, but lets at least make sure there's nothing odd happening there.

Regards

[Brad Hodgins]

Steve,

No connectivity to VLAN 200 either. We have many VLANs, all being routed by an older Avaya stack. I chose to start with 200 for simplicity's sake because it's not routed, so I figured that two hosts in the same subnet, same VLAN would have no trouble talking. No luck here either. Not even an entry in the MAC table for 200 after a ping.

Funny thing is that I've seen VLAN 200 in this table before. Previously I was running RSTP on the 2960x, and thinking that it could have been the problem, as the Avaya systems don't recognize it, so yesterday I changed the mode to MST. Before the change the 2960x thought that it was the root bridge for all VLANs. (and looking now it still does). But that wouldn't affect VLAN 200 right?

I have bpdu filter enabled on PO2 because in fabric mode, the only thing the Brocade switches do with a bpdu is flood it out all ports, so the recommendation from Brocade in the initial setup was to turn it off. Currently we have no other 10G fiber, and the VDX cluster is 10G fiber only, so I'm not worried about someone accidentally plugging in.

Output below. Nothing being blocked...

sh spann int po 2

Mst Instance        Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
MST0                Desg FWD 1000      128.464  P2p

2960x-48P#sh spann int po 2 det
 Port 464 (Port-channel2) of MST0 is designated forwarding
   Port path cost 1000, Port priority 128, Port Identifier 128.464.
   Designated root has priority 32768, address 2c3e.cf9c.5980
   Designated bridge has priority 32768, address 2c3e.cf9c.5980
   Designated port id is 128.464, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default, Internal
   Bpdu filter is enabled
   BPDU: sent 0, received 0

Thank you!

Brad

[Steve Fuller]

Hi Brad,

Looking around I came across a couple of Brocade design guides. There's the Brocade Data Center Solution-Design Guide: Connecting Brocade VCS Fabric with NOS 3.0 to Existing STP Networks. This document talks about passing STP BPDU straight through and so I presume this is the setup you have, rather than a NOS 4.0.X setup as shown in Data Center Solution-Design Guide: Connecting VCS Fabric to Spanning Tree Protocol (STP) Networks?

Does the Cisco switch see any MACs learned via the port-channel interface now? You said previously you were seeing them, but your comment "Not even an entry in the MAC table for 200 after a ping" seems to suggest nothing is being learned now.

What do you see if you run the command show mac address interface port-channel 24? Do you see any MAC addresses and are they those assigned to the servers connected to the VDX fabric?

Regards

[Brad Hodgins]

Steve,

You are correct, we are currently using Brocade NOS 3.01c. Scratch the comment earlier on the STP root incorrectly being the 2960x. It is the Avaya stack.

I cleared the dynamic MACs from the 2960x, and immediately the 4 hosts repopulate on port channel 2. All on VLAN 1 though. Nothing on VLAN 200, despite trying to ping a host there. I've even had someone directly connect a laptop onto the 2960x and configured it with a static IP on the 200 VLAN to eliminate any possible upstream problems.

However, I am no longer seeing anything in the Brocade MAC table for it's side of the port channel.The LACPDUs are still incrementing on either side through.

Brad

[Steve Fuller]

Hi Brad,

From what you posted previously the LAG is bundling the physical interfaces and LACP seems to be working OK. What we have is more of a trunking / spanning tree issue whereby the LAG is carrying the VLANs correctly or spanning tree is blocking it for some reason.

Are you able to post a diagram of the topology so I can see what the connectivity is between the Cisco switch, the Brocade VDX and the Avaya routers? If you can do that can you also indicate where the servers A, B, C and D that you referred to previously are connected and the IP addresses assigned to them?

Where is the Avaya stack connected and do any of the servers on the Cisco or Brocade switches have connectivity to that?

Additionally could you attach the configurations from the Cisco and Brocade switches?

Regards

[Brad Hodgins]

WOW! Finally getting somewhere! The problem was twofold: lack of port channel config sync between Brocade switches, and default VLAN tagging.

On the Brocade VDX cluster IP I had created the port channel 24, as well as configured the trunking of VLANs. Unlike the Cisco IOS where the ports need have identical configurations before they are added to the port channel, the Brocade requires there be no configuration on the switchports being aggregated, no access, no trunk. That I understood, but where the problem was, as I made the port channel config on the cluster IP, it made the port channel 24 across both VDX1 and VDX2, but the configuration of the switchport and trunking of all VLANs was only happening on the primary switch for port channel 24. When I ran a sh run int po 24 on VDX1 I saw the expected config and was confused as to why it wan't working. It wasn't until I logged on to VDX2 and ran the same that I found the port channel 24 (which was created from the master) had an empty config.

After entering the config again on VDX2, I find VLAN 200 coming to life, both ways! I added on a routed VLAN 132, which is being routed through out Avaya infrastructure, and it came up as well, immediately populating the port channel mac table.

VLAN 1 which is where most of our production nodes are (I inherited this structure) continues to be a show stopper. Servers A and B's MAC addresses come up on the 2960x's mac table for port channel 2, but servers C and D's MACs do not populate the VDXs MAC table in relation to it's port channel 24.

Actually there is only one MAC entry in that list for VLAN 1, and it is for an ASUSTEK manufacturer, which confuses me even further.

I believe that the icmp packets are making it through to the server connected to the VDX as the sh int command shows incrementing receive and transmit wit ha continuous ping, however I believe that the VDX is tagging the default VLAN as it leaves on port channel 24, destined to return the reply towards the 2960x.

Changing the production VLAN 1 isn't something easily done right now. How can I address the default VLAN issue?

Brad

[Brad Hodgins]

And it's all working now. After entering the no switchport trunk tag native-vlan on port channel 24 on both Brocade VDX switches, all is now working.

It really wasn't what I'm used to. Thank you Steve for pointing me in the right direction, getting LACP working correctly.

This really was a Brocade config issue. I had posted originally to the Brocade forums, but after seeing a lot of questions with a 1000 views before a response, I thought I'd ask here. After all it was a switching issue, just really odd how some manufacturers systems work.

Brad

[Steve Fuller]

Hi Brad,

Glad to hear it's all working now.

Regards