08-08-2015 04:31 PM - edited 03-08-2019 01:17 AM
This problem has stumped me for days (and nights).
We have a multi-vendor network consisting of Cisco and Avaya. We are now trying to merge 2 Brocade VDX switches as TOR for our VSphere infrastructure.
Both VDXs are in VCS fabric mode. VLANS 186 and 200 are defined in the Cisco infrastructure, as well as the Brocade. Creating Etherchannel was done without incident on a 2960x.
The VLAG was created on the VDXs, no problem, but I cannot communicate across that link no matter what.
Servers A and B connected to VDXs. - Both are 192.168.1.x/22 on VLAN 1 and 10.10.10.x/24 on VLAN 200
Servers C and D connected to Cisco network. Both are 192.168.1.x on VLAN 1 and 10.10.10.x/24 on VLAN 200.
Server A can talk to B, so the Ethernet Fabric is working
Server C can talk to D (production network)
A and B cannot talk to C or D and vice-versa. The are connected via 2960 te1/0/1 and 1/0/2 on port channel 24
This is not a media issue, as remote MACs are populating the port channel tables. On the 2960x when I enter sh mac addr int po 24, I can see the MAC addresses of servers A and B on both VLAN 1 and 200 repopulating within seconds after clearing it. Same with sh lldp nei
Same with the other way on the Brocade port channels.
On the Cisco side the Etherchannel is set to LACP, active.
Here is some of the setup, if someone help me out, maybe I can finally get some sleep.
Thanks !!!
Cisco 2960x port group
2960x-48P#sh run in po 24
Building configuration...
Current configuration : 112 bytes
!
interface Port-channel24
switchport mode trunk
switchport nonegotiate
spanning-tree bpdufilter enable
end
2960x-48P#sh run in te 1/0/1
Building configuration...
Current configuration : 129 bytes
!
interface TenGigabitEthernet1/0/1
switchport mode trunk
switchport nonegotiate
no cdp enable
channel-group 24 mode on
end
2960x-48P#sh run int te 1/0/2
Building configuration...
Current configuration : 129 bytes
!
interface TenGigabitEthernet1/0/2
switchport mode trunk
switchport nonegotiate
no cdp enable
channel-group 24 mode on
2960x-48P#sh int po 24
Port-channel24 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 2c3e.cf9c.59b4 (bia 2c3e.cf9c.59b4)
MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is unknown
input flow-control is off, output flow-control is unsupported
Members in this channel: Te1/0/1 Te1/0/2
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 19:36:27, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2000 bits/sec, 2 packets/sec
5 minute output rate 11000 bits/sec, 13 packets/sec
VDX-1# sh run int te 1/0/24
interface TenGigabitEthernet 1/0/24
no fabric isl enable
no fabric trunk enable
channel-group 24 mode active type standard
lacp timeout long
no shutdown
VDX-1# sh run in po 24
interface Port-channel 24
vlag ignore-split
switchport
switchport mode trunk
switchport trunk allowed vlan all
switchport trunk tag native-vlan
no shutdown
VDX-1# sh lldp nei
Local Intf Dead Interval Remaining Life Remote Intf Chassis ID Tx Rx
Te 1/0/17 180 155 0050.5654.6278 vmnic4 16218 7963
Te 1/0/18 180 148 0050.5654.6c84 vmnic4 16219 7963
Te 1/0/19 180 165 0050.5654.6aa0 vmnic4 16223 7964
Te 1/0/20 180 138 0000.0000.0000 vmnic4 16222 7963
Te 1/0/24 120 115 Te1/0/2 2c3e.cf9c.5980 1826 2171--------------This is our Cisco 2960x
VDX-2# sh run int te 2/0/24
interface TenGigabitEthernet 2/0/24
no fabric isl enable
no fabric trunk enable
channel-group 24 mode active type standard
lacp timeout long
no shutdown
VDX-2# sh run int po 24
interface Port-channel 24
vlag ignore-split
switchport
switchport mode trunk
switchport trunk allowed vlan all
switchport trunk tag native-vlan
no shutdown
sh lldp nei
Local Intf Dead Interval Remaining Life Remote Intf Chassis ID Tx Rx
Te 2/0/9 180 166 0050.5652.1a80 vmnic0 16201 2998
Te 2/0/10 180 159 0050.5651.d128 vmnic0 16205 2998
Te 2/0/11 180 177 0050.5652.1af8 vmnic0 16201 2998
Te 2/0/17 180 166 0050.5654.6a7c vmnic6 16201 7965
Te 2/0/18 180 159 0050.5654.6a9c vmnic6 16201 7965
Te 2/0/19 180 177 0050.5654.8758 vmnic6 16205 7965
Te 2/0/20 180 150 0000.0000.0000 vmnic6 16205 7965
Te 2/0/24 120 103 Te1/0/1 2c3e.cf9c.5980 16131 16074
VDX-2# sh fab all
VCS Id: 10
Config Mode: Local-Only
Rbridge-id WWN IP Address Name
----------------------------------------------------------------------------
1 10:00:00:27:F8:F9:C9:E1 192.168.136.232 >"VDX-1"
2 10:00:00:27:F8:F9:3C:B9 192.168.136.233 "VDX-2"*
The Fabric has 2 Rbridge(s)
VDX-2# sh fab islports
Name: VDX-2
Type: 95.2
State: Online
Role: Fabric Subordinate
VCS Id: 10
Config Mode:Local-Only
Rbridge-id: 2
WWN: 10:00:00:27:f8:f9:3c:b9
FCF MAC: 00:27:f8:f9:3c:b9
Index Interface State Operational State
===================================================================
0 Te 2/0/1 Up ISL (Trunk port, Primary is Te 2/0/2 )
1 Te 2/0/2 Up ISL 10:00:00:27:f8:f9:c9:e1 "VDX-1" (upstream)(Trunk Primary)
2 Te 2/0/3 Down
3 Te 2/0/4 Down
4 Te 2/0/5 Down
5 Te 2/0/6 Down
6 Te 2/0/7 Down
7 Te 2/0/8 Down
8 Te 2/0/9 Down
9 Te 2/0/10 Down
10 Te 2/0/11 Down
11 Te 2/0/12 Down
12 Te 2/0/13 Down
13 Te 2/0/14 Down
14 Te 2/0/15 Down
15 Te 2/0/16 Down
16 Te 2/0/17 Down
17 Te 2/0/18 Down
18 Te 2/0/19 Down
19 Te 2/0/20 Down
20 Te 2/0/21 Down
21 Te 2/0/22 Down
22 Te 2/0/23 Down
23 Te 2/0/24 Down Down (Admin)
Solved! Go to Solution.
08-09-2015 08:03 AM
Hi,
I don't know the Brocade switches, but a couple of things look odd here:
1) The Brocade switch has the command switchport trunk tag native-vlan on its port-channel interface, which means that its expecting IEEE 802.1Q tags on frames it receives, and adds tags to those it transmits.
I don't see the switchport trunk native vlan tag command configured on the Catalyst interface so unless you're using the global vlan dot1q tag native, the trunking on the two switches is not compatible and could be part of the problem.
Try adding no switchport trunk tag native-vlan to the Brocade or adding the switchport trunk native vlan tag command on the interface of the Catalyst switch.
2) You say you're using Link Aggregation Control Protocol (LACP), but the Catalyst switch is configured with channel-group 24 mode on. This is not using LACP, but rather it's a static LAG. This means it does not send LACPDU.
By contrast the Brocade is configured with channel-group 24 mode active type standard on its interfaces and so sends LACPDU and also expects to receive them. As it is not going to be receiving them I suspect the port-channel is not formed correctly.
You should change the command on the Cisco switch to channel-group 24 mode active.
Try the above and then using the show etherchannel summary command ensure the port-channel shows the S (Layer-2) and U (in use) flags, and that both interface appear in the ports column with the P (bundled) flag.
You can verify whether the switch is sending and receiving LACPDU with the show lacp counters command, and the show lacp neighbor should provide details of the Brocade switch.
Regards
08-09-2015 08:03 AM
Hi,
I don't know the Brocade switches, but a couple of things look odd here:
1) The Brocade switch has the command switchport trunk tag native-vlan on its port-channel interface, which means that its expecting IEEE 802.1Q tags on frames it receives, and adds tags to those it transmits.
I don't see the switchport trunk native vlan tag command configured on the Catalyst interface so unless you're using the global vlan dot1q tag native, the trunking on the two switches is not compatible and could be part of the problem.
Try adding no switchport trunk tag native-vlan to the Brocade or adding the switchport trunk native vlan tag command on the interface of the Catalyst switch.
2) You say you're using Link Aggregation Control Protocol (LACP), but the Catalyst switch is configured with channel-group 24 mode on. This is not using LACP, but rather it's a static LAG. This means it does not send LACPDU.
By contrast the Brocade is configured with channel-group 24 mode active type standard on its interfaces and so sends LACPDU and also expects to receive them. As it is not going to be receiving them I suspect the port-channel is not formed correctly.
You should change the command on the Cisco switch to channel-group 24 mode active.
Try the above and then using the show etherchannel summary command ensure the port-channel shows the S (Layer-2) and U (in use) flags, and that both interface appear in the ports column with the P (bundled) flag.
You can verify whether the switch is sending and receiving LACPDU with the show lacp counters command, and the show lacp neighbor should provide details of the Brocade switch.
Regards
08-09-2015 08:03 AM
Thank you Steve,
I had since corrected the LACP setting on the 2960x, but still no joy. I am seeing the Brocade port channel MAC address on the sh lacp nei though. Here are my results:
Group Port-channel Protocol Ports
------+-------------+-----------+----------------------------------------------
1 Po1(SU) - Gi1/0/47(P) Gi1/0/48(P)
2 Po2(SU) LACP Te1/0/1(P) Te1/0/2(P)
Channel group 2 neighbors
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Te1/0/1 SA 32768 01e0.5200.0001 23s 0x0 0x18 0x418 0x3D
Te1/0/2 SA 32768 01e0.5200.0001 8s 0x0 0x18 0x218 0x3D
2960x-48P#sh lacp count
LACPDUs Marker Marker Response LACPDUs
Port Sent Recv Sent Recv Sent Recv Pkts Err
---------------------------------------------------------------------
Channel group: 2
Te1/0/1 1040 964 0 0 0 0 0
Te1/0/2 1047 967 0 0 0 0 0
Brocade VDX:
sw0# sh lacp sys-id
Port-channel Po 24 - System ID: 0x8000,01-e0-52-00-00-01 - SID Master: N/A
sw0# sh lacp count
Traffic statistics
Port LACPDUs Marker Pckt err
Sent Recv Sent Recv Sent Recv
Aggregator Po 24
Te 1/0/24 2457 1440 0 0 0 0
I'm not sure what to do with the native VLAN tagging on that port channel.
08-09-2015 11:24 PM
Hi,
So the port-channel is now up and working correctly now so some progress at least.
As far as tagging of the native VLAN, assuming the Brocade switches also use VLAN 1 as their native VLAN, any issue here would only affect VLAN 1. You said you also had servers in VLAN 200 using 10.10.10.x/24. Are you able to verify whether you have any connectivity between them?
When I look back at you original post I can also see you have spanning-tree bpdufilter enable configured on the port-channel interface. Is there a specific reason for this? It's never a good idea to disable spanning tree if it can be avoided.
Also can you post the output of a show spanning-tree interface port-channel 24 from the Cisco switch? I don't believe this would be a problem as you said you were seeing MAC addresses being learned across the port-channel, but lets at least make sure there's nothing odd happening there.
Regards
08-10-2015 05:24 AM
Steve,
No connectivity to VLAN 200 either. We have many VLANs, all being routed by an older Avaya stack. I chose to start with 200 for simplicity's sake because it's not routed, so I figured that two hosts in the same subnet, same VLAN would have no trouble talking. No luck here either. Not even an entry in the MAC table for 200 after a ping.
Funny thing is that I've seen VLAN 200 in this table before. Previously I was running RSTP on the 2960x, and thinking that it could have been the problem, as the Avaya systems don't recognize it, so yesterday I changed the mode to MST. Before the change the 2960x thought that it was the root bridge for all VLANs. (and looking now it still does). But that wouldn't affect VLAN 200 right?
I have bpdu filter enabled on PO2 because in fabric mode, the only thing the Brocade switches do with a bpdu is flood it out all ports, so the recommendation from Brocade in the initial setup was to turn it off. Currently we have no other 10G fiber, and the VDX cluster is 10G fiber only, so I'm not worried about someone accidentally plugging in.
Output below. Nothing being blocked...
sh spann int po 2
Mst Instance Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
MST0 Desg FWD 1000 128.464 P2p
2960x-48P#sh spann int po 2 det
Port 464 (Port-channel2) of MST0 is designated forwarding
Port path cost 1000, Port priority 128, Port Identifier 128.464.
Designated root has priority 32768, address 2c3e.cf9c.5980
Designated bridge has priority 32768, address 2c3e.cf9c.5980
Designated port id is 128.464, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default, Internal
Bpdu filter is enabled
BPDU: sent 0, received 0
Thank you!
Brad
08-10-2015 07:12 AM
Hi Brad,
Looking around I came across a couple of Brocade design guides. There's the Brocade Data Center Solution-Design Guide: Connecting Brocade VCS Fabric with NOS 3.0 to Existing STP Networks. This document talks about passing STP BPDU straight through and so I presume this is the setup you have, rather than a NOS 4.0.X setup as shown in Data Center Solution-Design Guide: Connecting VCS Fabric to Spanning Tree Protocol (STP) Networks?
Does the Cisco switch see any MACs learned via the port-channel interface now? You said previously you were seeing them, but your comment "Not even an entry in the MAC table for 200 after a ping" seems to suggest nothing is being learned now.
What do you see if you run the command show mac address interface port-channel 24? Do you see any MAC addresses and are they those assigned to the servers connected to the VDX fabric?
Regards
08-10-2015 07:30 AM
Steve,
You are correct, we are currently using Brocade NOS 3.01c. Scratch the comment earlier on the STP root incorrectly being the 2960x. It is the Avaya stack.
I cleared the dynamic MACs from the 2960x, and immediately the 4 hosts repopulate on port channel 2. All on VLAN 1 though. Nothing on VLAN 200, despite trying to ping a host there. I've even had someone directly connect a laptop onto the 2960x and configured it with a static IP on the 200 VLAN to eliminate any possible upstream problems.
However, I am no longer seeing anything in the Brocade MAC table for it's side of the port channel.The LACPDUs are still incrementing on either side through.
Brad
08-10-2015 08:17 AM
Hi Brad,
From what you posted previously the LAG is bundling the physical interfaces and LACP seems to be working OK. What we have is more of a trunking / spanning tree issue whereby the LAG is carrying the VLANs correctly or spanning tree is blocking it for some reason.
Are you able to post a diagram of the topology so I can see what the connectivity is between the Cisco switch, the Brocade VDX and the Avaya routers? If you can do that can you also indicate where the servers A, B, C and D that you referred to previously are connected and the IP addresses assigned to them?
Where is the Avaya stack connected and do any of the servers on the Cisco or Brocade switches have connectivity to that?
Additionally could you attach the configurations from the Cisco and Brocade switches?
Regards
08-10-2015 12:54 PM
WOW! Finally getting somewhere! The problem was twofold: lack of port channel config sync between Brocade switches, and default VLAN tagging.
On the Brocade VDX cluster IP I had created the port channel 24, as well as configured the trunking of VLANs. Unlike the Cisco IOS where the ports need have identical configurations before they are added to the port channel, the Brocade requires there be no configuration on the switchports being aggregated, no access, no trunk. That I understood, but where the problem was, as I made the port channel config on the cluster IP, it made the port channel 24 across both VDX1 and VDX2, but the configuration of the switchport and trunking of all VLANs was only happening on the primary switch for port channel 24. When I ran a sh run int po 24 on VDX1 I saw the expected config and was confused as to why it wan't working. It wasn't until I logged on to VDX2 and ran the same that I found the port channel 24 (which was created from the master) had an empty config.
After entering the config again on VDX2, I find VLAN 200 coming to life, both ways! I added on a routed VLAN 132, which is being routed through out Avaya infrastructure, and it came up as well, immediately populating the port channel mac table.
VLAN 1 which is where most of our production nodes are (I inherited this structure) continues to be a show stopper. Servers A and B's MAC addresses come up on the 2960x's mac table for port channel 2, but servers C and D's MACs do not populate the VDXs MAC table in relation to it's port channel 24.
Actually there is only one MAC entry in that list for VLAN 1, and it is for an ASUSTEK manufacturer, which confuses me even further.
I believe that the icmp packets are making it through to the server connected to the VDX as the sh int command shows incrementing receive and transmit wit ha continuous ping, however I believe that the VDX is tagging the default VLAN as it leaves on port channel 24, destined to return the reply towards the 2960x.
Changing the production VLAN 1 isn't something easily done right now. How can I address the default VLAN issue?
Brad
08-10-2015 03:01 PM
And it's all working now. After entering the no switchport trunk tag native-vlan on port channel 24 on both Brocade VDX switches, all is now working.
It really wasn't what I'm used to. Thank you Steve for pointing me in the right direction, getting LACP working correctly.
This really was a Brocade config issue. I had posted originally to the Brocade forums, but after seeing a lot of questions with a 1000 views before a response, I thought I'd ask here. After all it was a switching issue, just really odd how some manufacturers systems work.
Brad
08-10-2015 10:35 PM
Hi Brad,
Glad to hear it's all working now.
Regards
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide