Hello guys! Hows everybody doing?!
TL;DR: 'l2protocol-tunnel stp' command was preventing a LACP port-channel to come back up. And I don't know why!
Today I faced a problem and apparently found the solution, but I can't understand the fix.
We are connecting a nexus 9000 pair with vPC to a 3750 stack. This 3750 stack is the edge of a QinQ tunnel. See the topology below:
And here are the config snippets!
N9K-LEG-01
interface port-channel17
description vPC SW_WAN_3750
switchport mode trunk
switchport trunk allowed vlan 320-321
spanning-tree port type edge trunk
speed 1000
mtu 9216
vpc 17
!
interface Ethernet1/18
description SW_WAN_3750
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 320-321
spanning-tree port type edge trunk
speed 1000
mtu 9216
channel-group 17 mode active
N9K-LEG-02
interface port-channel17
description vPC SW_WAN_3750
switchport mode trunk
switchport trunk allowed vlan 320-321
spanning-tree port type edge trunk
speed 1000
mtu 9216
vpc 17
!
interface Ethernet1/18
description SW_WAN_3750
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 320-321
spanning-tree port type edge trunk
speed 1000
mtu 9216
channel-group 17 mode active
SW_WAN
interface Port-channel16
description NEXUS 9000
switchport access vlan 216
switchport mode dot1q-tunnel
load-interval 30
l2protocol-tunnel stp
spanning-tree portfast
end
!
interface GigabitEthernet1/0/16
switchport access vlan 216
switchport mode dot1q-tunnel
l2protocol-tunnel stp
no cdp enable
channel-group 16 mode active
spanning-tree portfast
end
!
interface GigabitEthernet2/0/16
switchport access vlan 216
switchport mode dot1q-tunnel
l2protocol-tunnel stp
no cdp enable
channel-group 16 mode active
spanning-tree portfast
end
This is what happened: As part of a test, I manually disconnected N9K-LEG-01 interface's eth1/18. Everything still worked just fine as expected. I connected it back on and to my surprised the port did not come back. On the Nexus it complained that it didn't received any LACP PDUs and suspended the port. Look:
N9K-LEG-01# sh int eth1/18
Ethernet1/18 is down (suspended(no LACP PDUs))
admin state is up, Dedicated Interface
Belongs to Po17
Hardware: 1000/10000 Ethernet, address: 5c83.8fee.6547 (bia 5c83.8fee.6547)
Description: SW_WAN_3750
MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
auto-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 00:12:56
Last clearing of "show interface" counters 00:17:24
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 32 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 32 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 35 multicast packets 0 broadcast packets
35 output packets 4480 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
On the 3750 I got this error message:
Jul 29 16:44:45.588 GMT-3: %EC-5-CANNOT_BUNDLE2: Gi2/0/16 is not compatible with Gi1/0/16 and will be suspended (L2 protocol tunneling configuration doesn't match etherchannel)
I tried everything on the playbook to make this interface come back online: shut/no shut, re-configuring the entire interface, removing and re-adding the interface to the channel. Nothing worked.
But then I thought to myself: Why do I need "l2protocol-tunnel stp" on the 3750? There's no switch on the other side of this tunnel. So I removed it from port-channel16 and voilá! Like magic, the port came back up.
I even ran my test one more time. Removed eth1/18 on N9K-LEG-01 again and put it back on. Still worked like a charm. Tried removing eth1/18 on N9K-LEG-02 and plugging it back on. Still working.
Soo... I fixed the problem, but I don't know why! Any ideas? One hypotheses is that for some reason "l2protocol-tunnel stp" was receiving the LACPs PDUs and injecting it on the tunnel instead of getting it to the 3750 CPU. Maybe a bug?
Thanks!
