Problem with vPC on Nexus 5K and HP1920-24G

[vasilij]

Hello,

we've got a strange problem in our network environment with connecting one HP1920-24G (JG924A) through LACP to two Nexus C5548UP switches with configured vPC. To be precisive we are connecting our HP switch through two Fabric Extenders C2248TP managed by those N5K of course.
Let's name Nexuses as N5K-1 and N5K-2 and HP1920-24G as HP-1.
The thing is from time to time nexuses just flap the interfaces that are members of vPC and put those messages into logfile:

HP-1:

%Jan 31 05:22:34:666 2018 sw-oc-4 LAGG/5/LAGG_ACTIVE: Member port GigabitEthernet1/0/36 of aggregation group BAGG7 becomes ACTIVE.
%Jan 31 05:22:34:653 2018 sw-oc-4 LAGG/5/LAGG_INACTIVE_PARTNER: Member port GigabitEthernet1/0/36 of aggregation group BAGG7 becomes INACTIVE because the port's partner is improper for being attached.

N5K-1:

2018 Jan 31 05:23:05.672 CET: %STP-6-PORT_RANGE_STATE: new_state=forwarding interface=port-channel202 vlan=900-901,2076-2077
2018 Jan 31 05:23:05.670 CET: %STP-6-PORT_RANGE_STATE: new_state=blocking interface=port-channel202 vlan=900-901,2076-2077
2018 Jan 31 05:23:05.668 CET: %STP-6-PORT_RANGE_ROLE: new_role=designated interface=port-channel202 vlan=900-901,2076-2077
2018 Jan 31 05:23:04.538 CET: %ETHPORT-5-IF_UP: Interface port-channel202 is up in mode trunk
2018 Jan 31 05:23:04.511 CET: %STP-6-PORT_RANGE_ADDED: Interface port-channel202 added to vlan=900-901,2076-2077 with cost 1, priority 128, link-type P2p
2018 Jan 31 05:23:04.485 CET: %ETHPORT-5-IF_UP: Interface Ethernet192/1/2 is up in mode trunk
2018 Jan 31 05:23:03.331 CET: %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel202: first operational port changed from none to Ethernet192/1/2
2018 Jan 31 05:23:02.057 CET: %ETH_PORT_CHANNEL-5-PORT_UP: port-channel202: Ethernet192/1/2 is up
2018 Jan 31 05:23:01.962 CET: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 202 configuration (interface port-channel202) is consistent
2018 Jan 31 05:23:01.910 CET: %VPC-6-PEER_VPC_CFGD_VLANS_CHANGED: Peer vPC 202 configured vlans changed
2018 Jan 31 05:23:00.609 CET: %VPC-5-VPC_UP: vPC 202 (interface port-channel202) is up
2018 Jan 31 05:22:59.467 CET: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 202 configuration (interface port-channel202) is consistent
2018 Jan 31 05:22:59.432 CET: %VPC-6-VPC_ES_EGRESS_DF_STATE_CHANGED: Egress designated forwarder state changed from BLOCK to PARTIAL for vPC 202.
2018 Jan 31 05:22:59.405 CET: %VPC-6-PEER_VPC_CFGD_VLANS_CHANGED: Peer vPC 202 configured vlans changed
2018 Jan 31 05:22:59.350 CET: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 202 configuration (interface port-channel202) is consistent
2018 Jan 31 05:22:55.400 CET: %VPC-6-PEER_VPC_CFGD_VLANS_CHANGED: Peer vPC 202 configured vlans changed
2018 Jan 31 05:22:50.186 CET: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 202 configuration (interface port-channel202) is consistent
2018 Jan 31 05:22:43.695 CET: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel202, operational Transmit Flow Control state changed to on
2018 Jan 31 05:22:43.693 CET: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel202, operational Receive Flow Control state changed to off
2018 Jan 31 05:22:43.692 CET: %ETHPORT-5-IF_DUPLEX: Interface port-channel202, operational duplex mode changed to Full
2018 Jan 31 05:22:43.690 CET: %ETHPORT-5-SPEED: Interface port-channel202, operational speed changed to 1 Gbps
2018 Jan 31 05:22:43.688 CET: %ETHPORT-3-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel202 is down (No operational members)
2018 Jan 31 05:22:41.164 CET: %ETHPORT-3-IF_DOWN_INITIALIZING: Interface Ethernet192/1/2 is down (Initializing)
2018 Jan 31 05:22:41.162 CET: %STP-6-PORT_RANGE_STATE: new_state=disabled interface=port-channel202 vlan=900-901,2076-2077
2018 Jan 31 05:22:38.624 CET: %STP-6-PORT_RANGE_DELETED: Interface port-channel202 removed from vlan=900-901,2076-2077
2018 Jan 31 05:22:37.433 CET: %ETHPORT-3-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel202 is down (No operational members)
2018 Jan 31 05:22:37.297 CET: %VPC-6-VPC_ES_EGRESS_DF_STATE_CHANGED: Egress designated forwarder state changed from PARTIAL to BLOCK for vPC 202.
2018 Jan 31 05:22:34.957 CET: %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel202: first operational port changed from Ethernet192/1/2 to none
2018 Jan 31 05:22:33.605 CET: %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel202: port-channel202 is down
2018 Jan 31 05:22:33.603 CET: %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel202: Ethernet192/1/2 is down

N5K-2:

2018 Jan 31 05:22:45.832 CET: %STP-6-PORT_RANGE_STATE: new_state=forwarding interface=port-channel202 vlan=901,2076-2077
2018 Jan 31 05:22:45.829 CET: %STP-6-PORT_RANGE_STATE: new_state=forwarding interface=port-channel202 vlan=900
2018 Jan 31 05:22:45.826 CET: %STP-6-PORT_RANGE_ROLE: new_role=designated interface=port-channel202 vlan=900-901,2076-2077
2018 Jan 31 05:22:45.822 CET: %ETHPORT-5-IF_UP: Interface port-channel202 is up in mode trunk
2018 Jan 31 05:22:45.819 CET: %STP-6-PORT_RANGE_ADDED: Interface port-channel202 added to vlan=900-901,2076-2077 with cost 1, priority 128, link-type P2p
2018 Jan 31 05:22:45.704 CET: %ETHPORT-5-IF_UP: Interface Ethernet192/1/2 is up in mode trunk
2018 Jan 31 05:22:45.701 CET: %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel202: first operational port changed from none to Ethernet192/1/2
2018 Jan 31 05:22:45.697 CET: %ETH_PORT_CHANNEL-5-PORT_UP: port-channel202: Ethernet192/1/2 is up
2018 Jan 31 05:22:45.478 CET: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 202 configuration (interface port-channel202) is consistent
2018 Jan 31 05:22:45.368 CET: %VPC-6-PEER_VPC_CFGD_VLANS_CHANGED: Peer vPC 202 configured vlans changed
2018 Jan 31 05:22:44.291 CET: %VPC-5-VPC_UP: vPC 202 (interface port-channel202) is up
2018 Jan 31 05:22:44.247 CET: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 202 configuration (interface port-channel202) is consistent
2018 Jan 31 05:22:44.213 CET: %VPC-6-VPC_ES_EGRESS_DF_STATE_CHANGED: Egress designated forwarder state changed from BLOCK to PARTIAL for vPC 202.
2018 Jan 31 05:22:44.163 CET: %VPC-6-PEER_VPC_CFGD_VLANS_CHANGED: Peer vPC 202 configured vlans changed
2018 Jan 31 05:22:44.066 CET: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 202 configuration (interface port-channel202) is consistent
2018 Jan 31 05:22:40.584 CET: %VPC-6-PEER_VPC_CFGD_VLANS_CHANGED: Peer vPC 202 configured vlans changed
2018 Jan 31 05:22:35.754 CET: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 202 configuration (interface port-channel202) is consistent
2018 Jan 31 05:22:32.900 CET: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel202, operational Transmit Flow Control state changed to on
2018 Jan 31 05:22:32.895 CET: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel202, operational Receive Flow Control state changed to off
2018 Jan 31 05:22:32.891 CET: %ETHPORT-5-IF_DUPLEX: Interface port-channel202, operational duplex mode changed to Full
2018 Jan 31 05:22:32.837 CET: %ETHPORT-5-SPEED: Interface port-channel202, operational speed changed to 1 Gbps
2018 Jan 31 05:22:32.833 CET: %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel202 is down (No operational members)
2018 Jan 31 05:22:31.714 CET: %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet192/1/2 is down (Initializing)
2018 Jan 31 05:22:31.334 CET: %STP-6-PORT_RANGE_STATE: new_state=disabled interface=port-channel202 vlan=900-901,2076-2077
2018 Jan 31 05:22:31.204 CET: %STP-6-PORT_RANGE_DELETED: Interface port-channel202 removed from vlan=900-901,2076-2077
2018 Jan 31 05:22:31.200 CET: %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel202 is down (No operational members)
2018 Jan 31 05:22:31.197 CET: %VPC-6-VPC_ES_EGRESS_DF_STATE_CHANGED: Egress designated forwarder state changed from PARTIAL to BLOCK for vPC 202.
2018 Jan 31 05:22:30.103 CET: %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel202: first operational port changed from Ethernet192/1/2 to none
2018 Jan 31 05:22:30.099 CET: %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel202: port-channel202 is down
2018 Jan 31 05:22:30.095 CET: %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel202: Ethernet192/1/2 is down

On the HP Switch sometimes we've got information about flapping but sometimes there is nothing. As you can see above HP-1 reported that only one LAG member is down but two of them flapped.

This is the current configuration on the devices:

HP-1:

version 5.20.99, Release 1112

interface Bridge-Aggregation7
 description "n5k-1-2"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 900 to 901 2076 to 2077
 link-aggregation mode dynamic
 stp disable
 stp edged-port enable

interface GigabitEthernet1/0/35
 description "br7"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 900 to 901 2076 to 2077
 speed 1000
 bpdu-drop any
 broadcast-suppression pps 1000
 multicast-suppression pps 1000
 unicast-suppression pps 1000
 stp disable
 stp edged-port enable
 lacp period short
 port link-aggregation group 7

interface GigabitEthernet1/0/36
 description "br7"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 900 to 901 2076 to 2077
 speed 1000
 bpdu-drop any
 broadcast-suppression pps 1000
 multicast-suppression pps 1000
 unicast-suppression pps 1000
 stp disable
 stp edged-port enable
 lacp period short
 port link-aggregation group 7

N5K-1:

version 7.0(7)N1(1)

interface port-channel202
  description sw-oc-4
  switchport mode trunk
  switchport trunk allowed vlan 900-901,2076-2077
  spanning-tree bpdufilter enable
  storm-control broadcast level 1.00
  storm-control multicast level 1.00
  vpc 202

interface Ethernet192/1/2
  description part po202
  switchport mode trunk
  switchport trunk allowed vlan 900-901,2076-2077
  storm-control broadcast level 1.00
  storm-control multicast level 1.00
  channel-group 202 mode active
  no shutdown

N5K-2:

version 7.0(7)N1(1)

interface port-channel202
  description sw-oc-4
  switchport mode trunk
  switchport trunk allowed vlan 900-901,2076-2077
  spanning-tree bpdufilter enable
  storm-control broadcast level 1.00
  storm-control multicast level 1.00
  vpc 202

interface Ethernet192/1/2
  description part po202
  lacp rate fast
  switchport mode trunk
  switchport trunk allowed vlan 900-901,2076-2077
  storm-control broadcast level 1.00
  storm-control multicast level 1.00
  channel-group 202 mode active
  no shutdown

Interfaces connections:

Gi 1/0/35 -> Eth192/1/2 N5K-2

Gi 1/0/36 -> Eth192/1/2 N5K-1

We do not know where is the problem, because we've got many switches connected to the N5Ks like that and everything is OK. Those switches are Huawei S1720, HP 1910-24G and HP 1920-48G.
We've already tried to setup manually speed 1000 on both sides and we've upgraded the firmware on HPs. The time of flapping is totally random and we've run out of ideas what is wrong with this setup :) is there any other way to debug this more?

Regards,
vasilij

[vasilij · Accepted solution]

I don't like when I read topics without solutions :)

We've managed to resolve this problem with HPE support. All HP1920s and 1910s doesn't support option:

lacp period short

and this N5K option had to be deleted too:

lacp rate fast

Below configuration on N5Ks and HP works perfectly without issues:

HP-1:

version 5.20.99, Release 1112

interface Bridge-Aggregation7
 description "n5k-1-2"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 900 to 901 2076 to 2077
 link-aggregation mode dynamic
 stp disable
 stp edged-port enable

interface GigabitEthernet1/0/35
 description "br7"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 900 to 901 2076 to 2077
 speed 1000
 bpdu-drop any
 broadcast-suppression pps 1000
 multicast-suppression pps 1000
 unicast-suppression pps 1000
 stp disable
 stp edged-port enable
 port link-aggregation group 7

interface GigabitEthernet1/0/36
 description "br7"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 900 to 901 2076 to 2077
 speed 1000
 bpdu-drop any
 broadcast-suppression pps 1000
 multicast-suppression pps 1000
 unicast-suppression pps 1000
 stp disable
 stp edged-port enable
 port link-aggregation group 7

N5K-1:

version 7.0(7)N1(1)

interface port-channel202
  description sw-oc-4
  switchport mode trunk
  switchport trunk allowed vlan 900-901,2076-2077
  spanning-tree bpdufilter enable
  storm-control broadcast level 1.00
  storm-control multicast level 1.00
  vpc 202

interface Ethernet192/1/2
  description part po202
  switchport mode trunk
  switchport trunk allowed vlan 900-901,2076-2077
  storm-control broadcast level 1.00
  storm-control multicast level 1.00
  channel-group 202 mode active
  no shutdown

N5K-2:

version 7.0(7)N1(1)

interface port-channel202
  description sw-oc-4
  switchport mode trunk
  switchport trunk allowed vlan 900-901,2076-2077
  spanning-tree bpdufilter enable
  storm-control broadcast level 1.00
  storm-control multicast level 1.00
  vpc 202

interface Ethernet192/1/2
  description part po202
  switchport mode trunk
  switchport trunk allowed vlan 900-901,2076-2077
  storm-control broadcast level 1.00
  storm-control multicast level 1.00
  channel-group 202 mode active
  no shutdown

Hope this helps someone ;)

BR,

vasilij