hi out there
We encountered a problem where a portchannel to a IBM host suddenly died - the link is running as 2 link Portchannel ( ethernet 101/1/11 on nx1 and ethernet 102/1/11 on nx2)
These links are build as portchannel spanning the two nexus trough a vPC
nx #1
interface port-channel111
description SAP
switchport
switchport access vlan 2
vpc 111
vPC status
------------------------------------------------------
id Port Status Consistency Active VLANs
----- ------------ ------ ----------- ----------------
111 Po111 up success 2
interface Ethernet101/1/11
description SAP
switchport
switchport access vlan 2
channel-group 111 mode active
no shutdown
sh int po 111
port-channel111 is up
admin state is up
vPC Status: Up, vPC number: 111
Hardware: Port-Channel, address: 0041.d228.374c (bia 0041.d228.374c)
Description: SAP
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 10 Gb/s
Input flow-control is off, output flow-control is on
Auto-mdix is turned off
Switchport monitor is off
EtherType is 0x8100
Members in this channel: Eth101/1/11
Last clearing of "show interface" counters never
6 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 2216 bits/sec, 2 packets/sec
30 seconds output rate 2091560 bits/sec, 742 packets/sec
input rate 2.22 Kbps, 2 pps; output rate 2.09 Mbps, 742 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 2120 bits/sec, 2 packets/sec
300 seconds output rate 3123960 bits/sec, 793 packets/sec
input rate 2.12 Kbps, 2 pps; output rate 3.12 Mbps, 793 pps
RX
4282567481 unicast packets 690254 multicast packets 128062 broadcast packe
ts
4283385797 input packets 2388294822502 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC/FCS 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
9456756464 unicast packets 76216532 multicast packets 6493734 broadcast pa
ckets
9539466730 output packets 4672927026231 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
similar on the other nexus
here is the logging:
2018 Sep 2 17:10:14 DCDISTSW01 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel111: Ethernet101/1/11 is down
2018 Sep 2 17:10:14 DCDISTSW01 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel111: first operational port changed from Ethernet101/1/11 to none
2018 Sep 2 17:10:14 DCDISTSW01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel111 is down (No operational members)
2018 Sep 2 17:10:15 DCDISTSW01 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel111,bandwidth changed to 100000 Kbit
2018 Sep 2 17:10:15 DCDISTSW01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet101/1/11 is down (Initializing)
2018 Sep 2 17:10:15 DCDISTSW01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel111 is down (No operational members)
2018 Sep 2 17:10:15 DCDISTSW01 %ETHPORT-5-SPEED: Interface port-channel111, operational speed changed to 10 Gbps
2018 Sep 2 17:10:15 DCDISTSW01 %ETHPORT-5-IF_DUPLEX: Interface port-channel111, operational duplex mode changed to Full
2018 Sep 2 17:10:15 DCDISTSW01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel111, operational Receive Flow Control state changed to off
2018 Sep 2 17:10:15 DCDISTSW01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel111, operational Transmit Flow Control state changed to on
2018 Sep 2 17:10:25 DCDISTSW01 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet101/1/11: Ethernet101/1/11 is suspended by protocol, no LACP PDUs received
2018 Sep 2 17:31:11 DCDISTSW01 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel111: Ethernet101/1/11 is up
2018 Sep 2 17:31:11 DCDISTSW01 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel111: first operational port changed from none to Ethernet101/1/11
2018 Sep 2 17:31:11 DCDISTSW01 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel111,bandwidth changed to 10000000 Kbit
2018 Sep 2 17:31:11 DCDISTSW01 %ETHPORT-5-IF_UP: Interface Ethernet101/1/11 is up in mode access
2018 Sep 2 17:31:11 DCDISTSW01 %ETHPORT-5-IF_UP: Interface port-channel111 is up in mode access
2018 Sep 2 17:47:23 DCDISTSW01 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel111: Ethernet101/1/11 is down
2018 Sep 2 17:47:23 DCDISTSW01 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel111: first operational port changed from Ethernet101/1/11 to none
2018 Sep 2 17:47:23 DCDISTSW01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel111 is down (No operational members)
2018 Sep 2 17:47:23 DCDISTSW01 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel111,bandwidth changed to 100000 Kbit
2018 Sep 2 17:47:23 DCDISTSW01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet101/1/11 is down (Initializing)
2018 Sep 2 17:47:23 DCDISTSW01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel111 is down (No operational members)
2018 Sep 2 17:47:24 DCDISTSW01 %ETHPORT-5-SPEED: Interface port-channel111, operational speed changed to 10 Gbps
2018 Sep 2 17:47:24 DCDISTSW01 %ETHPORT-5-IF_DUPLEX: Interface port-channel111, operational duplex mode changed to Full
2018 Sep 2 17:47:24 DCDISTSW01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel111, operational Receive Flow Control state changed to off
2018 Sep 2 17:47:24 DCDISTSW01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel111, operational Transmit Flow Control state changed to on
2018 Sep 2 17:47:34 DCDISTSW01 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet101/1/11: Ethernet101/1/11 is suspended by protocol, no LACP PDUs received
2018 Sep 2 18:44:15 DCDISTSW01 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet101/1/11 is down(Config change)
2018 Sep 2 18:44:16 DCDISTSW01 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet101/1/11 is down (Administratively down)
2018 Sep 2 18:44:24 DCDISTSW01 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet101/1/11 is admin up .
2018 Sep 2 18:44:24 DCDISTSW01 %ETHPORT-5-SPEED: Interface Ethernet101/1/11, operational speed changed to 10 Gbps
2018 Sep 2 18:44:24 DCDISTSW01 %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/11, operational duplex mode changed to Full
2018 Sep 2 18:44:24 DCDISTSW01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/11, operational Receive Flow Control state changed to off
2018 Sep 2 18:44:24 DCDISTSW01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/11, operational Transmit Flow Control state changed to on
2018 Sep 2 18:44:24 DCDISTSW01 %ETHPORT-5-SPEED: Interface port-channel111, operational speed changed to 10 Gbps
2018 Sep 2 18:44:24 DCDISTSW01 %ETHPORT-5-IF_DUPLEX: Interface port-channel111, operational duplex mode changed to Full
2018 Sep 2 18:44:24 DCDISTSW01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel111, operational Receive Flow Control state changed to off
2018 Sep 2 18:44:24 DCDISTSW01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel111, operational Transmit Flow Control state changed to on
2018 Sep 2 18:44:34 DCDISTSW01 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet101/1/11: Ethernet101/1/11 is suspended by protocol, no LACP PDUs received
2018 Sep 2 18:46:09 DCDISTSW01 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel111: Ethernet101/1/11 is up
2018 Sep 2 18:46:09 DCDISTSW01 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel111: first operational port changed from none to Ethernet101/1/11
2018 Sep 2 18:46:09 DCDISTSW01 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel111,bandwidth changed to 10000000 Kbit
2018 Sep 2 18:46:09 DCDISTSW01 %ETHPORT-5-IF_UP: Interface Ethernet101/1/11 is up in mode access
2018 Sep 2 18:46:09 DCDISTSW01 %ETHPORT-5-IF_UP: Interface port-channel111 is up in mode access
the main problem is that the link went down - and the traffic didn't continue on the other interface.
as you can see we had to disable the interface and bring it up again to get connectivity up again
can some tell me if there are some known issues with the fairly simple setup we have here?
br ti
