Interface reset causes Port-channel %ETHPORT-3-IF_ERROR_VLANS_REMOVED

[long jun lin]

Hi all,

I am having the issues explained below , I did not find any bug related to the below error messages , your help is appreciated.

Here is an example of what each switch logs when a server drops offline. Sample logs taken  on Feb 1. This particular one was having problems all weekend. 

N9K-2# show logging last 50
2020 Feb 1 11:45:14 N9K-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel15: first operational port changed from Ethernet1/5 to none
2020 Feb 1 11:45:14 N9K-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel15 is down (No operational members)
2020 Feb 1 11:45:14 N9K-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel15,bandwidth changed to 100000 Kbit
2020 Feb 1 11:45:14 N9K-2 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/5 is down (Initializing)
2020 Feb 1 11:45:14 N9K-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel15 is down (No operational members)
2020 Feb 1 11:45:14 N9K-2 last message repeated 1 time
2020 Feb 1 11:45:14 N9K-2 %ETHPORT-5-SPEED: Interface port-channel15, operational speed changed to 10 Gbps
2020 Feb 1 11:45:14 N9K-2 %ETHPORT-5-IF_DUPLEX: Interface port-channel15, operational duplex mode changed to Full
2020 Feb 1 11:45:14 N9K-2 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel15, operational Receive Flow Control state changed to off
2020 Feb 1 11:45:14 N9K-2 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel15, operational Transmit Flow Control state changed to off
2020 Feb 1 11:45:18 N9K-2 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel15: Ethernet1/5 is up
2020 Feb 1 11:45:18 N9K-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel15: first operational port changed from none to Ethernet1/5
2020 Feb 1 11:45:18 N9K-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel15,bandwidth changed to 10000000 Kbit
2020 Feb 1 11:45:18 N9K-2 %ETHPORT-5-IF_UP: Interface Ethernet1/5 is up in mode access
2020 Feb 1 11:45:18 N9K-2 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 110 on Interface port-channel15 are removed from suspended state.
2020 Feb 1 11:45:18 N9K-2 %ETHPORT-5-IF_UP: Interface port-channel15 is up in mode access
2020 Feb 1 12:29:18 N9K-2 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel15: Ethernet1/5 is down
2020 Feb 1 12:29:18 N9K-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel15: first operational port changed from Ethernet1/5 to none
2020 Feb 1 12:29:18 N9K-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel15 is down (No operational members)
2020 Feb 1 12:29:18 N9K-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel15,bandwidth changed to 100000 Kbit
2020 Feb 1 12:29:18 N9K-2 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/5 is down (Initializing)
2020 Feb 1 12:29:18 N9K-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel15 is down (No operational members)
2020 Feb 1 12:29:18 N9K-2 last message repeated 1 time
2020 Feb 1 12:29:18 N9K-2 %ETHPORT-5-SPEED: Interface port-channel15, operational speed changed to 10 Gbps
2020 Feb 1 12:29:18 N9K-2 %ETHPORT-5-IF_DUPLEX: Interface port-channel15, operational duplex mode changed to Full
2020 Feb 1 12:29:18 N9K-2 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel15, operational Receive Flow Control state changed to off
2020 Feb 1 12:29:18 N9K-2 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel15, operational Transmit Flow Control state changed to off
2020 Feb 1 12:29:21 N9K-2 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel15: Ethernet1/5 is up
2020 Feb 1 12:29:21 N9K-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel15: first operational port changed from none to Ethernet1/5
2020 Feb 1 12:29:21 N9K-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel15,bandwidth changed to 10000000 Kbit
2020 Feb 1 12:29:21 N9K-2 %ETHPORT-5-IF_UP: Interface Ethernet1/5 is up in mode access
2020 Feb 1 12:29:21 N9K-2 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 110 on Interface port-channel15 are removed from suspended state.
2020 Feb 1 12:29:21 N9K-2 %ETHPORT-5-IF_UP: Interface port-channel15 is up in mode access
2020 Feb 1 12:31:23 N9K-2 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel15: Ethernet1/5 is down
2020 Feb 1 12:31:23 N9K-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel15: first operational port changed from Ethernet1/5 to none
2020 Feb 1 12:31:23 N9K-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel15 is down (No operational members)
2020 Feb 1 12:31:23 N9K-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel15,bandwidth changed to 100000 Kbit
2020 Feb 1 12:31:23 N9K-2 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/5 is down (Initializing)
2020 Feb 1 12:31:23 N9K-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel15 is down (No operational members)
2020 Feb 1 12:31:23 N9K-2 last message repeated 1 time
2020 Feb 1 12:31:23 N9K-2 %ETHPORT-5-SPEED: Interface port-channel15, operational speed changed to 10 Gbps
2020 Feb 1 12:31:23 N9K-2 %ETHPORT-5-IF_DUPLEX: Interface port-channel15, operational duplex mode changed to Full
2020 Feb 1 12:31:23 N9K-2 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel15, operational Receive Flow Control state changed to off
2020 Feb 1 12:31:23 N9K-2 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel15, operational Transmit Flow Control state changed to off
2020 Feb 1 12:31:26 N9K-2 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel15: Ethernet1/5 is up
2020 Feb 1 12:31:26 N9K-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel15: first operational port changed from none to Ethernet1/5
2020 Feb 1 12:31:26 N9K-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel15,bandwidth changed to 10000000 Kbit
2020 Feb 1 12:31:26 N9K-2 %ETHPORT-5-IF_UP: Interface Ethernet1/5 is up in mode access
2020 Feb 1 12:31:26 N9K-2 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 110 on Interface port-channel15 are removed from suspended state.
2020 Feb 1 12:31:26 N9K-2 %ETHPORT-5-IF_UP: Interface port-channel15 is up in mode access

[balaji.bandi]

Interface Ethernet1/5 is down  - can you check physical connection first instance here.

after physcally ok - please post below information -

show run inter port-chan 5

show interface eth 1/5

show etherchannel 5 summanry

what is the connected to Eth 1/5 ?

[long jun lin]

what is the connected to Eth 1/5 ?

connect  fortigate firewall

N9K-2# show run interface port-channel15

!Command: show running-config interface port-channel15
!Running configuration last done at: Tue Jan 21 14:39:12 2020
!Time: Mon Feb 3 15:27:49 2020

version 9.2(3) Bios:version 05.33

interface port-channel15
description Connect to FW1
switchport
switchport access vlan 110
vpc 15

N9K-2# show interface ethernet 1/5
Ethernet1/5 is up
admin state is up, Dedicated Interface
Belongs to Po15
Hardware: 100/1000/10000/25000 Ethernet, address: a0b4.39b5.d5ac (bia a0b4.39b
5.d5ac)
Description: Connect to FW1
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 01:45:16
Last clearing of "show interface" counters never
143 interface resets
--More-- Load-Interval #1: 30 seconds
30 seconds input rate 64 bits/sec, 0 packets/sec
30 seconds output rate 4848 bits/sec, 3 packets/sec
input rate 64 bps, 0 pps; output rate 4.85 Kbps, 3 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 32 bits/sec, 0 packets/sec
300 seconds output rate 4536 bits/sec, 1 packets/sec
input rate 32 bps, 0 pps; output rate 4.54 Kbps, 1 pps
RX
781860 unicast packets 38224 multicast packets 106 broadcast packets
820190 input packets 472674745 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
962991 unicast packets 3546676 multicast packets 2206453 broadcast packets
6716120 output packets 1071446272 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
--More-- 0 Tx pause

N9K-2# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SU) Eth LACP Eth1/47(P) Eth1/48(P)
11 Po11(SU) Eth LACP Eth1/1(P)
12 Po12(SU) Eth LACP Eth1/2(P)
13 Po13(SU) Eth LACP Eth1/3(P)
14 Po14(SU) Eth LACP Eth1/4(P)
15 Po15(SU) Eth LACP Eth1/5(P)

N9K-1# show run interface port-channel15

!Command: show running-config interface port-channel15
!Running configuration last done at: Wed Jan 22 02:28:37 2020
!Time: Mon Feb 3 15:27:46 2020

version 9.2(3) Bios:version 05.33

interface port-channel15
description Connect to FW1
switchport
switchport access vlan 110
vpc 15

N9K-1# show interface ethernet 1/5
Ethernet1/5 is up
admin state is up, Dedicated Interface
Belongs to Po15
Hardware: 100/1000/10000/25000 Ethernet, address: a0b4.39b5.d30c (bia a0b4.39b
5.d30c)
Description: Connect to FW1
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 1week(s) 6day(s)
Last clearing of "show interface" counters never
1 interface resets
--More-- Load-Interval #1: 30 seconds
30 seconds input rate 64 bits/sec, 0 packets/sec
30 seconds output rate 4000 bits/sec, 2 packets/sec
input rate 64 bps, 0 pps; output rate 4.00 Kbps, 2 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 32 bits/sec, 0 packets/sec
300 seconds output rate 3640 bits/sec, 1 packets/sec
input rate 32 bps, 0 pps; output rate 3.64 Kbps, 1 pps
RX
700022 unicast packets 37620 multicast packets 10644 broadcast packets
748286 input packets 419612844 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
1764453 unicast packets 2730671 multicast packets 4814136 broadcast packet
s
9309260 output packets 2105615511 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
--More-- 0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

N9K-1# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SU) Eth LACP Eth1/47(P) Eth1/48(P)
11 Po11(SU) Eth LACP Eth1/1(P)
12 Po12(SU) Eth LACP Eth1/2(P)
13 Po13(SU) Eth LACP Eth1/3(P)
14 Po14(SU) Eth LACP Eth1/4(P)
15 Po15(SU) Eth LACP Eth1/5(P)

[pinia0001]

I have from yesterday the same problem. Would you share your solution? Thanks

2022 Dec 3 17:56:03 Nexus-9372 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel45: Ethernet1/45 is down
2022 Dec 3 17:56:03 Nexus-9372 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel45: first operational port changed from Etherne
t1/45 to Ethernet1/47
2022 Dec 3 17:56:03 Nexus-9372 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/45 is down (Initializing)
2022 Dec 3 17:56:06 Nexus-9372 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel45: Ethernet1/45 is up
2022 Dec 3 17:56:06 Nexus-9372 %ETHPORT-5-IF_UP: Interface Ethernet1/45 is up in mode trunk

[balaji.bandi]

as per the information you posted port-channel 15, it has interface eth 1/5 both the side

the logs show port-channel 45 and interface 1/45 and 1/47 

as per the output you do not have port-channel 45 or you may be  (since we do not see the output)

1/47  belong to port-channel1 so check  the config and right interface in the right group.