06-09-2021 11:36 AM - edited 06-09-2021 01:09 PM
We recently moved from Nexus6001 to NexusC93180YC-EX and noticed that port-channel members would sometimes shutdown and then come back up.
As an example a port-channel with 4 members might see all 4 member shutdown or just one or two or 3 of them. It is completely random, no specific time of day.
Here is log showing port-channel members go down and then come back up:
2021 Jun 2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/23 is down 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 30000000 Kbit 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/23 is down (Initializing) 2021 Jun 2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/21 is down 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 20000000 Kbit 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/21 is down (Initializing) 2021 Jun 2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from Ethernet1/22 to Ethernet1/24 2021 Jun 2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/22 is down 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 10000000 Kbit 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/22 is down (Initializing) 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel19 is down (No operational members) 2021 Jun 2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from Ethernet1/24 to none 2021 Jun 2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/24 is down 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 100000 Kbit 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/24 is down (Initializing) 2021 Jun 2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel19 is down (No operational members) 2021 Jun 2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/24 is up 2021 Jun 2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from none to Ethernet1/24 2021 Jun 2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/22 is up 2021 Jun 2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/21 is up 2021 Jun 2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/23 is up 2021 Jun 2 11:25:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 10000000 Kbit 2021 Jun 2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/24 is up in mode trunk 2021 Jun 2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface port-channel19 is up in mode trunk 2021 Jun 2 11:25:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 20000000 Kbit 2021 Jun 2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk 2021 Jun 2 11:25:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 30000000 Kbit 2021 Jun 2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/21 is up in mode trunk 2021 Jun 2 11:25:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 40000000 Kbit 2021 Jun 2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/23 is up in mode trunk 2021 Jun 2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/21 is down 2021 Jun 2 11:30:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 30000000 Kbit 2021 Jun 2 11:30:18 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/21 is down (Initializing) 2021 Jun 2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from Ethernet1/24 to Ethernet1/22 2021 Jun 2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/24 is down 2021 Jun 2 11:30:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 20000000 Kbit 2021 Jun 2 11:30:18 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/24 is down (Initializing) 2021 Jun 2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from Ethernet1/22 to Ethernet1/23 2021 Jun 2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/22 is down 2021 Jun 2 11:30:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 10000000 Kbit 2021 Jun 2 11:30:18 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/22 is down (Initializing) 2021 Jun 2 11:30:21 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/22 is up 2021 Jun 2 11:30:21 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/24 is up 2021 Jun 2 11:30:21 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 20000000 Kbit 2021 Jun 2 11:30:21 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk 2021 Jun 2 11:30:21 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/21 is up 2021 Jun 2 11:30:21 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 30000000 Kbit 2021 Jun 2 11:30:21 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/24 is up in mode trunk 2021 Jun 2 11:30:21 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 40000000 Kbit 2021 Jun 2 11:30:21 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/21 is up in mode trunk
Here is config for port-channel and one of its members:
interface port-channel19 switchport switchport mode trunk switchport trunk allowed vlan 19-36 interface Ethernet1/23 no cdp enable switchport switchport mode trunk switchport trunk allowed vlan 19-36 channel-group 19 mode active no shutdown
We talked to service provider who is connected to the other side of this port-channel and they have not found any issues on their end. Also we had no issues when we had N6001 in place.
06-09-2021 01:05 PM - edited 06-09-2021 01:07 PM
before dig in to issue config look odd for me ?
interface port-channel19 switchport switchport mode trunk switchport trunk allowed vlan 1901-1936 interface Ethernet1/23 no cdp enable switchport switchport mode trunk switchport trunk allowed vlan 19-36 channel-group 19 mode active no shutdown
what is the device other side connected?
can you post-show version?
06-09-2021 01:14 PM
Regarding vlans, that was a typo in my post. I corrected it.
As for device on the other side, that is an unknown.
Here is "show version" output for our device:
Software BIOS: version 07.68 NXOS: version 9.3(7) BIOS compile time: 05/18/2020 NXOS image file is: bootflash:///nxos.9.3.7.bin NXOS compile time: 3/10/2021 3:00:00 [03/10/2021 08:39:29] Hardware cisco Nexus9000 C93180YC-EX chassis Intel(R) Xeon(R) CPU @ 1.80GHz with 24631956 kB of memory. Processor Board ID FDO22241DHP Device name: SW9300EX bootflash: 39603933 kB Kernel uptime is 7 day(s), 13 hour(s), 45 minute(s), 23 second(s) Last reset at 471231 usecs after Wed Jun 2 02:23:13 2021 Reason: Reset due to upgrade System version: 7.0(3)I7(9) Service: plugin Core Plugin, Ethernet Plugin Active Package(s):
06-09-2021 01:18 PM
It appears that all the physical ports are flapping. Try setting the speed for each physical port.
speed xxxx
HTH
06-15-2021 09:44 AM
I have set speed to 10000 and duplex to full. However I could not disable auto-negotiate. When trying to disable it, I would get and error:
SW9300EX(config-if)# no negotiate auto ERROR: cannot turn off auto-negotiate at 10G speed
I'm going to leave it like this and monitor for 2-3 weeks. Hopefully it resolves the issue. I'll write an update after.
06-09-2021 01:38 PM
this required both the side need to match some time speed and port you connected another side.
1. check the Physical connection.
2. right speed and SFP used.
06-09-2021 03:02 PM
We're using the same SFPs we used in N6001. Literally we took SFPs from N6001 and inserted them into this one.
We're using DWDM SFPs.
our switch <> DWDM <> DWDM wavelength converter to 1310nm <> provider's device
I can try hard coding speed of our interfaces. I'll need to schedule night maintenance window to get this done.
I'll post my results once its done.
06-10-2021 12:11 AM
since it is connecting to the provider i suggest set the port no auto-negotiate and speed and test and observe.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide