Nexus C93180YC-EX - random port-channel member shutdown

[vitalio]

We recently moved from Nexus6001 to NexusC93180YC-EX and noticed that port-channel members would sometimes shutdown and then come back up.

As an example a port-channel with 4 members might see all 4 member shutdown or just one or two or 3 of them. It is completely random, no specific time of day.
Here is log showing port-channel members go down and then come back up:

2021 Jun  2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/23 is down
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 30000000 Kbit
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/23 is down (Initializing)
2021 Jun  2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/21 is down
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 20000000 Kbit
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/21 is down (Initializing)
2021 Jun  2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from Ethernet1/22 to Ethernet1/24
2021 Jun  2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/22 is down
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 10000000 Kbit
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/22 is down (Initializing)
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel19 is down (No operational members)
2021 Jun  2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from Ethernet1/24 to none
2021 Jun  2 11:25:15 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/24 is down
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 100000 Kbit
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/24 is down (Initializing)
2021 Jun  2 11:25:15 SW9300EX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel19 is down (No operational members)
2021 Jun  2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/24 is up
2021 Jun  2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from none to Ethernet1/24
2021 Jun  2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/22 is up
2021 Jun  2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/21 is up
2021 Jun  2 11:25:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/23 is up
2021 Jun  2 11:25:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 10000000 Kbit
2021 Jun  2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/24 is up in mode trunk
2021 Jun  2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface port-channel19 is up in mode trunk
2021 Jun  2 11:25:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 20000000 Kbit
2021 Jun  2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2021 Jun  2 11:25:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 30000000 Kbit
2021 Jun  2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/21 is up in mode trunk
2021 Jun  2 11:25:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 40000000 Kbit
2021 Jun  2 11:25:18 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/23 is up in mode trunk
2021 Jun  2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/21 is down
2021 Jun  2 11:30:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 30000000 Kbit
2021 Jun  2 11:30:18 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/21 is down (Initializing)
2021 Jun  2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from Ethernet1/24 to Ethernet1/22
2021 Jun  2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/24 is down
2021 Jun  2 11:30:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 20000000 Kbit
2021 Jun  2 11:30:18 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/24 is down (Initializing)
2021 Jun  2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel19: first operational port changed from Ethernet1/22 to Ethernet1/23
2021 Jun  2 11:30:18 SW9300EX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel19: Ethernet1/22 is down
2021 Jun  2 11:30:18 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 10000000 Kbit
2021 Jun  2 11:30:18 SW9300EX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/22 is down (Initializing)
2021 Jun  2 11:30:21 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/22 is up
2021 Jun  2 11:30:21 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/24 is up
2021 Jun  2 11:30:21 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 20000000 Kbit
2021 Jun  2 11:30:21 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2021 Jun  2 11:30:21 SW9300EX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel19: Ethernet1/21 is up
2021 Jun  2 11:30:21 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 30000000 Kbit
2021 Jun  2 11:30:21 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/24 is up in mode trunk
2021 Jun  2 11:30:21 SW9300EX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel19,bandwidth changed to 40000000 Kbit
2021 Jun  2 11:30:21 SW9300EX %ETHPORT-5-IF_UP: Interface Ethernet1/21 is up in mode trunk

Here is config for port-channel and one of its members:

interface port-channel19
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 19-36

interface Ethernet1/23
  no cdp enable
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 19-36
  channel-group 19 mode active
  no shutdown

We talked to service provider who is connected to the other side of this port-channel and they have not found any issues on their end. Also we had no issues when we had N6001 in place.

[balaji.bandi]

before dig in to issue config look odd for me ?

interface port-channel19
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 1901-1936

interface Ethernet1/23
  no cdp enable
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 19-36
  channel-group 19 mode active
  no shutdown

what is the device other side connected?

can you post-show version?

[vitalio]

Regarding vlans, that was a typo in my post. I corrected it.
As for device on the other side, that is an unknown.

Here is "show version" output for our device:

Software
  BIOS: version 07.68
 NXOS: version 9.3(7)
  BIOS compile time:  05/18/2020
  NXOS image file is: bootflash:///nxos.9.3.7.bin
  NXOS compile time:  3/10/2021 3:00:00 [03/10/2021 08:39:29]


Hardware
  cisco Nexus9000 C93180YC-EX chassis
  Intel(R) Xeon(R) CPU  @ 1.80GHz with 24631956 kB of memory.
  Processor Board ID FDO22241DHP

  Device name: SW9300EX
  bootflash:   39603933 kB
Kernel uptime is 7 day(s), 13 hour(s), 45 minute(s), 23 second(s)

Last reset at 471231 usecs after Wed Jun  2 02:23:13 2021
  Reason: Reset due to upgrade
  System version: 7.0(3)I7(9)
  Service:

plugin
  Core Plugin, Ethernet Plugin

Active Package(s):

[Reza Sharifi]

It appears that all the physical ports are flapping. Try setting the speed for each physical port.

speed xxxx

HTH

[vitalio]

I have set speed to 10000 and duplex to full. However I could not disable auto-negotiate. When trying to disable it, I would get and error:

SW9300EX(config-if)# no negotiate auto
ERROR: cannot turn off auto-negotiate at 10G speed

I'm going to leave it like this and monitor for 2-3 weeks. Hopefully it resolves the issue. I'll write an update after.

[balaji.bandi]

this required both the side need to match some time speed and port you connected another side.

1. check the Physical connection.

2. right speed and SFP used.

[vitalio]

We're using the same SFPs we used in N6001. Literally we took SFPs from N6001 and inserted them into this one.
We're using DWDM SFPs.
our switch <> DWDM <> DWDM wavelength converter to 1310nm <> provider's device
I can try hard coding speed of our interfaces. I'll need to schedule night maintenance window to get this done.

I'll post my results once its done.

[balaji.bandi]

since it is connecting to the provider i suggest set the port no auto-negotiate and speed and test and observe.