Solved: Best practice for a trunk Etherchannel between two Cisco 9300 stack.

billy_vaughn · ‎04-13-2022

I'm looking for some best practices for setting up a trunked Etherchannel between two 9300 switch stacks. We have had issues with Etherchannel setup since moving to the 9300 platform. I've always configured a port channel > then setup my interfaces with the following and they work until we got 9300's. I'm not sure whether the non negotiate commands are a problem or not but this setup below is currently working but we are trying to add another new stack and the interfaces go into err disable state as soon as you do a shut/no shut. Same config as below on the new stack but with a new

Switch 1

interface Port-channel1
switchport mode trunk
switchport nonegotiate

interface GigabitEthernet1/1/1
description Uplink to core
switchport mode trunk
switchport nonegotiate
channel-group 1 mode on
!
interface GigabitEthernet1/1/2
description Uplink to core
switchport mode trunk
switchport nonegotiate
channel-group 1 mode on

Switch 2

interface Port-channel23
description ESX_DEV_SWITCH Trunk
switchport mode trunk

interface GigabitEthernet1/0/5
description ESX Dev Sw1 Uplink
switchport mode trunk
channel-group 23 mode on
!
interface GigabitEthernet1/0/6
description ESX Dev Sw3 Uplink
switchport mode trunk
channel-group 23 mode on

billy_vaughn · ‎04-18-2022

We got this working. We did remove the nonnegotiate commands and then brought up the interfaces one at a time until they were joined and working.

View solution in original post

billy_vaughn · ‎04-13-2022

I wanted to add that I have setup a new port channel for the new stack. port channel 5 on my core. It cut that off my original post.

Reza Sharifi · ‎04-13-2022

What happens if you delete "switchport nonegotiate" from the Portchannel and the physical interfaces?

If removing "switchport nonegotiate" command does not help, can you try the following:

delete the portchannel

delete the channel-group x mode on from the physical interface.

shut down the physical interface

and redo everything this way

Do not use the same Portchannle number that was not working. use a new one.

under the physical interface add channel-group x mode on

un-shut the interfaces

By doing a no sh to the interfaces the new Portchannle should be created and the Portchannel should come up, if not make sure that the Portchannel is not shut and test again.

HTH

billy_vaughn · ‎04-13-2022

Same thing, it goes into the err disable state with a shut/no shut.

David Ruess · ‎04-13-2022

Hello,

There could be just some issues with the ether channel configuration to put it into err-disable mode. As I'm sure you know the ports have to be the same config to bundle and pair with the other side. Sometimes speed/duplex are negotiated if not specifically configured and could cause issues. I would hard set these values, as well as the no negotiate command on ALL interfaces participating if thats what you are trying to achieve.

Some ether channel configurations also need a minimum amount of ports to be involved before comin gup.

The err-disable state makes me think that Ether channel Misconfiguration Guard is on (default setting I believe) to detect misconfigurations.

Can you show err-disable command?

and

show spanning-tree summary

-David

billy_vaughn · ‎04-13-2022

I currently don't have anything configured for the err-disable so it's using the default.

David Ruess · ‎04-13-2022

Yes I was saying the misconfiguration guard for ether channel is ON by default if I remember correctly.

Tee only difference on your commands I see is the negotiate command. Try removing it from all ports and the ether channel.

David Ruess · ‎04-13-2022

Can you show the output of the below command:

show spanning-tree summary

-David

billy_vaughn · ‎04-14-2022

I checked and I didn't see anything odd.

balaji.bandi · ‎04-13-2022

There are many reasons port go in to error disabled, You did not mention which side error is disable ?

Looking at your config : (post-show version from both the switches)

1. Switch 1 you are using the uplink module to connect to the port-channel (how many switches in the stack ) - post-show switch and show modules help here

2. Switch 2 you are using normal gig ports (in the description you mentioned switch3 uplink, are you sure you connecting right ports here ? (I suspect) - show cdp neigh (help here)

are you sure you connecting these 2 switches only, or do any other devices in the network participate here, do you have any small network diagram, I see one side you connecting core and the other side esxi server?

Let's do a simple test.

switch 1 :

(remove the cables)

default interface GigabitEthernet1/1/1

default interface GigabitEthernet1/1/2

interface GigabitEthernet1/1/1
switchport mode trunk
channel-group 1 mode on
!
interface GigabitEthernet1/1/2
switchport mode trunk
channel-group 1 mode on

!

interface Port-channel1
switchport mode trunk
no shutdown

Switch 2

remove the cables

default interface GigabitEthernet1/0/5

default interface GigabitEthernet1/0/6

interface GigabitEthernet1/0/5
description ESX Dev Sw1 Uplink
switchport mode trunk
channel-group 23 mode on
!
interface GigabitEthernet1/0/6
description ESX Dev Sw3 Uplink
switchport mode trunk
channel-group 23 mode on

interface Port-channel23
description ESX_DEV_SWITCH Trunk
switchport mode trunk

no shut

connect only a single cable and what is the outcome.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

billy_vaughn · ‎04-14-2022

I hadn't considered that the module ports might be a little different that the standard gig ports. This is a stack in an ESX server rack for the many connections an ESX servers needs. The stack it's connecting to is another 9300 stack with other stacks connected to it just fine. I have 3750 stacks and another 9300 stack connected and working fine. On the other 9300 stack that is working I'm also using 4 ports from the module for uplinks to the main 9300 stack. The main 9300 stack is only using the standard gig ports and not the module. It just seems the 9300 stacks don't operate like other switch stacks I've worked with with Etherchannel. Maybe they made changes to STP on the 9300. I may just open up a case with TAC and see what they say. Weird I have one 9300 stack working fine but can't get this 9300 stack up and I've configured the port channel and ports exactly the same for the uplinks.

billy_vaughn · ‎04-14-2022

I did end up getting one standard gig port up and was able to pull in my VTP info and assign an IP on our MGMT VLAN so I could get to the device remotely. I did try one port at at time with the module ports and it goes into err disable as soon as I do a shut/no shut. I was only able to get the standard gig port talking so I'm thinking it's something with the module interfaces. Maybe I need to set the speed on those module interfaces to full/1000.