Increasing MTU on port-channel with VPC

[Anthony O'Reilly]

Hello,

I need to enabled jumbo frame on the port-channels on Nexus 7706. These Nexus 7Ks also have OTV which is use to stretch VLANs between Data Centres.

On all the port-channels, the MTU is : 1500 as below:

interface port-channel101
description ### VPC for OTV ###
switchport
switchport mode trunk
switchport trunk allowed vlan 2
spanning-tree port type normal
mtu 1500
lacp max-bundle 16
vpc 101

When I increased the MTU from 1500 to 9216, I get:

Nexus_A# sh int po101
port-channel101 is down (suspended by vpc)
admin state is up
vPC Status: Down, vPC number: 101 [packets forwarded via vPC peer-link]

I have auto-recovery enabled on the VPC as below:

vpc domain 1
peer-switch
role priority 4086
peer-keepalive destination 1.1.1.1 source 1.1.1.2 vrf Keepalive
peer-gateway exclude-vlan 998
layer3 peer-router
no layer3 peer-router syslog
auto-recovery reload-delay 600
ip arp synchronize

How can I make the port-channel come up smoothly? Do I need to do:

• channel-group 101 force mode active
• shut and no shut
• remove vpc 101 and add it again
• wait for 600 seconds 

Thanks

Anthony.

[shoppman]

Hi Anthony,

I see it's been a while since you asked for help on this.  However, here is an explanation in case you would still like to know more about this behavior.

Nexus VPC port-channels have parameters that must match in order for the port-channel to be up and active from both VPC peers down to the 3rd device.  If the parameters do not match between VPC peers, it will result in failed consistency check.  These consistency check failures, and their subsequent reactions, depend on the specific parameter.    Some mismatched parameters result in unexpected forwarding behavior.  These are considered Type-2 failures and are generally less impactful.  Other mismatched parameters are more impactful and considered Type-1 failures.  MTU falls into this later category.  As a result, when this mismatch is detected, the switch takes action to lessen the impact.  Specifically, the VPC member port on the VPC Secondary device will shut it's member port where the Type-1 mismatch is present.  This port-channel will remain shut on the Secondary device until the inconsistency is corrected by ensuring the MTU of port-channel 101 is the same on both VPC peers.

For instance, in your example, it looks like the MTU was changed to jumbo on port-channel 101 only on Nexus A.  If the MTU on Nexus B has not yet been raised, whoever the VPC Secondary switch is will shut port-channel 101.   Port-channel 101 is expected to remain up and passing traffic in the VPC Primary in this scenario.  Nexus A appears to have been the Standby in your case since we can see the port-channel was shut/suspended on this switch.  As a result, any traffic that needs to leave po 101 would then have to be forwarded across the peer link to Nexus B who would then send it out of po 101.

To resolve this state, you need only raise the MTU on the same port-channel on the VPC peer Nexus B, which would correct the mismatch, thereby removing the port from suspension on Nexus A.

You can learn more about VPC Type-1 and Type-2 consistency parameters and how the switch handles them in the VPC Design and Configuration Guide.

https://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf#%5B%7B%22num%22%3A18%2C%22gen%22%3A0%7D%2C%7B%22name%22%3A%22XYZ%22%7D%2C105%2C289%2C0%5D

I hope this helps!

-Scott Hoppmann

[Anthony O'Reilly]

Hi @shoppman 

Thanks for all the detail. Apologies for the late reply, I was away.

For interfaces that were not part of a port-channel, I added the command "mtu 9216" under the interface and for interfaces that were a member of a port-channel, I added the command "mtu 9216" under the port-channel interface.

This was not service impacting for the links or routing between the DCs. This change went very well. Wish they all went as well

[ddihealth]

Hi Anthony

I am having similar issue on a new switch I am bringing online-it doesn't allow me to increase MTU on the peer-link:

SW11-L(config)# interface port-channel936
(config-if)# description To DC3NS-SW12-L E1/47-48
(config-if)# switchport
(config-if)# switchport mode trunk
(config-if)# spanning-tree port type network
(config-if)# mtu 9216
ERROR: Cannot configure port MTU on vPC Peer-link  <<<<<<<<<<<<<<<<<<<<<<<<<
(config-if)# vpc peer-link
(config-if)#

However, the predecessor had MTU 9216 on the VPC Peer-link , any suggestions ?

[shoppman]

Hi ddihealth,

The VPC peer-link MTU is managed by the VPC process directly and is not able to be manually modified.  It should be adjusted by default to the max MTU of 9216 when the 'vpc peer-link' configuration is applied. After configuring the peer-link, I would suggest checking the running-configuration and other related outputs that show the MTU to confirm.

I hope this helps!

-Scott Hoppmann