We have an occurring issue across the board with our N9k's. Each site has a pair of N9k's connected redundantly to many catalyst switches, and all traffic going out to these switches only use one link, and the N9k's do not load balance the traffic going out to the connected switches. Fail over works as expected.
For example and for the easiest setup, we have a pair of N9k's that have redundant connection to a customer's Cisco router with the ports configured as access ports, the customer is a service provider. This provider offers free Wi-Fi to many customers. All traffic leaving our N9k's prefer switch 1, and does not load balance at all. We are contracted to provide a redundant 2Gbps (1Gbps + 1Gbps) service. We are currently maxing out one of the 1Gbps during peak times, resulting in drop packets. Again, failover works as expected. The customer router does not have 10Gbps ports.
We were on nxos.7.0.3.I7.10.1, we tried upgrading to nxos.9.3.9, and that did not change anything.
Both switches have this config:
interface port-channel60
description LAG-60: Trunk to XXXXX 2Gbps service
switchport access vlan 299
vpc 60
interface Ethernet1/30
description LAG-60 XXXXX 2Gbps Circuit
switchport access vlan 299
spanning-tree port type edge
channel-group 60 mode active
switch 1:
show port-channel load-balance
System config:
Non-IP: src-dst mac
IP: src-dst mac rotate 0
Port Channel Load-Balancing Configuration for all modules:
Module 1:
Non-IP: src-dst mac
IP: src-dst mac rotate 0
sh port-channel traffic int port 60
NOTE: Clear the port-channel member counters to get accurate statistics
ChanId Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst
------ --------- ------- ------- ------- ------- ------- -------
60 Eth1/30 92.06% 100.00% 89.53% 93.95% 0.0% 0.0%
sh int e1/30
Ethernet1/30 is up
admin state is up, Dedicated Interface
Belongs to Po60
Hardware: 1000/10000 Ethernet, address: cc46.d6b3.9af1 (bia cc46.d6b3.9af1)
Description: LAG-60 MCCS 2Gbps Circuit
MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 255/255, txload 138/255, rxload 7/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 00:52:22
Last clearing of "show interface" counters 00:42:48
0 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 28293864 bits/sec, 9282 packets/sec
30 seconds output rate 542656384 bits/sec, 52958 packets/sec
input rate 28.29 Mbps, 9.28 Kpps; output rate 542.66 Mbps, 52.96 Kpps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 29151240 bits/sec, 8958 packets/sec
300 seconds output rate 505885544 bits/sec, 50074 packets/sec
input rate 29.15 Mbps, 8.96 Kpps; output rate 505.89 Mbps, 50.07 Kpps
RX
24968459 unicast packets 92 multicast packets 0 broadcast packets
24968551 input packets 9663539018 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
130388484 unicast packets 346 multicast packets 0 broadcast packets
130388830 output packets 165826559252 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 3254413 output discard
0 Tx pause
switch 2:
sh port-channel load-balance
System config:
Non-IP: src-dst mac
IP: src-dst mac rotate 0
Port Channel Load-Balancing Configuration for all modules:
Module 1:
Non-IP: src-dst mac
IP: src-dst mac rotate 0
sh port-channel traffic int port 60
NOTE: Clear the port-channel member counters to get accurate statistics
ChanId Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst
------ --------- ------- ------- ------- ------- ------- -------
60 Eth1/30 76.56% 0.0% 65.21% 85.14% 0.0% 0.0%
sh int e1/30
Ethernet1/30 is up
admin state is up, Dedicated Interface
Belongs to Po60
Hardware: 1000/10000 Ethernet, address: cc46.d6b3.9e55 (bia cc46.d6b3.9e55)
Description: LAG-60 MCCS 2Gbps Circuit
MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 255/255, txload 1/255, rxload 9/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 00:38:29
Last clearing of "show interface" counters 00:51:43
1 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 36408568 bits/sec, 9263 packets/sec
30 seconds output rate 456 bits/sec, 0 packets/sec
input rate 36.41 Mbps, 9.26 Kpps; output rate 456 bps, 0 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 45266168 bits/sec, 10345 packets/sec
300 seconds output rate 248 bits/sec, 0 packets/sec
input rate 45.27 Mbps, 10.35 Kpps; output rate 248 bps, 0 pps
RX
23668767 unicast packets 101 multicast packets 0 broadcast packets
23668868 input packets 8934126866 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 1560 multicast packets 0 broadcast packets
1560 output packets 142946 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
