Packets dropped on port-channel c9300

[ZainChaudhry]

Hi guys, 

I have a scenario in the data centre where I have catalyst c9300 connected to the firewall. Two ports on c9300 are on port-channel LACP active mode where the ports on the firewall are LACP passive. I am seeing packets dropped on one of the switch interfaces that is part of the port channel and I am not sure why.

Could this be because of LACP config not matching on both devices?

Thanks in advance.

[ZainChaudhry]

HI Georg,

I have set the load-balancing to src-dst-mixed-ip-port and it resolved the issue. Traffic seems to be load balanced equally now with no packet drops.

Thanks a lot 

[balaji.bandi]

what is the existing arrangement -

#show EtherChannel load-balance 

I see Lot of output drops

post output :

#show platform hardware fed switch active qos queue config interface gigabitEthernet 1/0/3
#show platform hardware fed switch active qos queue config interface gigabitEthernet 2/0/3

If you like to increase softmax - suggested config.

(config)#qos queue-softmax-multiplier 1200

This is 2GB Port-channel, Do you have traffic more than 1GB  all time ? if not try below test

since you mentioned packet drops, when you get chance, shutdown 1 of the Physical Link and see if there any improvement, do other link same.

[ZainChaudhry]

Hi Balaji,

Please see the outputs below:

GSUISW01#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-mac

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
IPv4: Source MAC address
IPv6: Source MAC address

GSUISW01#show platform hardware fed switch active qos queue config interface gigabitEthernet 1/0/3
Asic:0 Core:1 DATA Port:2 GPN:1354 LinkSpeed:0x1
AFD:Disabled FlatAFD:Disabled QoSMap:0 HW Queues: 16 - 23
DrainFast:Disabled PortSoftStart:2 - 1800
DTS Hardmax Softmax PortSMin GlblSMin PortStEnd
----- -------- -------- -------- -------- ---------
0 1 2 200 3 800 19 475 0 0 3 2400
1 1 0 0 4 1200 19 712 8 300 3 2400
2 1 0 0 0 0 0 0 0 0 3 2400
3 1 0 0 0 0 0 0 0 0 3 2400
4 1 0 0 0 0 0 0 0 0 3 2400
5 1 0 0 0 0 0 0 0 0 3 2400
6 1 0 0 0 0 0 0 0 0 3 2400
7 1 0 0 0 0 0 0 0 0 3 2400
Priority Shaped/shared weight shaping_step sharpedWeight
-------- ------------- ------ ------------ -------------
0 0 Shared 50 0 0
1 0 Shared 75 0 0
2 0 Shared 10000 0 0
3 0 Shared 10000 0 0
4 0 Shared 10000 0 0
5 0 Shared 10000 0 0
6 0 Shared 10000 0 0
7 0 Shared 10000 0 0
Port Port Port Port
Priority Shaped/shared weight shaping_step
-------- ------------- ------ ------------
2 Shaped 254 255

Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
------- ------- ------- ------- ------- ------- ------- ------- ------
0 0 796 0 0 890 0 0 1000 0
1 0 956 0 0 1068 0 0 1200 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0

GSUISW01#show platform hardware fed switch active qos queue config interface gigabitEthernet 2/0/3
Interface GigabitEthernet2/0/3 not found, err 18

Not sure why the above error on interface gig2/0/3

Thanks,

[ZainChaudhry]

My bad, please see the output below:

GSUISW01#show platform hardware fed switch 2 qos queue config interface gigabitEthernet 2/0/3
Asic:0 Core:1 DATA Port:2 GPN:1356 LinkSpeed:0x1
AFD:Disabled FlatAFD:Disabled QoSMap:0 HW Queues: 16 - 23
DrainFast:Disabled PortSoftStart:2 - 1800
DTS Hardmax Softmax PortSMin GlblSMin PortStEnd
----- -------- -------- -------- -------- ---------
0 1 2 200 3 800 19 475 0 0 3 2400
1 1 0 0 4 1200 19 712 8 300 3 2400
2 1 0 0 0 0 0 0 0 0 3 2400
3 1 0 0 0 0 0 0 0 0 3 2400
4 1 0 0 0 0 0 0 0 0 3 2400
5 1 0 0 0 0 0 0 0 0 3 2400
6 1 0 0 0 0 0 0 0 0 3 2400
7 1 0 0 0 0 0 0 0 0 3 2400
Priority Shaped/shared weight shaping_step sharpedWeight
-------- ------------- ------ ------------ -------------
0 0 Shared 50 0 0
1 0 Shared 75 0 0
2 0 Shared 10000 0 0
3 0 Shared 10000 0 0
4 0 Shared 10000 0 0
5 0 Shared 10000 0 0
6 0 Shared 10000 0 0
7 0 Shared 10000 0 0
Port Port Port Port
Priority Shaped/shared weight shaping_step
-------- ------------- ------ ------------
2 Shaped 254 255

Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
------- ------- ------- ------- ------- ------- ------- ------- ------
0 0 796 0 0 890 0 0 1000 0
1 0 956 0 0 1068 0 0 1200 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0

Thanks,

[balaji.bandi]

try change this :

qos queue-softmax-multiplier 1200

see any improvement, we do on msot 9300 have good persoformance improved.

then you need to look at etherchannel load-balance.

[Leo Laohoo]

What firmware is the 9300 running on?

Kindly provide the complete output to the following commands: 

sh platform pm interface-flap Gi1/0/3
sh platform pm interface-flap Gi2/0/3

NOTE:  Do not provide screenshots and remove all relevant VLAN information. 

[ZainChaudhry]

HI Leo,

The Software Image is CAT9K_IOSXE and the SW version is 16.12.4

GSUISW01#show platform pm interface-flaps gigabitEthernet 1/0/3

Field AdminFields OperFields
===============================================================
Access Mode Static Static
Access Vlan Id 1 0
Voice Vlan Id 4096 0
VLAN Unassigned 1
ExAccess Vlan Id 32767
Native Vlan Id 1
Port Mode dynamic trunk
Encapsulation 802.1Q 802.1Q
disl trunk on
Media unknown
DTP Nonegotiate 0 0
Port Protected 0 0
Unknown Unicast Blocked 0 0
Unknown Multicast Blocked 0 0
Vepa Enabled 0 0
Span Destination 0

Duplex auto full
Default Duplex auto
Speed auto 1000
Auto Speed Capable 1 1
No Negotiate 0 0
No Negotiate Capable 0 0
Flow Control Receive ON ON
Flow Control Send Off Off
Jumbo 0 0
saved_holdqueue_out 0
saved_input_defqcount 2000
Jumbo Size 1500

Forwarding Vlans : none
Current Pruned Vlans : none
Previous Pruned Vlans : none

Sw LinkNeg State : LinkStateUp
No.of LinkDownEvents : 1
XgxsResetOnLinkDown(10GE):
Time Stamp Last Link Flapped(U) : Oct 7 04:57:31.401
LastLinkDownDuration(sec) 22
LastLinkUpDuration(sec): 25654524
GSUISW01#show platform pm interface-flaps gigabitEthernet 2/0/3

Field AdminFields OperFields
===============================================================
Access Mode Static Static
Access Vlan Id 1 0
Voice Vlan Id 4096 0
VLAN Unassigned 1
ExAccess Vlan Id 32767
Native Vlan Id 1
Port Mode dynamic trunk
Encapsulation 802.1Q 802.1Q
disl trunk on
Media unknown
DTP Nonegotiate 0 0
Port Protected 0 0
Unknown Unicast Blocked 0 0
Unknown Multicast Blocked 0 0
Vepa Enabled 0 0
Span Destination 0

Duplex auto full
Default Duplex auto
Speed auto 1000
Auto Speed Capable 1 1
No Negotiate 0 0
No Negotiate Capable 0 0
Flow Control Receive ON ON
Flow Control Send Off Off
Jumbo 0 0
saved_holdqueue_out 0
saved_input_defqcount 2000
Jumbo Size 1500

Forwarding Vlans : none
Current Pruned Vlans : none
Previous Pruned Vlans : none

Sw LinkNeg State : LinkStateUp
No.of LinkDownEvents : 1
XgxsResetOnLinkDown(10GE):
Time Stamp Last Link Flapped(U) : Oct 7 04:59:45.630
LastLinkDownDuration(sec) 63
LastLinkUpDuration(sec): 25654417

[Leo Laohoo]

---

@ZainChaudhry wrote:

Time Stamp Last Link Flapped(U) : Oct 7 04:57:31.401
LastLinkUpDuration(sec): 25654524

Time Stamp Last Link Flapped(U) : Oct 7 04:59:45.630
LastLinkUpDuration(sec): 25654417

---

Ok, I got some good news and some bad news.   
Let me start with the good news:  Link is stable since 07 October 2020.  Next, look a the "sh interface Gi1/0/3" output.  Pay attention to the Total Output Drops (723423) and then compare the value against the Total Output packets (cannot see the value because it has been edited out).  The result is very, very, very TINY.  It is so tiny, in percentage, that I would even go as far as considering it as "insignificant".  

---

@ZainChaudhry wrote:

The Software Image is CAT9K_IOSXE and the SW version is 16.12.4

---

And for the bad news:  16.12.X (the entire train) is buggy AF.  I have so many TAC Cases with 16.12.X and avoid using the entire 16.12.X train.
NOTE:  I have several cases with 16.12.3 (including rebuild) and 16.12.5b but the most (>12 cases) troublesome is version 16.12.4.  

[ZainChaudhry]

HI Leo,

Thanks for your suggestions. I have one more question though. I have just cleared the counters on both the interfaces at the same time and please see the output of show interfaces below:

GSUISW01#sh interfaces gigabitEthernet 1/0/3
GigabitEthernet1/0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 3c51.0ec1.bb03 (bia 3c51.0ec1.bb03)
Description: "Uplink to PA-port1"
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 12/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:20, output 00:00:00, output hang never
Last clearing of "show interface" counters 04:06:32
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 649
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 10949000 bits/sec, 4056 packets/sec
5 minute output rate 48904000 bits/sec, 9296 packets/sec
53139654 packets input, 19282830691 bytes, 0 no buffer
Received 550 broadcasts (493 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 493 multicast, 0 pause input
0 input packets with dribble condition detected
118047359 packets output, 60413053941 bytes, 0 underruns
Output 101549 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

GSUISW01#sh interfaces gigabitEthernet 2/0/3
GigabitEthernet2/0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 3c51.0ec1.c503 (bia 3c51.0ec1.c503)
Description: "Uplink to PA-port2"
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output 00:00:00, output hang never
Last clearing of "show interface" counters 04:06:33
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 9754000 bits/sec, 2961 packets/sec
5 minute output rate 3000 bits/sec, 4 packets/sec
42079374 packets input, 18917814844 bytes, 0 no buffer
Received 494 broadcasts (494 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 494 multicast, 0 pause input
0 input packets with dribble condition detected
58592 packets output, 5544688 bytes, 0 underruns
Output 0 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

The output packets on interface gig1/0/3 are so much higher as compare to interface gig2/0/3. What might be causing this behavior?

Thanks,

[paul driver]

Hello

---

@ZainChaudhry wrote:

Hi guys, 

I have a scenario in the data centre where I have catalyst c9300 connected to the firewall. Two ports on c9300 are on port-channel LACP active mode where the ports on the firewall are LACP passive.

---

On a side note possibley change the LB  forwarding method

Fw side ->  destintation based towards mutilpe hosts 
Switch>  source based towards a single host (FW)