Discards on LACP interfaces connected to PAN Firewalls

[scottish]

I have two interfaces bound to a port channel that face an AE on a PAN firewall. The site is operating incredibly slowly and I'm seeing a lot of discards on the physical interfaces and the port channel interfaces facing the PAN. PAN is telling me this is normal and these are drops from the PAN because they're hitting a drop policy. This doesn't seem right to me though as it should be the PAN dropping them, not the switch. The PAN is also seeing drops. Since yesterday at 3pm Pacific, the pan has racked up 2936203 drops, and the Catalyst 9300-P has accumulated 2341443 discards. The port channel consists of 2 vlans that are ingress/egress internet traffic.

Thoughts and experiences would be appreciated.

[balaji.bandi]

Is this stack switch, what IOS XE code running.

can you post show interface port-channel X and show interface x/x x/y (both interface output)

the drop you mentioning is Interface input and output drops ? 

what is the link usage peak time ? (what speed links these are 1G x 2 or 12 X10G >?)

Also check the Physical cables see if you can replace them .

If you are using LACP what kind of Load share mechanism using ?

[Reza Sharifi]

Is the Po configured with mode "on"?

Also, for testing, what happens if you shut the redundant interface on the switch?

HTH

[scottish]

BTW, TY Both for chiming in.

PO5 has Gig1/0/5 and 2/0/5 in this 9300 Catalyst switch stack. Connections between the switch and firewall have been replaced one at a time. Even while running on one interface, the discards continue to increment. I am not seeing any LACP errors. The switch counters were cleared yesterday at 2:30pm PST

IOSXE 17.09.05

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Gi1/0/5 0 0 0 0 0 1293343
Gi2/0/5 0 0 0 0 0 1497359
Po5 0 0 0 0 0 2790897

edgesw01#sh run int gig 1/0/5
interface GigabitEthernet1/0/5
description Eth1/1pan01b
switchport trunk allowed vlan 911,912
switchport mode trunk
channel-group 5 mode active
end

edgesw01#sh run int gig 2/0/5
interface GigabitEthernet2/0/5
description Eth1/2pan01b
switchport trunk allowed vlan 911,912
switchport mode trunk
channel-group 5 mode active
end
!
edgesw01#sh run int po5
interface Port-channel5
description pan01b.eth1/1&2
switchport trunk allowed vlan 911,912
switchport mode trunk
end
!
edgesw01#show int gig 1/0/5
GigabitEthernet1/0/5 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 8c44.a54a.1f05 (bia 8c44.a54a.1f05)
Description: Eth1/1pan01b
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 22/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:24, output 00:00:04, output hang never
Last clearing of "show interface" counters 21:36:35
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1295925
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 10215000 bits/sec, 4491 packets/sec
5 minute output rate 89257000 bits/sec, 8628 packets/sec
134980747 packets input, 105826758168 bytes, 0 no buffer
Received 7909 broadcasts (5186 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5186 multicast, 0 pause input
0 input packets with dribble condition detected
248470704 packets output, 246114434228 bytes, 0 underruns
Output 9456 broadcasts (9456 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
!

edgesw01#sh int gig 2/0/5
GigabitEthernet2/0/5 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 8c44.a548.c885 (bia 8c44.a548.c885)
Description: Eth1/2pan01b
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 19/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:03, output hang never
Last clearing of "show interface" counters 21:36:45
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1499705
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 11753000 bits/sec, 6819 packets/sec
5 minute output rate 75773000 bits/sec, 7719 packets/sec
143919930 packets input, 115565938639 bytes, 0 no buffer
Received 5186 broadcasts (5186 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5186 multicast, 0 pause input
0 input packets with dribble condition detected
236744701 packets output, 230355749624 bytes, 0 underruns
Output 478764 broadcasts (87670 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
!

edgesw01#sh int po5
Port-channel5 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 8c44.a54a.1f05 (bia 8c44.a54a.1f05)
Description: pan01b.eth1/1&2
MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 21/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is N/A
input flow-control is on, output flow-control is unsupported
Members in this channel: Gi1/0/5 Gi2/0/5
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 21:36:47
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2795678
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 22889000 bits/sec, 11511 packets/sec
5 minute output rate 164712000 bits/sec, 16374 packets/sec
279012229 packets input, 221434772321 bytes, 0 no buffer
Received 13098 broadcasts (10375 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 10375 multicast, 0 pause input
0 input packets with dribble condition detected
485330418 packets output, 476597340174 bytes, 0 underruns
Output 488264 broadcasts (97135 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
!

edgesw01#show lacp 5 internal detail
Flags: S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode

Channel group 5

Actor (internal) information:

Actor Actor Actor
Port System ID Port Number Age Flags
Gi1/0/5 32768,8c44.a54a.1f00 0x106 18s SA

LACP Actor Actor Actor
Port Priority Oper Key Port State
32768 0x5 0x3D

Port State Flags Decode:
Activity: Timeout: Aggregation: Synchronization:
Active Long Yes Yes

Collecting: Distributing: Defaulted: Expired:
Yes Yes No No
Actor Actor Actor
Port System ID Port Number Age Flags
Gi2/0/5 32768,8c44.a54a.1f00 0x206 15s SA

LACP Actor Actor Actor
Port Priority Oper Key Port State
32768 0x5 0x3D

Port State Flags Decode:
Activity: Timeout: Aggregation: Synchronization:
Active Long Yes Yes

Collecting: Distributing: Defaulted: Expired:
Yes Yes No No

[Elliot Dierksen]

I am going to make a guess that you are overflowing the output queue. Your input queue size is 2000 packets, but the output queue is only 40 packets. This article might help clarify things.https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9600-series-switches/220491-understand-output-drops-on-high-speed-in.html 

[Joseph W. Doherty]

". . . but the output queue is only 40 packets."

BTW, on Cisco switches, the "interface" output queue is generally meaningless.

I believe the reference provided has all the necessary information for possible mitigation of the switch's egress drops.

I won't rehash the information in that reference, but if you have questions about the information in that reference, post follow-up questions, possibly I might be able to help.

[MHM Cisco World]

Send below to me as PM

Show lacp count 

Show lacp neighbor 

MHM

[paul driver]

Hello
Try either setting the cisco switch LACP passive with the PAs active or use a static PC instead?