Lots of discards on Switchport Connected to FTD

[ericdavishop]

I have about 8 FTD's deployed port-channeled and trunked to a Catalyst 9200/9300 switch.  All of the switch ports connected to the FTDs show a lot of discards.  I honed in one of them to see if bandwidth was overutilized and found only 100Mb/s was used.  The ports are set to 1Gb/s and the FTD 1010 is rated for 890Mb/s throughput.  Any ideas why I am seeing so many discards?  I would expect discards if the switch was sending more traffic than the FTD could handle, but at 100Mb/s I cannot understand why I am seeing so many discards.

[MHM Cisco World]

Can you share show interface (port channel and any port member)

MHM

[ericdavishop]

Hi MHM,

HOP-076-SubCellar-STACK#sh int Po1
Port-channel1 is up, line protocol is up (connected)
Hardware is EtherChannel, address is f8e9.4fbc.1bae (bia f8e9.4fbc.1bae)
Description: Firewall01
MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 3/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is N/A
input flow-control is on, output flow-control is unsupported
Members in this channel: Gi1/0/46 Gi1/0/48
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 10481254
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 28575000 bits/sec, 4497 packets/sec
5 minute output rate 29618000 bits/sec, 4718 packets/sec
320352549114 packets input, 310261720029543 bytes, 0 no buffer
Received 35225801 broadcasts (3083976 multicasts)
0 runts, 0 giants, 0 throttles
1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3083976 multicast, 0 pause input
0 input packets with dribble condition detected
325697286617 packets output, 311648364354647 bytes, 0 underruns
Output 685804972 broadcasts (0 multicasts)
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

HOP-076-SubCellar-STACK#sh etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator

M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

A - formed by Auto LAG

Number of channel-groups in use: 8
Number of aggregators: 8

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
1 Po1(SU) LACP Gi1/0/46(P) Gi1/0/48(P)

HOP-076-SubCellar-STACK#sh int Gi1/0/46
GigabitEthernet1/0/46 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is f8e9.4fbc.1bae (bia f8e9.4fbc.1bae)
Description: FirepowerEther chan
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 3/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:08, output 00:00:04, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1837021
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 7141000 bits/sec, 1337 packets/sec
5 minute output rate 12409000 bits/sec, 2343 packets/sec
85125333577 packets input, 66439051181587 bytes, 0 no buffer
Received 1541971 broadcasts (1541971 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1541971 multicast, 0 pause input
0 input packets with dribble condition detected
126417842207 packets output, 104126822711379 bytes, 0 underruns
Output 413716854 broadcasts (0 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

HOP-076-SubCellar-STACK#sh int Gi1/0/48
GigabitEthernet1/0/48 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is f8e9.4fbc.1bb0 (bia f8e9.4fbc.1bb0)
Description: FirepowerEther chan
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 4/255, rxload 5/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:24, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 8644233
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 21713000 bits/sec, 3208 packets/sec
5 minute output rate 17554000 bits/sec, 2418 packets/sec
235227800152 packets input, 243823137055160 bytes, 0 no buffer
Received 33683905 broadcasts (1542013 multicasts)
0 runts, 0 giants, 0 throttles
1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1542013 multicast, 0 pause input
0 input packets with dribble condition detected
199280031113 packets output, 207522005320931 bytes, 0 underruns
Output 272090472 broadcasts (0 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[MHM Cisco World]

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 10481254 <<- this drops is huge 
this output from SW side or FTD side 
if SW side then use 

9300(config)#qos queue-softmax-multiplier 1200

MHM 

[Marius Gunnerud]

Yes that is a big number, but we need to take into consideration when the last reboot of the device was.  So, when was the last reboot of the switch?  The interface counters has never been cleared so if the uptime of the switch is a year or more the number might not actually be that bad as the counter is cumulative.

Also, that the current interface Rx / Tx rate is 100Mb might not be relevant.  How did you verify this? Was it through a show command on the switch or do you have monitoring software where the bandwidth is graphed over time?  If this is viewed just from the CLI show command then there could be times when there are traffic spikes at which time the drops happen.

Issue the following command on the ports to see if the drops were due to a lack of available buffer:

show platform hardware fed switch active qos queue stats interface Gig1/0/48

[ericdavishop]

MHM,

Thank you.  I will try that and monitor.

I have different scenarios too.  So that was one scenario where the switch port was discarding.  I have another scenario where the FTD sub-interface is discarding and the switch port is good.  What can i look at for the FTD side?

[Marius Gunnerud]

Interface drops on the FTD, and ASA for that matter, can be related to a lot of things.  But the most common are drops due to ACL rules.  So in this instance you would need to identify what issues you are facing (i.e. performance, access to services, etc.).

As with the switch counters, the FTD interface drops are cumulative and indicate drops from when they were last cleared or the device was rebooted.

[MHM Cisco World]

SW-FPR
the SW send data faster than the FPR can handle it 
FPR interface send flowcontrol to push send data  
if FPR not send the flowcontrol then the FPR show frame drops and the overrun count is increase for each frame drop

so what FPR platform you have ?
MHM

[ericdavishop]

MHM,

I have FTD 1010.  Do I need to use Flexconfig to configure flowcontrol on the FTD?   I saw mention of discards possibly being caused by ACL rules too?  Is that possible too?  I wouldn't think Layer discards would occur because of Firewall policy.

[MHM Cisco World]

You are correct 

The L1 discards frame dont relate to policy we apply.

So in fmc 

Device-> device management->edit->interface -> hardware 

Under it select flow control.

MHM

[ericdavishop]

The flow control feature does not to be available on the FTD 1010 model.  I think read it only is available on the 3100 model.  Not sure what other options I have to mitigate these discards.  Maybe flexconfig to enable it?

[MHM Cisco World]

let me double check
MHM

[Marius Gunnerud]

Flow Control is only supported on the FTD3100 firewalls

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/720/management-center-device-config-72/interfaces-settings-ifcs-overview.html#ID-2075-00000197

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/release-notes/threat-defense/720/threat-defense-release-notes-72/features.html

[Ruben Cocheno]

@ericdavishop 

Check if your VLANs match from both sides, and in case of Drops check "Modify Buffers to Resolve Output Drops" on document below

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-switch/216236-troubleshoot-output-drops-on-catalyst-90.html#toc-hId-1524744596