Packet Drops on the Network

[dannyh]

Hello,

So some time ago we started to notice something on our network. When downloading a large file, the download speed will gradually slow down and hit 0 and says "network failure." We managed to isolate the issue with the ISP (their devices were misconfigured) but now it's fixed according to them. So we checked the download straight from their router vs from a switch on our network. No more issues when downloading straight from their router, but when we are downloading through a switch on the network, we are experiencing the download issue. Everything else works fine. It's only when downloading a file that is large that this occurs.

We checked wireshark and when the download is happening, there is a lot of packet drops occuring.

I did some research and tried to apply what I found that I thought might be the issue, but nothing is working. I think the problem is between our Core Switch and our ASA (connection is CoreSW > ASA > ISP Router).

I applied speed auto to the uplink interface on the Core SW, but no change.

I looked through show interface command but didn't see any errors on the display.

We recently did a large IOS Update on all our devices. However we didn't update the ASA yet. I was thinking that may be the issue. Or it could be a buggy IOS that we installed on our switches and the CoreSW is faulting out. I am quite stuck and I am the only engineer on site where I work and could use some help resolving this.

[balaji.bandi]

ok what is the bandwidth from ISP ?

what ASA code running ? are you inspecting any file transfers here  ? what MTU it is configured ?

what is the switch here model, what IOS code running ? if possible post the below information :

show run interface x/x (where ASA connected)

show interface x/x (where ASA connected)

same information from ASA to Core switch port config, also same ASA to Router connected port.

[dannyh]

ASA: 55160-x

I don't understand what you mean by ASA code. Could you please elaborate? If you're asking if it's in FTD or ASA, it's in ASA.
MTU from what I've checked on 'show run' is 1500 across everything.
Inspection wise, it looks like they are in default mode.

Interface connecting to Core Switch:
show run interface gix/x:
interface GigabitEthernetx/x
nameif <name here>
security-level 100
ip address <ip here>

show interface gix/x:
Interface GigabitEthernetx/x "<name here>", is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address 7070.8b67.cc26, MTU 1500
IP address <ip here>, subnet mask <subnet here>
94620703865 packets input, 22451301487340 bytes, 0 no buffer
Received 11 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
105582949170 packets output, 37085831190819 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 120 output reset drops
input queue (blocks free curr/low): hardware (2043/1820)
output queue (blocks free curr/low): hardware (2047/1544)
Traffic Statistics for "<name here>":
94620360158 packets input, 20718571219308 bytes
105582949170 packets output, 35157775515061 bytes
9500526 packets dropped
1 minute input rate 8790 pkts/sec, 3536644 bytes/sec
1 minute output rate 18981 pkts/sec, 18004728 bytes/sec
1 minute drop rate, 1 pkts/sec
5 minute input rate 8337 pkts/sec, 3260134 bytes/sec
5 minute output rate 21449 pkts/sec, 20797819 bytes/sec
5 minute drop rate, 2 pkts/sec

Interface connecting to Router (the router is not under our control):
show run:
interface gix/x
nameif <name here>
security-level 0
ip address <ip here>

show interface gix/x:
Interface GigabitEthernetx/x "<name here>", is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address 7070.8b67.cc28, MTU 1500
IP address <ip here>, subnet mask <subnet here>
101628054749 packets input, 36275992383802 bytes, 0 no buffer
Received 1123 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
90225520109 packets output, 21541084777082 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 2195 output reset drops
input queue (blocks free curr/low): hardware (1931/1819)
output queue (blocks free curr/low): hardware (2047/1611)
Traffic Statistics for "<name here>":
101620664902 packets input, 34419436346410 bytes
90225520109 packets output, 19887848312659 bytes
432467528 packets dropped
1 minute input rate 12623 pkts/sec, 8823828 bytes/sec
1 minute output rate 7770 pkts/sec, 3658371 bytes/sec
1 minute drop rate, 66 pkts/sec
5 minute input rate 17034 pkts/sec, 15934737 bytes/sec
5 minute output rate 7808 pkts/sec, 3573346 bytes/sec
5 minute drop rate, 39 pkts/sec

As for the Core Switch, we are using a Catalyst 9500 40 port.
show run interface:
interface TenGigabitEthernetx/x/x
description Link_To_ASA
no switchport
ip address <ip here>

show interface tenx/x/x:
TenGigabitEthernetx/x/x is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 802d.bf53.bed0 (bia 802d.bf53.bed0)
Description: Link_To_ASA
Internet address is <ip here>
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 9/255, rxload 25/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:43:30, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/9262/0 (size/max/drops/flushes); Total output drops: 123912
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 100946000 bits/sec, 14977 packets/sec
5 minute output rate 37669000 bits/sec, 8556 packets/sec
22797288796 packets input, 9846494539942 bytes, 0 no buffer
Received 64 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
19569706628 packets output, 5321461645524 bytes, 0 underruns
Output 2 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

As for our ISP, the bandwidth is 1gbps.

[balaji.bandi]

Input queue: 0/375/9262/0 (size/max/drops/flushes); Total output drops: 123912

i see some drops here, what device mode and what IOS Code running ?

[dannyh]

it's running in install mode and the IOS version is 17.06.03.

[balaji.bandi]

it's running in install mode and the IOS version is 17.06.03.

I might have missed what device is this ?

if this cat 9K look input queue drops :

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-switch/216236-troubleshoot-output-drops-on-catalyst-90.html

[dannyh]

Yes this is a Cat 9500-40

[balaji.bandi]

check my advised URL>

[dannyh]

I tried to create the policy-map and the class to apply the buffer command. However, I am getting this 'queue-buffer is not allowed without bandwidth, shape or priority command' when trying to apply the buffer command. I am quite confused on how to control the buffer. I think I am just spiraling into a rabbit hole now because I can't find a document that explains what to do exactly. I'm currently following document after document and reading about QoS but I can't seem to find solutions.

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/115985-asa-overrun-product-tech-note-00.pdf
check the ASA-CoreSW interface do you see overrun? if yes then check above link how you can solve this issue 

[MHM Cisco World]

check link i share this issue you face

[dannyh]

Will do! Thank you for the resource.

[Georg Pauwen]

Hello,

what happens when the client is connected directly to the ASA, does that problem still occur ? Try and set the speed of the interfaces between the Core switch, the ASA, and the router to a fixed value (not to auto)...