Buy or Renew
Buy or Renew
Duo Security forums now LIVE! Get answers to all your Duo Security questions. Learn more
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
6689
Views
15
Helpful
7
Replies

ASA input errors/overruns

cshannahan
Level 1
Level 1

‎08-27-2018 06:53 AM - edited ‎02-21-2020 08:09 AM

Hello, we're seeing input errors and overruns on mainly 2 interfaces that are apart of a port channel on our ASA.  Our load balancing on this device is Source and Destination IP.  I'm wondering if I should change this as when we do backups these interfaces get used very heavily over the others.

 

Any advice would be appreciated.

 

These were just cleared. I'm going to see if this still happens after the backups are done.

The firewall is so busy it hardly can send anything to the SFR module it seems.

 

 

Interface GigabitEthernet0/1 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel10
MAC address 58ac.785c.4544, MTU not set
IP address unassigned
36586565 packets input, 49485809904 bytes, 0 no buffer
Received 543 broadcasts, 0 runts, 0 giants
766 input errors, 0 CRC, 0 frame, 766 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
3068188 packets output, 1713735889 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (469/311)
output queue (blocks free curr/low): hardware (511/0)

 

FW1/pri/act# show processes cpu-hog
Hardware: ASA5555
Cisco Adaptive Security Appliance Software Version 9.6(3)9
ASLR enabled, text region 7f240107c000-7f24050be4fc

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:39:50 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d050ace0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:39:59 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d9b8daf0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:40:22 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d9fabcf0

 

Process: ssh_init, NUMHOG: 2, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:40:22 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d3baddf0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:40:23 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d1833140

 

Process: ssh_init, NUMHOG: 2, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:40:49 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23da2118b0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:40:50 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d8e7c190

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:41:01 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23f116d0a0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:41:22 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23fa8a7990

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:41:22 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23e33b1530

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:41:22 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23dab5fe90

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:41:23 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23cfd7a610

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:41:45 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d3387f90

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:41:49 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23ee5480b0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:41:50 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23df2a7000

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:07 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23cfbdbcd0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:08 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d2658200

 

Process: ssh_init, NUMHOG: 2, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:08 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d847df10

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:12 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d8d347b0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:22 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23f2ad38b0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:22 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23f5cb7cb0

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:22 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23efb98730

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:23 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d923ab40

 

Process: ssh, PROC_PC_TOTAL: 4, MAXHOG: 1, LASTHOG: 1
LASTHOG At: 10:42:29 ADT Aug 27 2018
PC: 0x00007f240335ea71 (suspend)

Process: ssh, NUMHOG: 4, MAXHOG: 1, LASTHOG: 1
LASTHOG At: 10:42:29 ADT Aug 27 2018
PC: 0x00007f240335ea71 (suspend)
Call stack: 0x00007f2401950c83


Process: Event mib process, PROC_PC_TOTAL: 4, MAXHOG: 23, LASTHOG: 23
LASTHOG At: 10:42:37 ADT Aug 27 2018
PC: 0x00007f2403286e52 (suspend)

Process: Event mib process, NUMHOG: 4, MAXHOG: 23, LASTHOG: 23
LASTHOG At: 10:42:37 ADT Aug 27 2018
PC: 0x00007f2403286e52 (suspend)
Call stack: 0x00007f2403286e52 0x00007f240328745a 0x00007f2401950c4b

 

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:49 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23dca907d0

 

Process: ssh_init, PROC_PC_TOTAL: 36, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:50 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)

Process: ssh_init, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3
LASTHOG At: 10:42:50 ADT Aug 27 2018
PC: 0x00007f2401adab79 (suspend)
Call stack: 0x00007f2401adab79 0x00007f2403c98549 0x00007f23d3ed5f70

 

Process: DATAPATH-0-2350, PROC_PC_TOTAL: 151, MAXHOG: 4, LASTHOG: 1
LASTHOG At: 10:43:01 ADT Aug 27 2018
PC: 0x0000000000000000 (suspend)

Process: DATAPATH-0-2350, NUMHOG: 140, MAXHOG: 4, LASTHOG: 1
LASTHOG At: 10:43:01 ADT Aug 27 2018
PC: 0x0000000000000000 (suspend)
Call stack: 0x00007f2401938681 0x00007f2401cd3c60 0x00007f24030616c1
0x00007f240306c76c 0x00007f240050c201


Process: DATAPATH-1-2351, PROC_PC_TOTAL: 146, MAXHOG: 4, LASTHOG: 1
LASTHOG At: 10:43:07 ADT Aug 27 2018
PC: 0x0000000000000000 (suspend)

Process: DATAPATH-1-2351, NUMHOG: 128, MAXHOG: 4, LASTHOG: 1
LASTHOG At: 10:43:07 ADT Aug 27 2018
PC: 0x0000000000000000 (suspend)
Call stack: 0x00007f2401938681 0x00007f2401cd3c60 0x00007f24030616c1
0x00007f240306c76c 0x00007f240050c201


CPU hog threshold (msec): 1.542
Last cleared: 10:39:13 ADT Aug 27 2018

0 Helpful
7 Replies 7

Ajay Saini
Level 7
Level 7

‎08-27-2018 11:11 PM

Hello,

 

The catch in these situations is that even if you change the load balance algorithm, the same interface might be used since the backup might use same ip and mac combination. Usually in these cases, recommendation is to use higher throughput interfaces (10G and higher). Preferably this traffic should be bypassed from SFR inspection and if possible should not come to ASA at all. This backup traffic better be handled by Core Switches with Jumbo frame capability and 10G/40G capacity.

 

But if traffic has to come to ASA, then we can try few combinations of load balancing criteria:

 

https://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/interface-echannel.pdf

 

Regards,

 

Ajay

 

cshannahan
Level 1
Level 1
In response to Ajay Saini

‎08-28-2018 04:41 AM

Backups came into play after things were purchased. Backups usually run at 400MB to 500MB/sec. The firewall is directly connected to Nexus 56128P Chassis with 4 or so GB interfaces in an etherchannel. I can bump that to 6 if needed.  When it was build 2 were made has standby's.

 

I plan on excluding the backup traffic in the ACL used to send to SFR tomorrow.  The SFR is setup in monitor only.

 

When backups aren't occurring the overruns are minimal but do occur from time to time.  I'm not sure why they do though, traffic isn't that high per interface and the CPU is under 50%.

 

Should I turn on flow control? 

0 Helpful

Ajay Saini
Level 7
Level 7
In response to cshannahan

‎08-28-2018 07:42 AM

Hello,

 

Sorry I should have mentioned that earlier. Enabling flow control should be a good option to tackle these kind of issues. The issue that you have at hand seems to be occuring due to burst of traffic rather than the overall bandwidth consumption.

 

For sure, remove this traffic from SFR inspection and enable flow control. That should help.

 

HTH

AJ

cshannahan
Level 1
Level 1
In response to Ajay Saini

‎08-28-2018 07:49 AM

It definitely seems like when the backup traffic stops the overruns and input errors stop as well, I had a few throughout the night but nothing like normal.

 

I'm thinking about adding my 2 standby interfaces to the etherchannel and maybe changing the load balancing to src port or something.

 

Also I will turn flow control on the ASA, I'm not sure if I need to turn it on the Nexus switch, I don't think I do.

SFR exclusions will start tomorrow.

0 Helpful

cshannahan
Level 1
Level 1
In response to cshannahan

‎08-28-2018 09:41 AM

Also, wouldn't I need input flow control, not output? ASA can only do output and the switch would need to do input? To the switch everything seems fine.

Interface GigabitEthernet0/1 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel10
MAC address 58ac.785c.4544, MTU not set
IP address unassigned
1911342418 packets input, 2399267253493 bytes, 0 no buffer
Received 84922 broadcasts, 0 runts, 0 giants
128633 input errors, 0 CRC, 0 frame, 128633 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
301390891 packets output, 183721244547 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (479/311)
output queue (blocks free curr/low): hardware (511/0)
0 Helpful

Ajay Saini
Level 7
Level 7
In response to cshannahan

‎08-28-2018 10:49 AM

Hello,

 

Overruns indicate that the device was unable to accommodate the frames and hence the frames are discarded as overruns. ASA needs to send the flow control pause frames out to adjacent device and hence needs out flow control.

Nexus switch interface would need this in in direction so that it can honor the pause frames sent by ASA and slow down as and when required. So, for Nexus, it needs to be enabled in IN direction.

 

HTH
AJ

cshannahan
Level 1
Level 1
In response to Ajay Saini

‎09-10-2018 05:54 AM

This is the response I got from Cisco on the errors after troubleshooting.

 

First (Tests done between 8AM-11AM, 500MB concurrent traffic)

Action Plan/troubleshooting Done:

 

  • Informed that getting overruns when the backups are running.
  • When checked we had a few thousand overruns while the backup was running.
  • We cleared the interface counters and found only a few hundred overruns which were less than 1% of traffic generated on the interface. This is within specs.
  • Will calculate overruns in peak time and update.
  • Will also check the device throughput.
  • Analyzed the "show tech" unable to see any issues.

 

Second (Tests done at 4am-6am, 750MB concurrent traffic)

 

Overruns which are there are due to  Intermittent Packet Bursts Oversubscribe the ASA Interface FIFO Queue.

Here, the overruns are 0.02-0.66 % which is an acceptable limit as it will not cause any network issues at your end and can be ignored. However, if you want to enable flow control – on ASA interface and the next hop that can be done as well. But in this case that is not required as this will not create any issues.

Also, CPU hogs seems to be in acceptable limit as well.

 

 

So I can turn it on if I want but they said it's not needed and within limits.  Once the backups finish at 11 the errors and traffic drops to minimal.

0 Helpful
Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card