Help with underrun between an ASA5540 – ASA-SSM-20.

[albert_coll]

Hello,

I have an ASA5540 running V8.2.1 equipped with an ASA-SSM-20 running IPS-K9-6.2-2.

The ASA internal-data0/0 interface, aimed to communicate with the ASA-SSM-20 Gi0/1 interface exhibits some underrun errors increase.Physical outside and inside gigabit interfaces remain steady.

The ASA redirects all traffic through the SSM for inspection between the inside interface to or from the outside interface, where many IPSec tunnels terminate. I observed that most of the times the internal-data0/0 interface underruns increase precisely when a server located in the remote side of an specific L2L tunnel is transfering large FTP files (several Gb long) to an internal FTP server.

According with the documentation, underruns are the number of times that the transmitter ran faster than the adaptive security appliance could handle .  All this leads me to consider that the ASA is unable to handle bursty traffic to or from the SSM-20 module, thus some kind of QoS should be necessary.

I would really appreciate any idea of what to do to mitigate this effect.

Regards in advance.

Interface Internal-Data0/0 "", is up, line protocol is up
  Hardware is i82547GI rev00, BW 1000 Mbps, DLY 10 usec
        (Full-duplex), (1000 Mbps)
        MAC address 0000.0001.0002, MTU not set
        IP address unassigned
        2488409935 packets input, 1619369509464 bytes, 0 no buffer
        Received 0 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 L2 decode drops
        2488409940 packets output, 1619369474024 bytes, 7719 underruns
        0 output errors, 0 collisions, 0 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (511/486)
        output queue (blocks free curr/low): hardware (511/0

[Kureli Sankar]

Check this out: https://supportforums.cisco.com/docs/DOC-13103

You are correct.

An Overrun is when an incoming (ingress) packet hits the PIX's NIC, and  the rx ring is full.  This is generally caused by elevated CPU, or cpu  hogs or infected hosts.

An Underrun is when part of the packet is in the tx ring, and the driver  starts transmitting it on the wire, but is unable to get the remaining  part of the packet by the time it has finished transmitting the first  part.

-KS

[albert_coll]

Thank you for your answer.

The ASA is an 5540 which includes 4 embedded GigabitEthernet only. With the AIP SSM inserted in the slot.

The current configuration diverts all traffic to the SSM. Which is configured as inline, fail-open mode:

access-list IPS extended permit ip any any

!

class-map class_IPS

match access-list IPS

!

policy-map global_policy

class class_IPS

  ips inline fail-open sensor vs0

I will take some steps like changing from inline to promiscuous (as it has less impact over traffic throughtpur according with the documentation), or streamlining the ACL to exclude some high throughput flows which are known to be secure, to alleviate the traffic between ASA and SSM.

[Kureli Sankar]

Sounds like a plan. Let us know if reducing the traffic seen by the IPS eliminates the underrun errors.

-KS

[albert_coll]

Thanks Poonguzhali. I'll try it along the next few weeks

[usman ali dar]

Hello Poonguzhali,

I was searching and got this response, currently I am having some issues with my ASA ASA5545 WITH 9.1.1 Version.

it started with a application issue that stops in middle of the operations and then no body can log into the application server and after sometime it start working again and then stops again after some time.

I figure out the switch it was connected was generating a lot errors on trunk so I moved the connections to powerfull nexus switch.

I am currently seeing this errors and unable to understand them completely I am requiring you help please

Interface GigabitEthernet0/0 "inside", is up, line protocol is up
  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
        Input flow control is unsupported, output flow control is off
        MAC address 0.0.0.0.0.0.0, MTU 1500
        IP address 192.168.1.1, subnet mask 255.255.255.0
        4427403602 packets input, 2151227118434 bytes, 0 no buffer
        Received 42396148 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        0 L2 decode drops
        5719508097 packets output, 5277823402324 bytes, 237 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 4 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (477/388)
        output queue (blocks free curr/low): hardware (479/71)
  Traffic Statistics for "inside":
        3893211898 packets input, 1264275976406 bytes
        5719508357 packets output, 5171974072466 bytes
        43272956 packets dropped
      1 minute input rate 5528 pkts/sec,  685081 bytes/sec
      1 minute output rate 14596 pkts/sec,  18382094 bytes/sec
      1 minute drop rate, 23 pkts/sec
      5 minute input rate 6259 pkts/sec,  799966 bytes/sec
      5 minute output rate 14373 pkts/sec,  17747799 bytes/sec
      5 minute drop rate, 24 pkts/sec
==================================================================

Interface GigabitEthernet0/7 "outside", is up, line protocol is up
  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
        Input flow control is unsupported, output flow control is off
        MAC address aaaaaaaaaaaaaaa, MTU 1500
        IP address 192.1.1.1, subnet mask 255.255.255.0
        36328017115 packets input, 18021883345657 bytes, 0 no buffer
        Received 164971366 broadcasts, 0 runts, 0 giants
        31781 input errors, 0 CRC, 0 frame, 31781 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        0 L2 decode drops
        39369900910 packets output, 29835988021321 bytes, 0 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 4 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 79 output reset drops
        input queue (blocks free curr/low): hardware (476/366)
        output queue (blocks free curr/low): hardware (456/378)
  Traffic Statistics for "outside":
        36327768794 packets input, 17325171050057 bytes
        39369901040 packets output, 29101930308442 bytes
        346411335 packets dropped
      1 minute input rate 22754 pkts/sec,  29334277 bytes/sec
      1 minute output rate 7454 pkts/sec,  1110781 bytes/sec
      1 minute drop rate, 46 pkts/sec
      5 minute input rate 16126 pkts/sec,  20051269 bytes/sec
      5 minute output rate 6501 pkts/sec,  1045907 bytes/sec
      5 minute drop rate, 46 pkts/sec
 

Regards

[ArchiTech89]

Lots of overruns shown in the output could indicate a large number of small packets (too few cores to handle them) and perhaps some CPU hog or other. Use show traffic and find out the average size of bytes per packets, and use sh processes cpu-hog to check the latter. Maybe that'll steer you in the right direction?

Oops. Saw the traffic at the end. Packets are averaging around 470 bytes per. That might be the reason for the overruns.

Is this a multi-core device? Also, what's the uptime on this device?

[usman ali dar]

Jeremy thank you for your support the device stays up and running with all small process without any issues but only communication that is disturbed is application that is used client and server communications from inside to outside. 

its a multicore device but i always see single core 0 is being used all the the time i do see cpu-hog data path every day.

all the internet traffic is good but only these three applications that connects with clients got kicked out any time of the day and then i have to wait for 30 mins or 1-3 hrs to logg back in

i also see the multi-session is allowed on xlate not per-session would that make any difference

if i enable ASP LOADBALANCE on asa would that make any difference 

regards