ASA 5505 Performance Problems

jrichterkessing · ‎06-28-2011

I have a Cisco ASA 5505 that is having some fairly serious performance problems when over 3Mb of traffic is going through it....anything below seems to be fine, but once I approach 3Mb I see significant slowness through the ASA to servers on the other side and my SSH session to the box gets really sluggish (i.e. significant dealy when typing and actually seeing characters appear in the SSH window).

Setup: Ethernet0/0 connected to a MPLS connection (30Mb bandwidth), our handoff from the MPLS vendor is a 100Mb ethernet connection. The MPLS service is a transparent L2 connection so we trunk (Dot1Q) three vlans over this connection. The other side (E0/5) is connected to a business partners Cisco 6509 via a 100Mb connection. The ASA has 512MB of memory running 8.3(1), during our most recent performance episode I checked and the ASA was running 16-17% CPU utilization and has 340MB of memory free.

I ran a test offhours and here is what I found:

I had traces (using PingPlotter utility) running to both the ASA VLAN address that is connected to the MPLS circuit, and a seperate trace running to a server at our business partner.....each had the same symptom. I have a utility that I can send specific amount of data accross the wire and ramp up the bandwidth utilization, at 2Mb utilization latency to both addresses is in the 33ms range almost a straight line graph, at 3Mb latency ranges from 33 to 85ms and the graphs becomes very erratic....alot of variation. Also, at 3Mb I start seeing several drops to both addresses.....I thought this may be just because I was using ICMP, so I changed my ping to use TCP 3389 (RDP) and had the same affect.

I have hammered on my vendor who can't find any issue on their side, I have looked over my config and I am not coming up with much......the only thing I can find are some drops on the VLAN interface and ingress policy drops on the ethernet interfaces. Any help would be GREATLY appreciated!

I'll paste my config below and also the interface stats showing the drops I had mentioned:

ASA Version 8.3(1)

!

terminal width 511

hostname ASA1

domain-name ******.com

enable password ****** encrypted

passwd ****** encrypted

names

!

interface Vlan1

nameif WAN

security-level 100

ip address 192.168.1.202 255.255.255.0

!

interface Vlan901

nameif WANdata

security-level 100

ip address 10.52.250.202 255.255.255.0

!

interface Vlan909

nameif Management

security-level 100

ip address 10.52.251.202 255.255.255.0

management-only

!

interface Vlan999

nameif

security-level 10

ip address 10.58.56.2 255.255.255.252

!

interface Vlan1009

nameif Unused

security-level 0

no ip address

!

interface Ethernet0/0

switchport trunk allowed vlan 1,901,909

switchport mode trunk

speed 100

duplex full

!

interface Ethernet0/1

switchport access vlan 1009

shutdown

!

interface Ethernet0/2

switchport access vlan 1009

shutdown

!

interface Ethernet0/3

switchport access vlan 1009

shutdown

!

interface Ethernet0/4

switchport access vlan 1009

shutdown

!

interface Ethernet0/5

switchport access vlan 999

speed 100

duplex full

!

interface Ethernet0/6

switchport access vlan 1009

shutdown

!

interface Ethernet0/7

switchport access vlan 1009

shutdown

!

ftp mode passive

clock timezone EST -5

clock summer-time EDT recurring

dns server-group DefaultDNS

domain-name ******.com

object network ClientLAN

subnet 10.52.0.0 255.255.0.0

object network DataCenterLAN

subnet 10.32.0.0 255.255.0.0

object network LoadBalancing

subnet 10.63.0.0 255.255.0.0

object network DC

subnet 10.40.0.0 255.255.0.0

object network Cleveland

subnet 10.58.0.0 255.255.0.0

object network Server192.168.1.104

host 192.168.1.104

object network Server192.168.1.64

host 192.168.1.64

object network Server192.168.1.88

host 192.168.1.88

object network Citrix

subnet 167.19.238.0 255.255.255.0

object network Server192.168.1.206

host 192.168.1.206

object network Server192.168.1.41

host 192.168.1.41

object-group network DM_INLINE_NETWORK_1

network-object 10.32.0.0 255.255.0.0

network-object 10.40.0.0 255.255.0.0

network-object 10.52.0.0 255.255.0.0

network-object 10.63.0.0 255.255.0.0

network-object object _Citrix

object-group network DM_INLINE_NETWORK_2

network-object 10.58.0.0 255.255.0.0

network-object host 192.168.1.104

network-object host 192.168.1.64

network-object host 192.168.1.88

network-object object Server192.168.1.206

network-object object Server192.168.1.41

access-list _access_in extended permit ip object-group DM_INLINE_NETWORK_2 object-group DM_INLINE_NETWORK_1

access-list _access_in extended permit icmp any any

access-list _access_in extended permit ip 10.58.0.0 255.255.0.0 10.58.56.0 255.255.255.252

access-list _access_in extended deny ip any any

access-list eigrpACL_FR_1 standard permit 10.52.0.0 255.255.0.0

access-list eigrpACL_FR_1 standard permit 10.32.0.0 255.255.0.0

access-list eigrpACL_FR_1 standard permit 10.40.0.0 255.255.0.0

access-list eigrpACL_FR_1 standard permit 10.63.132.0 255.255.255.0

access-list eigrpACL_FR_1 standard permit 10.63.140.0 255.255.255.0

access-list eigrpACL_FR_1 standard permit 10.63.40.0 255.255.255.0

access-list eigrpACL_FR_1 standard permit 167.19.238.0 255.255.255.0

access-list eigrpACL_FR_1 standard deny any

access-list eigrpACL_FR_2 standard deny 10.58.56.0 255.255.255.252

access-list eigrpACL_FR_2 standard permit any

access-list eigrpACL_FR_3 standard permit 10.58.0.0 255.255.0.0

access-list eigrpACL_FR_3 standard permit host 192.168.1.64

access-list eigrpACL_FR_3 standard permit host 192.168.1.88

access-list eigrpACL_FR_3 standard permit host 192.168.1.104

access-list eigrpACL_FR_3 standard permit host 192.168.1.41

access-list eigrpACL_FR_3 standard permit host 192.168.1.206

access-list eigrpACL_FR_3 standard deny any

pager lines 24

logging enable

logging timestamp

logging buffer-size 50000

logging buffered debugging

logging trap notifications

logging asdm informational

logging host Management 10.32.100.206

logging host Management 10.32.100.22

mtu WAN 1500

mtu WANdata 1500

mtu Management 1500

mtu 1500

mtu Unused 1500

no failover

icmp unreachable rate-limit 1 burst-size 1

no asdm history enable

arp timeout 14400

access-group _access_in in interface

!

router eigrp 1

no auto-summary

distribute-list eigrpACL_FR_2 out interface WANdata

distribute-list eigrpACL_FR_1 out interface

distribute-list eigrpACL_FR_3 in interface

network 10.52.250.0 255.255.255.0

network 10.58.56.0 255.255.255.252

passive-interface WAN

passive-interface Management

passive-interface Unused

!

route Management 10.32.100.22 255.255.255.255 10.52.251.1 1

route Management 10.32.100.206 255.255.255.255 10.52.251.1 1

route Management 10.52.132.11 255.255.255.255 10.52.251.4 1

route 192.168.1.41 255.255.255.255 10.58.56.1 1

route 192.168.1.206 255.255.255.255 10.58.56.1 1

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

dynamic-access-policy-record DfltAccessPolicy

aaa authentication http console LOCAL

aaa authentication ssh console LOCAL

http server enable

http 10.52.132.0 255.255.255.0 Management

http 10.32.100.206 255.255.255.255 Management

http 192.168.1.0 255.255.255.0 WAN

snmp-server host Management 10.32.100.206 community ****** version 2c

snmp-server contact Network Admin

snmp-server community ******

snmp-server enable traps snmp authentication linkup linkdown coldstart

crypto ipsec security-association lifetime seconds 28800

crypto ipsec security-association lifetime kilobytes 4608000

crypto ca trustpoint ASDM_TrustPoint0

enrollment self

subject-name CN=ASA1

crl configure

telnet timeout 5

ssh 192.168.1.0 255.255.255.0 WAN

ssh 10.52.132.0 255.255.255.0 Management

ssh 10.32.100.206 255.255.255.255 Management

ssh 10.52.72.0 255.255.255.0 Management

ssh 10.58.0.0 255.255.0.0

ssh timeout 5

console timeout 0

threat-detection basic-threat

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

ntp server 10.52.251.1 source Management prefer

ntp server 10.52.251.4 source Management

webvpn

username **** password **** encrypted privilege 15

!

class-map inspection_default

match default-inspection-traffic

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect ip-options

inspect netbios

inspect rsh

inspect rtsp

inspect skinny

inspect esmtp

inspect sqlnet

inspect sunrpc

inspect tftp

inspect sip

inspect xdmcp

!

service-policy global_policy global

smtp-server 10.63.40.26

prompt hostname context

call-home

profile CiscoTAC-1

no active

destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService

destination address email callhome@cisco.com

destination transport-method http

subscribe-to-alert-group diagnostic

subscribe-to-alert-group environment

subscribe-to-alert-group inventory periodic monthly

subscribe-to-alert-group configuration periodic monthly

subscribe-to-alert-group telemetry periodic daily

Cryptochecksum:3f0dedfca44aece023efa74e3a562f67

: end

VLAN FOR DATA ACCROSS MPLS

sh int vlan 901

Interface Vlan901 "WANdata", is up, line protocol is up

Hardware is EtherSVI, BW 100 Mbps, DLY 100 usec

MAC address f866.f26b.55f3, MTU 1500

IP address 10.52.250.202, subnet mask 255.255.255.0

Traffic Statistics for "WANdata":

76828506 packets input, 29972111977 bytes

87835577 packets output, 62993248308 bytes

197867 packets dropped

1 minute input rate 457 pkts/sec, 179650 bytes/sec

1 minute output rate 569 pkts/sec, 470467 bytes/sec

1 minute drop rate, 1 pkts/sec

5 minute input rate 312 pkts/sec, 111139 bytes/sec

5 minute output rate 359 pkts/sec, 253573 bytes/sec

5 minute drop rate, 1 pkts/sec

INTERFACE TO MPLS

Interface Ethernet0/0 "", is up, line protocol is up

Hardware is 88E6095, BW 100 Mbps, DLY 100 usec

Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)

Input flow control is unsupported, output flow control is unsupported

Available but not configured via nameif

MAC address f866.f26b.55eb, MTU not set

IP address unassigned

85916459 packets input, 32748513505 bytes, 0 no buffer

Received 395339 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 L2 decode drops

3386809 switch ingress policy drops

98805192 packets output, 67716035616 bytes, 0 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 0 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops

0 rate limit drops

0 switch egress policy drops

INTERFACE TO BUSINESS PARTNER

Interface Ethernet0/5 "", is up, line protocol is up

Hardware is 88E6095, BW 100 Mbps, DLY 100 usec

Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)

Input flow control is unsupported, output flow control is unsupported

Available but not configured via nameif

MAC address f866.f26b.55f0, MTU not set

IP address unassigned

87627440 packets input, 64497857446 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 L2 decode drops

282955 switch ingress policy drops

74918220 packets output, 31251776531 bytes, 0 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 0 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops

0 rate limit drops

0 switch egress policy drops

Maykol Rojas · ‎06-28-2011

Would you please do a file transfer, gather the show traffic while doing it and get the captures on the ingress an egress of the firewall to compare the average packets per second on both interfaces?

Mike

jrichterkessing · ‎06-29-2011

Thanks Mike......I will have to wait until tonight to do this since I don't want to cause our users any grief during the day. I will run the capture and show traffic tonight and post the results after.

Thanks for your help....Jeff