Bad performance on routing between two vlan on ACE 4710 routing

lucas restrepo · ‎08-01-2011

HI,

The topology I am working on is the connection of a web server running PHP to the internet using a load balancer between the internet and the server farm.

The server farm is connected to vlan 50 and the database servers are connected to vlan 60. Both vlans are connected to the load balancer ACE 4710.

What I can see is that the traffic between the two vlans is very slow. Some times the application can not even connect to the database and if I start a ssh to one of those servers the connection is very slow and it takes almost two minutes for the system to prompt me for the password.

I don’t know why this system is performing this way since I have all the configuration wright and I have a permit all ACL on both interfaces.

The other thing that a can see is that the ACE is holding back connections from the Internet to the server farm and that makes that connection slow too.

What can I do and how can i approach this issue.

Best regards.

Lucas.

Ahmad Basel Jaber · ‎08-01-2011

Hi Lucas,

As you may already know using the ACE to route the traffic between VLANs is not a good idea considering the fact that the ACE is meant to load balancing the traffic.

The ACE has muliple performance limitation; bandwidth, concurrent connection, sticky .. etc.

I would advice you to start with defining the issue more:

- How often are you facing this performance issue? Are you facing this issue under any specific condition?

- Have you noticed this issue newly or it has been effecting the setup since day one? If No, what have been changed recently which triggered the issue?

- Are you facing this performance issue for all kind of traffics going through the ACE or only between these two VLANs?

Second, monitoring the ACE resource usage while facing the issue:

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_(ACE)_Troubleshooting_Guide_--_Troubleshooting_Performance_Issues

Then, try to to capture the traffic on the source PC, destination PC and ACE simultaneously, then start comparing the time-stamp on wireshark to understand the source of the delay.

Best regards,

Ahmad

lucas restrepo · ‎08-03-2011

here is the configuration i have layed out for this load balancer.

i have motied some configuration like an access list for the vip and the external ip but all the interesting traffic is permited.

access-list ALL line 8 extended permit ip any any

access-list captura line 2 extended permit ip host 10.10.10.7 any

access-list captura line 3 extended permit ip any host 10.10.10.7

probe icmp pingeuropa

interval 10

faildetect 5

passdetect interval 60

probe http web-server

interval 5

passdetect interval 5

expect status 200 399

open 1

probe https web-server-europa

interval 5

passdetect interval 5

ssl version all

expect status 200 399

open 1

probe http web-server-portal

interval 5

passdetect interval 5

expect status 200 399

open 1

parameter-map type connection test

rserver host web-app

ip address 10.10.10.5

inservice

rserver host web-app-portal

ip address 10.10.10.6

inservice

serverfarm host Balanceo_WEB_APP

predictor leastconns

probe pingeuropa

rserver web-app

inservice

serverfarm host Balanceo_WEB_APP_https

predictor leastconns

probe pingeuropa

rserver web-app

inservice

serverfarm host Balanceo_WEB_APP_portal

predictor leastconns

probe pingeuropa

rserver web-app-portal

inservice

sticky ip-netmask 255.255.255.255 address source Balanceo-web-server-STICKY

timeout 14400

replicate sticky

serverfarm Balanceo_WEB_APP

sticky ip-netmask 255.255.255.255 address source Balanceo-web-server-portal-STICKY

timeout 14400

replicate sticky

serverfarm Balanceo_WEB_APP_portal

sticky ip-netmask 255.255.255.255 address source Balanceo-web-server-europa-STICKY

timeout 14400

replicate sticky

serverfarm Balanceo_WEB_APP_https

class-map match-any Balanceo-WEB-europa

2 match virtual-address 200.x.x.x tcp eq www

4 match virtual-address 190.x.x.x tcp eq www

class-map match-any Balanceo-WEB-europa-https

2 match virtual-address 200.x.x.x tcp eq https

4 match virtual-address 190.x.x.x tcp eq https

class-map match-any Balanceo-WEB-portales

2 match virtual-address 190.x.x.x tcp eq www

4 match virtual-address 200.x.x.x tcp eq www

class-map match-any TCP_CLASS

2 match any

class-map type management match-any remote_access

2 match protocol xml-https any

3 match protocol icmp any

4 match protocol telnet any

5 match protocol ssh any

6 match protocol http any

7 match protocol https any

8 match protocol snmp any

policy-map type management first-match remote_mgmt_allow_policy

class remote_access

permit

policy-map type loadbalance first-match L7_VIP_LB_WEB

class class-default

sticky-serverfarm Balanceo-web-server-STICKY

policy-map type loadbalance first-match L7_VIP_LB_WEB_europahttps

class class-default

sticky-serverfarm Balanceo-web-server-europa-STICKY

policy-map type loadbalance first-match L7_VIP_LB_WEB_portal

class class-default

sticky-serverfarm Balanceo-web-server-portal-STICKY

policy-map multi-match L4WEB

class Balanceo-WEB-europa

loadbalance vip inservice

loadbalance policy L7_VIP_LB_WEB

loadbalance vip icmp-reply active

class Balanceo-WEB-portales

loadbalance vip inservice

loadbalance policy L7_VIP_LB_WEB_portal

loadbalance vip icmp-reply active

class Balanceo-WEB-europa-https

loadbalance vip inservice

loadbalance policy L7_VIP_LB_WEB_europahttps

loadbalance vip icmp-reply active

policy-map multi-match TCP_POLICY

class TCP_CLASS

connection advanced-options test

interface vlan 10

ip address 200.x.x.x 255.255.255.248

peer ip address 200.x.x.x 255.255.255.248

access-group input web

access-group output web

service-policy input L4WEB

no shutdown

interface vlan 20

ip address 190.y.y.x 255.255.255.248

peer ip address 190.w.y.x 255.255.255.248

access-group input web

access-group output web

no shutdown

interface vlan 50

ip address 10.10.10.2 255.255.255.0

peer ip address 10.10.10.3 255.255.255.0

access-group input ALL

access-group output ALL

service-policy input remote_mgmt_allow_policy

no shutdown

interface vlan 60

ip address 10.10.20.1 255.255.255.0

peer ip address 10.10.20.2 255.255.255.0

access-group input ALL

access-group output ALL

service-policy input remote_mgmt_allow_policy

no shutdown

ip route 0.0.0.0 0.0.0.0 200.13.228.137

ip route 192.168.0.0 255.255.0.0 10.10.10.1

-----------------------------------------

this is the output of the resource.

Allocation

Resource Current Peak Min Max Denied

-------------------------------------------------------------------------------

Context: Aplication

conc-connections 24 2002 800000 1200000 0

mgmt-connections 4 44 40000 60000 0

proxy-connections 0 23 104857 157286 0

xlates 0 0 26214 39321 0

acc-connections 0 0 20 30 0

bandwidth 131252 9356738 53687091 205530637 0

throughput 180 4965826 53687091 80530637 0

mgmt-traffic rate 131072 4390912 0 125000000 0

connection rate 0 257 400000 600000 0

ssl-connections rate 0 1 400 600 0

mac-miss rate 0 88 800 1200 0

inspect-conn rate 0 0 16000 24000 0

http-comp rate 0 0 5242880 7864320 0

acl-memory 49328 76720 15018736 22531277 0

sticky 0 1880 327680 0 0

regexp 914 914 419430 629146 0

syslog buffer 0 0 418816 629760 0

syslog rate 0 0 40000 60000 0

as i have said the connection between the network 10.10.10.0 and the 10.10.20.0 are very slow and the conections fron the internet the load balancer cut allot of connections of the web aplication.

best regards.