Ask the Expert: Configuration and Troubleshooting the Cisco Application Control Engine (ACE) load balancer

[ciscomoderator]

With Ajay Kumar and Telmo Pereira 

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about configuration and troubleshooting the Cisco Application Control Engine (ACE) load balancer with Cisco expert Ajay Kumar and Telmo Pereira. The Cisco ACE Application Control Engine Module for Cisco Catalyst 6500 Series Switches and Cisco 7600 Series Routers is a next-generation load-balancing and application-delivery solution. A member of the Cisco family of Data Center 3.0 solutions, the module: Helps ensure business continuity by increasing application availability Improves business productivity by accelerating application and server performance Reduces data center power, space, and cooling needs through a virtualized architecture Helps lower operational costs associated with application provisioning and scaling

Ajay Kumar  is a customer support engineer in the Cisco Technical Assistance Center in Brussels, covering content delivery network technologies including Cisco Application Control Engine, Cisco Wide Area Application Services, Cisco Content Switching Module, Cisco Content Services Switches, and others. He has been with Cisco for more than four years, working with major customers to help resolve their issues related to content products. He holds DCASI and VCP certifications. 

Telmo Pereira is a customer support engineer in the Cisco Technical Assistance Center in Brussels, where he covers all Cisco content delivery network technologies including Cisco Application Control Engine (ACE), Cisco Wide Area Application Services (WAAS), and Digital Media Suite. He has worked with multiple customers around the globe, helping them solve interesting and often highly complex issues. Pereira has worked in the networking field for more than 7 years. He holds a computer science degree as well as multiple certifications including CCNP, DCASI, DCUCI, and VCP

Remember to use the rating system to let Ajay know if you have received an adequate response.

Ajay and Telmo might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community discussion forum Application Networking shortly after the event.

This event lasts through July 26, 2013. Visit this forum often to view responses to your questions and the questions of other community members.

[John Ventura]

Hello Experts,

I am planning to upgrade my ACE and would like to know the best practices for the same?

Will there be a downtime or it can be a hitless upgrade. I think it should be simple but need your opinion. I think I can start with upgrade of standby.  If for some reason ACE doesn’t boot up what would be the recovery steps.

Appreciate your quick response.

-John

[ajayku2]

Hi John,

Will there be a downtime or it can be a hitless upgrade.

You dont need downtime to upgrade ACE. It can be a hitless upgrade.

You can follow the procedure as described in link below:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/vA5_1_x/release/note/ACE_mod_rn_A51x.html#wp791479

From the above link :

Note  : Ensure that the preempt command is disabled before the upgrade procedure begins.

It is also true that you first upgrade standby and then Primary. The above mentioned link is the best way to upgrade ACE.

------------------------------------------------------------

If for some reason ACE does not boot up. Below is the recovery procedure:

Usually it get stuck in rommon mode:

You can refer the following link : 

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_(ACE)_Troubleshooting_Guide_--_Troubleshooting_ACE_Boot_Issues

refer topic : " Booting the ACE from the ROMMON Prompt"

In the above you can mention the old image and ACE should boot properly with the old ACE image.

Let me know if that answer your question.

regards,

Ajay Kumar

[John Ventura]

Thanks Ajay.

[Telmo Pereira]

Hello John!!!

Thanks for the first post on this session! Just to add some additional information to what Ajay has shared.

We do have a hitless upgrade, but refers to L4 connections which can be replicated (if you do have connection replication enabled on your system).

However be aware that there will be a hit on L7 connections (SSL offload, TCP Reuse, Inspect, etc). Meaning those connections will have to be reestablished on the secondary ACE.

In that sense, if you do have any contexts with layer 7 configuration, or even if do have only L4, the general recommendation for that matter is to schedule a maintenance window for the upgrade operation.

Also as per best practice we recommend:

1. To disable preemption (as Ajay mentioned) and upgrade the standby box for the Admin context

2. Then you reboot the standby box on the new version of code and you do a failover of the contexts to that box.

You will see how the system behaves. If there are any issues, you can simply fallback to the other ACE that will still be on the old version of code.

3. Assuming everything goes well on 2, you will go ahead and upgrade the other ACE and once if comes back, you can failover the traffic to it again and reenabe preempt.

This is documented, on a step by step basis on the link provided by Ajay.

HTH,

Telmo

[John Ventura]

Thanks Telmo.

[Jing Ren]

Dear AJay and Telmo,

Thanks for your time to look at this. We have a pair of ACE4710 running in active/standby. We've setup a web services to load balance https://www.mydomain.com/project/ to 6 backend servers. The servers could accept uri path of /project/ and /project (with or without a /).

What are the steps to configure on ACE to be able to accept traffic for both / and without /?

my second question is, do I need to configure both access-group and service-policy on all the interfaces to pass the traffic? What are the differences betweeen an access-group and a service-policy?

Thank you in advance,

James Ren

[Telmo Pereira]

Jing,

Thanks for posting!

In order to meet your requirements you would have a similar configuration to the one below:

If the requested URL is /project, then send to the PROJECT-FARM, otherwise send to web farm.

-- CONF snippet --

access-list ANYONE line 10 extended permit tcp any any

rserver host WWW_SERVER_01

  ip address 10.10.10.10

  inservice

rserver host WWW_SERVER_02

  ip address 10.10.10.11

  inservice

rserver host WWW_SERVER_03

  ip address 10.10.10.12

  inservice

rserver host LOGIN_SERVER_04

  ip address 10.10.10.15

  inservice

serverfarm host WWW-FARM

  probe TCP

  rserver WWW_SERVER_01

    inservice

  rserver WWW_SERVER_02

    inservice

  rserver WWW_SERVER_03

    inservice

serverfarm host PROJECT-FARM

  probe TCP

  rserver LOGIN_SERVER_04

    inservice

class-map match-all WWW-VIP

  2 match virtual-address 20.20.20.10 tcp eq www

class-map type http loadbalance match-any CLASS-PROJECT

  2 match http url /project.*

policy-map type loadbalance first-match SLB_LOGIC

  class CLASS-PROJECT

    serverfarm PROJECT-FARM

  class class-default

    serverfarm WWW-FARM

policy-map multi-match CLIENT_VIPS

  class WWW-VIP

    loadbalance vip inservice

    loadbalance policy SLB_LOGIC

    loadbalance vip icmp-reply

interface vlan 10

  description Servers vlan

  access-group input ANYONE

  ip address 10.10.10.5 255.255.255.0

  no shutdown

interface vlan 20

  description Client vlan

  ip address 20.20.20.9 255.255.255.0

  access-group input ANYONE

  service-policy input CLIENT_VIPS

  no shutdown

ip route 0.0.0.0 0.0.0.0 20.20.20.1

For the second query, you just need to be aware that by default ACE will not pass any traffic (will deny everything), unless we permit it.

The access-group will be used to tie the access-list to an interface and this is normally needed on all interfaces if you are going to pass any sort of traffic on them.

The service-policy may or not be needed on all interfaces. In the example I gave you, I only apply service-policy on the client vlan as I will only have hits for the VIP on that side.

If on the other hand, servers could also initiate connections to the VIP, you would have to apply the service policy on that other interface also.

Let me know if this makes things clearer, if not I can provide more details.

HTH,

Telmo

[Jing Ren]

Thanks for your explanation and detailed config sample. I helped a lot for me to understand the concept but it might not be applicable to our environment. We have a multi-tenant load balancing setup and multiple class-maps are applied. Futhermore, class-default has already been used for some other servers, which is a different sticky server farm.

Is it possible to set a uri redirect, similar to an iRule in F5 to force client browser to go to /project/ if they are coming without a / (force /project to /project/) ?

The current setup is

2 match http url /project.*

which works fine for /project/ but it does not respond to /project

Your explanation on the access-group and policy-map is very clear. Thank you!

Best Wishes,

James Ren

[Telmo Pereira]

James,

The pleasure is all mine!

It is possible to configure URL redirection on ACE, but it seems this would purely polute your configuration unnecessarily.

  2 match http url /project.*

Should match both /project and /project/.

Actually it should also match URLs like /project/anythingelse.something

.* is an expression that is supposed to match zero or more characters.

It would come as a suprise if it is not matching the request for /project. Is the request HTTP/1.1? What ACE version do you have?

Anyway, to give you an example of what a redirection would look like on ACE here it goes:

class-map type http loadbalance match-any redirect-l7

  2 match http url /project

policy-map type loadbalance first-match redirect-policy

  class redirect-l7

    serverfarm redirect-sf

serverfarm redirect redirect-sf

  rserver redirect-sf

    inservice

rserver redirect redirect-sf

  webhost-redirection http://%h/project/ 301

  inservice

%h represents the hostname.

This has been documented here:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A2/configuration/slb/guide/rsfarms.html#wp1013201

Still, as mentioned, I don't see a need to apply redirection on this case. If "match http url /project.*" is not working for some reason just let me know and I will advise on next steps.

Regards,

Telmo

[Jing Ren]

Dear Telmo,

Sorry for the late reply. I was caught up with something else.

Your explanation on the match-all regex is what I thought would work. But in fact it didn't work as intended.

I have tested on the a random backend server and they accept with or without /.

I tested on the frontend, / works but without a / it doesn't.

Please refer to the attached screenshots.

Below is the relevant configuration:-

class-map type http loadbalance match-all CLASS_MCTP

  2 match http url /.*

policy-map type loadbalance http first-match APP

  class CLASS_MCTP

    sticky-serverfarm STICKY_MCTP

    action SSL_REWRITE_MCTP

policy-map multi-match POLICY

  class xxxxxxxd11_443_CLASS

loadbalance vip inservice

    loadbalance policy APP

    loadbalance vip icmp-reply active

    appl-parameter http advanced-options CASE_PARAM

    ssl-proxy server xml_xxxx

action-list type modify http SSL_REWRITE_MCTP

  ssl url rewrite location ".*xml\.mydomain\.com.*" clearport 9380

[Telmo Pereira]

Dear James,

Welcome back to the thread.

The issue is actually your configuration, you are matching at least one ´/´ on the URL, if it is not there it will not work, meaning ACE will not match the policy for loadbalancing, and as you don't have a class class-default as fallback it will simply fail.

Can you please try to change the class-map CLASS_MCTP to

class-map type http loadbalance match-all CLASS_MCTP

  2 match http url /mctp.* 

That should do the trick.

If you want something even more generic you can use:

class-map type http loadbalance match-all CLASS_MCTP

  2 match http url .* 

HTH,

Telmo

[ajayku2]

Hi Jing,

Just to add a bit on Telmo's reply.

What are the differences betweeen an access-group and a service-policy?

We apply access group to any interface to define what traffic should be allowed or dropped when hitting a interface. Similar to any firewall access list/access group.

Service policy is like a set of instruction to match the interesting traffic based on defined associated class match and to define how to load balance the traffic. So I would say that load balancing decisions are based on service policy.

regards,

Ajay Kumar

[jeongdae.lee]

Hello, Expert.

In the case that two servers are on the each end sides of firewall (one on DMZ and another on INSIDE), is there a way to load-balance the traffic from one to another server between firewall A and B, using the ACE 4710? Or, there is a way to load-balance the traffic using source port number?

Thank you for reading it.

[ajayku2]

Hello Jeongdae,

Thanks for the question.

As per my understanding you want to load balance in the following way :

Firewall DMZ ( Server in DMZ zone acting as client ) >> Cisco ACE >> Firewall Inside ( Server in INSIDE zone ) 

This is usual load balancing scenerio in routing mode. This can be achieved by simple routing mode config:

Please refer following link for routing mode config.

http://docwiki.cisco.com/wiki/Basic_Load_Balancing_Using_Routed_Mode_on_the_Cisco_Application_Control_Engine_Configuration_Example

It is possible to load balance using source IP address.

Sample config :

class-map match-any TEST-VIP
  3 match virtual-address x.x.x.x eq any

serverfarm SF1

  rserver A
    inservice

  rserver B

    inservice

class-map type http loadbalance match-any SRC-IP-MATCH
  2 match source-address a.a.a.a 255.255.255.255

policy-map type loadbalance first-match Policy1
  class SRC-IP-MATCH
    serverfarm SF1

policy-map multi-match Mpolicy1

  class TEST-VIP

    loadbalance vip inservice
    loadbalance policy Policy1
    loadbalance vip icmp-reply
    loadbalance vip advertise

interface vlan yyy

service-pilicy .....

Let me know if that helps.

regards,

Ajay Kumar