topic Ask the Expert: Configuration and Troubleshooting the Cisco Appl in Application Networking

Ask the Expert: Configuration and Troubleshooting the Cisco Application Control Engine (ACE) load balancer

ciscomoderator — Mon, 15 Jul 2013 08:30:25 GMT

With Ajay Kumar and Telmo Pereira

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about configuration and troubleshooting the Cisco Application Control Engine (ACE) load balancer with Cisco expert Ajay Kumar and Telmo Pereira. The Cisco ACE Application Control Engine Module for Cisco Catalyst 6500 Series Switches and Cisco 7600 Series Routers is a next-generation load-balancing and application-delivery solution. A member of the Cisco family of Data Center 3.0 solutions, the module: Helps ensure business continuity by increasing application availability Improves business productivity by accelerating application and server performance Reduces data center power, space, and cooling needs through a virtualized architecture Helps lower operational costs associated with application provisioning and scaling

Ajay Kumar is a customer support engineer in the Cisco Technical Assistance Center in Brussels, covering content delivery network technologies including Cisco Application Control Engine, Cisco Wide Area Application Services, Cisco Content Switching Module, Cisco Content Services Switches, and others. He has been with Cisco for more than four years, working with major customers to help resolve their issues related to content products. He holds DCASI and VCP certifications.

Telmo Pereira is a customer support engineer in the Cisco Technical Assistance Center in Brussels, where he covers all Cisco content delivery network technologies including Cisco Application Control Engine (ACE), Cisco Wide Area Application Services (WAAS), and Digital Media Suite. He has worked with multiple customers around the globe, helping them solve interesting and often highly complex issues. Pereira has worked in the networking field for more than 7 years. He holds a computer science degree as well as multiple certifications including CCNP, DCASI, DCUCI, and VCP

Remember to use the rating system to let Ajay know if you have received an adequate response.

Ajay and Telmo might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community discussion forum Application Networking shortly after the event.

This event lasts through July 26, 2013. Visit this forum often to view responses to your questions and the questions of other community members.

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

John Ventura — Mon, 22 Jul 2013 04:14:56 GMT

Hello Experts,

I am planning to upgrade my ACE and would like to know the best practices for the same?

Will there be a downtime or it can be a hitless upgrade. I think it should be simple but need your opinion. I think I can start with upgrade of standby. If for some reason ACE doesn’t boot up what would be the recovery steps.

Appreciate your quick response.

-John

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

ajayku2 — Mon, 22 Jul 2013 07:18:23 GMT

Hi John,

Will there be a downtime or it can be a hitless upgrade.

You dont need downtime to upgrade ACE. It can be a hitless upgrade.

You can follow the procedure as described in link below:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/vA5_1_x/release/note/ACE_mod_rn_A51x.html#wp791479

From the above link :

Note : Ensure that the preempt command is disabled before the upgrade procedure begins.

It is also true that you first upgrade standby and then Primary. The above mentioned link is the best way to upgrade ACE.

------------------------------------------------------------

If for some reason ACE does not boot up. Below is the recovery procedure:

Usually it get stuck in rommon mode:

You can refer the following link :

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_(ACE)_Troubleshooting_Guide_--_Troubleshooting_ACE_Boot_Issues

refer topic : " Booting the ACE from the ROMMON Prompt"

In the above you can mention the old image and ACE should boot properly with the old ACE image.

Let me know if that answer your question.

regards,

Ajay Kumar

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Telmo Pereira — Mon, 22 Jul 2013 07:24:51 GMT

Hello John!!!

Thanks for the first post on this session! Just to add some additional information to what Ajay has shared.

We do have a hitless upgrade, but refers to L4 connections which can be replicated (if you do have connection replication enabled on your system).

However be aware that there will be a hit on L7 connections (SSL offload, TCP Reuse, Inspect, etc). Meaning those connections will have to be reestablished on the secondary ACE.

In that sense, if you do have any contexts with layer 7 configuration, or even if do have only L4, the general recommendation for that matter is to schedule a maintenance window for the upgrade operation.

Also as per best practice we recommend:

1. To disable preemption (as Ajay mentioned) and upgrade the standby box for the Admin context

2. Then you reboot the standby box on the new version of code and you do a failover of the contexts to that box.

You will see how the system behaves. If there are any issues, you can simply fallback to the other ACE that will still be on the old version of code.

3. Assuming everything goes well on 2, you will go ahead and upgrade the other ACE and once if comes back, you can failover the traffic to it again and reenabe preempt.

This is documented, on a step by step basis on the link provided by Ajay.

HTH,

Telmo

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

John Ventura — Mon, 22 Jul 2013 09:55:38 GMT

Thanks Ajay.

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

John Ventura — Mon, 22 Jul 2013 09:59:44 GMT

Thanks Telmo.

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Jing Ren — Mon, 22 Jul 2013 11:57:00 GMT

Dear AJay and Telmo,

Thanks for your time to look at this. We have a pair of ACE4710 running in active/standby. We've setup a web services to load balance https://www.mydomain.com/project/ to 6 backend servers. The servers could accept uri path of /project/ and /project (with or without a /).

What are the steps to configure on ACE to be able to accept traffic for both / and without /?

my second question is, do I need to configure both access-group and service-policy on all the interfaces to pass the traffic? What are the differences betweeen an access-group and a service-policy?

Thank you in advance,

James Ren

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Telmo Pereira — Mon, 22 Jul 2013 12:14:02 GMT

Jing,

Thanks for posting!

In order to meet your requirements you would have a similar configuration to the one below:

If the requested URL is /project, then send to the PROJECT-FARM, otherwise send to web farm.

-- CONF snippet --

access-list ANYONE line 10 extended permit tcp any any

rserver host WWW_SERVER_01

ip address 10.10.10.10

inservice

rserver host WWW_SERVER_02

ip address 10.10.10.11

inservice

rserver host WWW_SERVER_03

ip address 10.10.10.12

inservice

rserver host LOGIN_SERVER_04

ip address 10.10.10.15

inservice

serverfarm host WWW-FARM

probe TCP

rserver WWW_SERVER_01

inservice

rserver WWW_SERVER_02

inservice

rserver WWW_SERVER_03

inservice

serverfarm host PROJECT-FARM

probe TCP

rserver LOGIN_SERVER_04

inservice

class-map match-all WWW-VIP

2 match virtual-address 20.20.20.10 tcp eq www

class-map type http loadbalance match-any CLASS-PROJECT

2 match http url /project.*

policy-map type loadbalance first-match SLB_LOGIC

class CLASS-PROJECT

serverfarm PROJECT-FARM

class class-default

serverfarm WWW-FARM

policy-map multi-match CLIENT_VIPS

class WWW-VIP

loadbalance vip inservice

loadbalance policy SLB_LOGIC

loadbalance vip icmp-reply

interface vlan 10

description Servers vlan

access-group input ANYONE

ip address 10.10.10.5 255.255.255.0

no shutdown

interface vlan 20

description Client vlan

ip address 20.20.20.9 255.255.255.0

access-group input ANYONE

service-policy input CLIENT_VIPS

no shutdown

ip route 0.0.0.0 0.0.0.0 20.20.20.1

For the second query, you just need to be aware that by default ACE will not pass any traffic (will deny everything), unless we permit it.

The access-group will be used to tie the access-list to an interface and this is normally needed on all interfaces if you are going to pass any sort of traffic on them.

The service-policy may or not be needed on all interfaces. In the example I gave you, I only apply service-policy on the client vlan as I will only have hits for the VIP on that side.

If on the other hand, servers could also initiate connections to the VIP, you would have to apply the service policy on that other interface also.

Let me know if this makes things clearer, if not I can provide more details.

HTH,

Telmo

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

ajayku2 — Mon, 22 Jul 2013 12:57:31 GMT

Hi Jing,

Just to add a bit on Telmo's reply.

What are the differences betweeen an access-group and a service-policy?

We apply access group to any interface to define what traffic should be allowed or dropped when hitting a interface. Similar to any firewall access list/access group.

Service policy is like a set of instruction to match the interesting traffic based on defined associated class match and to define how to load balance the traffic. So I would say that load balancing decisions are based on service policy.

regards,

Ajay Kumar

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

jeongdae.lee — Tue, 23 Jul 2013 07:48:49 GMT

Hello, Expert.

In the case that two servers are on the each end sides of firewall (one on DMZ and another on INSIDE), is there a way to load-balance the traffic from one to another server between firewall A and B, using the ACE 4710? Or, there is a way to load-balance the traffic using source port number?

Thank you for reading it.

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

ajayku2 — Tue, 23 Jul 2013 11:01:15 GMT

Hello Jeongdae,

Thanks for the question.

As per my understanding you want to load balance in the following way :

Firewall DMZ ( Server in DMZ zone acting as client ) >> Cisco ACE >> Firewall Inside ( Server in INSIDE zone )

This is usual load balancing scenerio in routing mode. This can be achieved by simple routing mode config:

Please refer following link for routing mode config.

http://docwiki.cisco.com/wiki/Basic_Load_Balancing_Using_Routed_Mode_on_the_Cisco_Application_Control_Engine_Configuration_Example

It is possible to load balance using source IP address.

Sample config :

class-map match-any TEST-VIP
3 match virtual-address x.x.x.x eq any

serverfarm SF1

rserver A
inservice

rserver B

inservice

class-map type http loadbalance match-any SRC-IP-MATCH
2 match source-address a.a.a.a 255.255.255.255

policy-map type loadbalance first-match Policy1
class SRC-IP-MATCH
serverfarm SF1

policy-map multi-match Mpolicy1

class TEST-VIP

    loadbalance vip inservice
    loadbalance policy Policy1
    loadbalance vip icmp-reply
    loadbalance vip advertise

interface vlan yyy

service-pilicy .....

Let me know if that helps.

regards,

Ajay Kumar

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Jing Ren — Tue, 23 Jul 2013 15:41:49 GMT

Thanks for your explanation and detailed config sample. I helped a lot for me to understand the concept but it might not be applicable to our environment. We have a multi-tenant load balancing setup and multiple class-maps are applied. Futhermore, class-default has already been used for some other servers, which is a different sticky server farm.

Is it possible to set a uri redirect, similar to an iRule in F5 to force client browser to go to /project/ if they are coming without a / (force /project to /project/) ?

The current setup is

2 match http url /project.*

which works fine for /project/ but it does not respond to /project

Your explanation on the access-group and policy-map is very clear. Thank you!

Best Wishes,

James Ren

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Telmo Pereira — Tue, 23 Jul 2013 19:30:55 GMT

James,

The pleasure is all mine!

It is possible to configure URL redirection on ACE, but it seems this would purely polute your configuration unnecessarily.

2 match http url /project.*

Should match both /project and /project/.

Actually it should also match URLs like /project/anythingelse.something

.* is an expression that is supposed to match zero or more characters.

It would come as a suprise if it is not matching the request for /project. Is the request HTTP/1.1? What ACE version do you have?

Anyway, to give you an example of what a redirection would look like on ACE here it goes:

class-map type http loadbalance match-any redirect-l7

2 match http url /project

policy-map type loadbalance first-match redirect-policy

class redirect-l7

serverfarm redirect-sf

serverfarm redirect redirect-sf

rserver redirect-sf

inservice

rserver redirect redirect-sf

webhost-redirection http://%h/project/ 301

inservice

%h represents the hostname.

This has been documented here:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A2/configuration/slb/guide/rsfarms.html#wp1013201

Still, as mentioned, I don't see a need to apply redirection on this case. If "match http url /project.*" is not working for some reason just let me know and I will advise on next steps.

Regards,

Telmo

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Krzysztof Obara — Tue, 23 Jul 2013 20:25:36 GMT

Dear Experts,

I'd like to ask you about configuration amendments for using sorry serverfarm (sorry page when primary sfarm fails) with regard to this specific example:

rserver host REAL_PRIMARY_1

ip address 10.0.0.1

inservice

rserver host REAL_PRIMARY_2

ip address 10.0.0.2

inservice

rserver host REAL_BACKUP_1

ip address 10.0.0.11

inservice

rserver host REAL_BACKUP_2

ip address 10.0.0.12

inservice

serverfarm host SFARM_PRIMARY

failaction reassign

predictor leastconns slowstart 30

probe PROBE_HTTP

rserver REAL_PRIMARY_1 80

conn-limit max 10000 min 9900

inservice

rserver REAL_PRIMARY_2 80

conn-limit max 10000 min 9900

inservice

serverfarm host SFARM_BACKUP

failaction reassign

predictor leastconns slowstart 30

probe PROBE_HTTP

rserver REAL_BACKUP_1 80

inservice

rserver REAL_BACKUP_2 80

inservice

sticky ip-netmask 255.255.255.255 address source STICKY_PRIMARY

timeout 120

serverfarm SFARM_PRIMARY backup SFARM_BACKUP

sticky http-cookie BACKUP_ID STICKY_BACKUP

cookie insert

timeout 60

serverfarm SFARM_BACKUP

action-list type modify http ACT_LIST_RW

ssl url rewrite location ".*"

policy-map type loadbalance first-match PM_L7_PRIMARY

class class-default

sticky-serverfarm STICKY_PRIMARY

action ACT_LIST_RW

policy-map type loadbalance first-match PM_L7_BACKUP

class class-default

sticky-serverfarm STICKY_BACKUP

class-map match-all CM_L3L4_PRIMARY

10 match virtual-address 10.0.0.100 tcp eq https

class-map match-all CM_L3L4_BACKUP

10 match virtual-address 10.0.0.200 tcp eq www

parameter-map type connection PARA_MAP_CONN_TIMEOUT_120

set timeout inactivity 120

parameter-map type http PARA_MAP_PERSIST_REBAL

persistence-rebalance

header modify per-request

policy-map multi-match PM_L3L4_PRIMARY

class CM_L3L4_PRIMARY

loadbalance vip inservice

loadbalance policy PM_L7_PRIMARY

loadbalance vip icmp-reply active

appl-parameter http advanced-options PARA_MAP_PERSIST_REBAL

ssl-proxy server SSL_PROXY_PRIMARY

connection advanced-options PARA_MAP_CONN_TIMEOUT_120

policy-map multi-match PM_L3L4_BACKUP

class CM_L3L4_BACKUP

loadbalance vip inservice

loadbalance policy PM_L7_BACKUP

loadbalance vip icmp-reply active

appl-parameter http advanced-options PARA_MAP_PERSIST_REBAL

interface vlan 100

description Client-side VLAN

bridge-group 100

mac-sticky enable

access-group input ACL_ALLOW_BPDU

access-group input ACL_ALL_IP

access-group output ACL_ALL_IP

service-policy input PM_L3L4_PRIMARY

service-policy input PM_L3L4_BACKUP

no shutdown

interface vlan 200

description Server-side VLAN

bridge-group 100

mac-sticky enable

access-group input ACL_ALLOW_BPDU

access-group input ACL_ALL_IP

access-group output ACL_ALL_IP

no shutdown

interface bvi 100

ip address 10.0.0.252 255.255.255.0

alias 10.0.0.254 255.255.255.0

peer ip address 10.0.0.253 255.255.255.0

no shutdown

My questions for the above example:

1) After making the first test, it seems that users who want to connect to a failed primary page, then they see the sorry page. However, after the primary sfarm is up, the users still see sorry page. What can cause that behaviour? Should I remove http cookie for the backup sfarm?

2) How will failaction reassign work, when the primary sfarm will go down reaching the MAXCONN state?

3) How will the ACE behave, when backup sfarm is configured under sticky of primary sfarm? Connections only for backup sfarm will be added into the sticky database?

4) Is there something else to configure for sorry sfarm if the requirement is to configure backup sfarm as host not as a redirect sfarm (from cisco.com - there is an example for configuring sorry sfarm as redirect)?

Kind of regards,

Krzysztof

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Telmo Pereira — Tue, 23 Jul 2013 21:55:05 GMT

Krzysztof,

Here it goes the answers to your queries:

1) This is expected behavior. Once the connections are on the backup serverfarm, they will stay there until they complete. But as you are suspecting stickiness also plays a role here. So assuming you have sticky for the backup serverfarm this is the behavior you should see:

–All new sticky connections that match existing sticky table entries for the real servers in the backup server farm are stuck to the same real servers in the backup server farm.

–All new non-sticky connections and those sticky connections that do not have an entry in the sticky table are load balanced to the real servers in the primary server farm.

Hopefully this is what you are seeing

2) When you reach the MAXCONN threshold, you can expect that new connections will be sent to the backup farm. Depending on the platform you are using, this threshold value may be divided by the amount of IXPs (Network processors).

3) That is normally what we see customers doing, so in spite of having:

sticky ip-netmask 255.255.255.255 address source STICKY_PRIMARY

timeout 120

serverfarm SFARM_PRIMARY backup SFARM_BACKUP

sticky http-cookie BACKUP_ID STICKY_BACKUP

cookie insert

timeout 60

serverfarm SFARM_BACKUP

You would simply do (note the sticky keyword after the backup farm):

sticky ip-netmask 255.255.255.255 address source STICKY_PRIMARY

timeout 120

serverfarm SFARM_PRIMARY backup SFARM_BACKUP sticky

And the behavior is exactly the same. If all the servers in the primary server farm go down, the ACE sends all new requests to the backup server farm. When the primary server farm comes back up (at least one server becomes active):

•If the sticky option is enabled, then:

–All new sticky connections that match existing sticky table entries for the real servers in the backup server farm are stuck to the same real servers in the backup server farm.

–All new non-sticky connections and those sticky connections that do not have an entry in the sticky table are load balanced to the real servers in the primary server farm.

•If the sticky option is not enabled, then the ACE load balances all new connections to the real servers in the primary server farm.

•Existing non-sticky connections to the servers in the backup server farm are allowed to complete in the backup server farm.

This has been documented here:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/vA2_3_0/configuration/slb/guide/sticky.html#wp1137791

4) No, at first sight your configuration looks fine. However if I understood you correctly, you would need to remove sticky for the backup farm to meet your requirements. Or at least to achieve a behavior close to what you are expecting.

HTH,

Telmo

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Krzysztof Obara — Tue, 23 Jul 2013 23:29:32 GMT

Hello Telmo,

Thank you very much for your answers

Here are my conclusions:

1) & 3) As suspected, the stickiness caused the problem with users when the sorry page was triggered by them. I am about to remove the sticky http-cookie from the backup sfarm but what about this sentence from the link that you provided (the last paragraph under that section):

If you want to configure sorry servers and you want existing connections to revert to the primary server farm after it comes back up, do not use stickiness.

Does it mean to not use stickiness for primary and backup sfarm or only for backup sfarm?

2) I'm using ACE module on Cisco 6500 switch, IOS ver: A2(3.5) and there are 2 NP's. When using sorry sfarm, should I also remove failaction reassign? This action is rather used for passing traffic to stateful firewalls (as backups). Please correct me if I'm wrong

4) Regardless of understanding the above sentence from ACE config guide (I guess, they had in mind backup sfarm only), removing sticky from backup sfarm will solve the problem

Much appreciated for your help,

Krzysztof

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

jeongdae.lee — Wed, 24 Jul 2013 01:22:59 GMT

Hello Ajay.

Thanks for reply.

My network diagram is the following :

Server1---->ACE1 ----> ASA1----->ACE2----->Server2

| ------>ASA2----------|

"DMZ" | "INSIDE"

Two servers are communicating with each other via specific tcp port numbers.

My problem is how to load-balance the traffic from server1 to server2 and vice versa through both ASA1 and ASA2.

Currently, the ASA1 only has the connections from server1 to server2.

Thank you for reading.

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

ajayku2 — Wed, 24 Jul 2013 07:02:52 GMT

Hello Jeongdae,

I understand it now. You can refer the following link to do firewall load balancing.

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA3_1_0/configuration/slb/guide/fwldbal.html

mac-sticky enable command is the one which does the trick here. It keeps the session with the same firewall.

let me know if you some specific question related to this setup.

regards,

Ajay Kumar

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Telmo Pereira — Wed, 24 Jul 2013 07:14:00 GMT

1) & 3) That sentence is only in reference to the backup farm.

2) You are correct. Failaction reassign will simply begin sending packets to the remaining/backup rserver. There must be some logic to sync the state on the OS/App across all of the rservers for this to work properly or the new server will simply drop the connection, since it never saw the handshake. Typically you see this feature used with firewall load

balancing because firewalls are replicating their connection state tables.

4) Correct.

Pleasure is all mine, hope this helps.

Regards,

Telmo

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Krzysztof Obara — Wed, 24 Jul 2013 15:28:24 GMT

Thank you Telmo.

It really helped me