Re: Ask the Expert: Understanding and Troubleshooting ACE Loadba - Page 3

ciscomoderator · ‎08-03-2012

With Sivakumar Sukumar

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about configuration and troubleshooting the Cisco Application Control Engine (ACE) loadbalancer with Sivakumar Sukumar. The Cisco ACE Application Control Engine Module for Cisco Catalyst 6500 Series Switches and Cisco 7600 Series Routers is a next-generation load-balancing and application-delivery solution. A member of the Cisco family of Data Center 3.0 solutions, the module:

Helps ensure business continuity by increasing application availability
Improves business productivity by accelerating application and server performance
Reduces data center power, space, and cooling needs through a virtualized architecture
Helps lower operational costs associated with application provisioning and scaling

Sivakumar Sukumar is an experienced support engineer with the High Touch Technical Support content team, covering all Cisco content delivery network technologies including Cisco Application Control Engine (ACE), Cisco Wide Area Application Services (WAAS), Cisco Content Switching Module, Cisco Content Services Switches, and other content products. He has been with Cisco for more than 2 years, working with major customers to help resolve their issues related to content products. He holds CCNP and DCASI certification.

Remember to use the rating system to let Sivakumar know if you have received an adequate response.

Sivakumar might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community discussion forum shortly after the event. This event lasts through August 24, 2012. Visit this forum often to view responses to your questions and the questions of other community members.

sivaksiv · ‎08-20-2012

Hi Ganesh,

Thanks for your question.

I dont have a lab setup to test your configuration right now but looking at the config can you verify if the ACE and switch are on the same subnet, looks like the there is a mismatch 2092/2093?

Regards,

Siva

ganessub · ‎08-20-2012

Siva,

Yes, they are. I am sory about the wrong config on the 6200 switch. It is in the 2092 subnet only.

Please let me know if you can find something wrong.

Thanks a ton !

sivaksiv · ‎08-20-2012

Hi Ganesh,

If you are using ipv6 then the access-list and the management policy should also be based on ipv6. Or you could first configure using ipv4, test the config and then migrate to ipv6.

sample config below:

access-list everyone extended permit ip anyv6 anyv6

access-list everyone extended permit icmpv6 anyv6 anyv6

class-map type management match-any ipv6

2 match protocol icmpv6 anyv6

Regards,

Siva

fd_case17 · ‎08-21-2012

HelloSiva,

can i have your action plan to determine what sort of traffic is causing reboot of our ACE 30 ?

(during upgrade ACE20 > ACE30 process )

i' m suspecting ANM but i' m not sure of that .

thanx in advance KR,

sivaksiv · ‎08-21-2012

Hi,

Thanks for your question.

The bug I mentioned above is specific to the legacy ACE modules which are the ACE10 and ACE20. It does not apply to the ACE30.

Can you please explain the problem in detail?

Did you see the module reload only at the time of migration? Were you able to complete the migration successfully?

Are there any core files generated under the core directory after reload?

If you could send me any traces/logs I should be able to figure out the reason for reload.

Let me know if you have any questions.

Regards,
Siva

fd_case17 · ‎08-22-2012

Hello Siva,

here is a log on core:snmpd_log

Service: snmpd

Description: SNMP Agent

Started at Sat Jul 28 17:19:52 2012 (265979 us)

Stopped at Sat Jul 28 17:22:55 2012 (808911 us)

Uptime: 3 minutes 3 seconds

Start type: SRV_OPTION_RESTART_STATELESS (23)

Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)

System image version: A5(1.1) 3.0(0)A5(1.1) adbuild_11:30:52-2011/10/25_/auto/adbure_nightly4/renumber/rel_a5_1_0_throttle/REL_3_0_0_A5_1_1

Image Version : A5(1.1)

System Image Interim version: 3.0(0)A5(1.1) adbuild_11:30:52-2011/10/25_/auto/adbure_nightly4/renumber/rel_a5_1_0_throttle/REL_3_0_0_A5_1_1

Thanx,

did you want more files ?

sivaksiv · ‎08-22-2012

Hi,

I would require the complete core file off the box for analysis, you could use "copy core: ftp:" to copy the core file to an ftp server.

But looking at the log it looks like the crash occurred on snmpd process.

Can you tell me how often do you see the reload happening?

I came across following defects and they are fixed in A5(2.0) and above. Did you see the problem with A5(2.0)?

CSCua48058 - ACE30 module crash, last boot reason: Service "snmpd"

CSCtx76952 - SNMP crash on ACE30 module

CSCts09006 - ACE crashed on snmpd after receiving SIG4 (SNMP)

Regards,

Siva

ganessub · ‎08-23-2012

Thanks Siva. The configuration is working . I guess , we missed this command out :

When enabling the ip address for a particular VLAN , this is for the ipv6 configuration which we are currently implementing, I overlooked the " ipv6 enable" command . This ensured that ipv6 routing is established for that vlan.

Thanks for your help !

Fernando Bello · ‎08-21-2012

Hi Siva,

Hope you're doing well.

Some weeks ago I had a weird problem with the ACE 4710 appliance.

We had 2 aces working in Fault Tolerance. Behind the ACE appliance, we have 2 Sharepoint 2010 being loadbalanced by the ace. Before the ACE appliance, there is a server that handles the requests from the LAN clients. After receiving those requests, it calls the Sharepoint WEBSERVICES behind the ACE appliance. So, this server, in turn, acts like a client to the call the webservices behind the ACE. The Server behind the ACE and the server (client) before the ACE are in the same VLAN. Since they are in the same VLAN, to get the request go through the ACE appliance, we made a SOURCE NAT. After doing that, every time that the client(server before the ACE) made a request to the Sharepoint servers, we got a 400 bad request error. In order to avoid this problem, we tested the same request from the same client but bypassing the ACE and everything was ok. We did many other tests and found that if we remove the line

"server-conn reuse" from the HTTP parameter PA_HTTP_PERSIS_TCP_REUSE, the error was not happening.

parameter-map type http PA_HTTP_PERSIS_TCP_REUSE

description Http parameter to HTTP Persistance rebalance and TCP Server connection reuse

server-conn reuse ---------------------- > remove

case-insensitive

persistence-rebalance

set header-maxparse-length 65535

set content-maxparse-length 65535

length-exceed continue

Why could this be happening?

thanks in advance for your help!

Fernando

sivaksiv · ‎08-21-2012

Hi Fernando,

Thanks for your question.

I'm glad you are able to isolate the problem by removing the conn reuse command off the ACE.

First let me explain how a conn resuse on ACE works:

If a client sends a HTTP/1.1 request with the "Connection: close" header, to achieve the back-end connection to be open even after the response, ACE will remove the "Connection: close" header and insert the "Connection: Keep-Alive" header in the request before forwarding it to the rserver.

Now if the client sends a request to the VIP and closes it after receiving the response, then only the client-side connection is removed from the connection database, and the server-side connection is kept in the reuse pool.

Now if the client opens a connection to the VIP and sends a request in it, then this time, instead of opening a new backend connection to the rserver, ACE uses the back-end connection in the reuse pool.

So basically TCP server reuse allows the ACE to reduce the number of open connections on a server by allowing connections to persist and be reused by multiple client connections.

The ACE maintains a pool of TCP connections based on TCP options. New client connections can reuse those connections in the pool provided that the new client connections and prior server connections share the same TCP options.

To ensure proper operation of this feature , we need to ensure below :-

Ensure that the ACE MSS is the same as the server MSS.
Configure port address translation (PAT) on the interface that is connected to the real server.
Configure on the ACE the same TCP options that exist on the TCP server.
Ensure that each server farm is homogeneous (all real servers within a server farm have identical configurations)

Now looking at the issue you are facing, there is no way for the ACE to force the servers to send the 400 or to send a 400 response on its own. Since the server is sending the 400 I would recommend you to check if it accepts the way ACE reuses the connection.

If you have a packet capture we can analyze at packet level and check if anything changed between ACE and server.

Let me know if you have any questions.

Regards,
Siva

Fernando Bello · ‎08-22-2012

Thanks siva, I think that the answer was good enough. No need to check the captures. Thanks a lot!!!!!

Akhtar Samo · ‎08-22-2012

Hi Siva,

We had a requirement from one of our customer that their client terminals (POS terminals) should be authenticated by the ACE which is terminating the SSL connection. Backend connections to the server is clear text.

In a normal SSL flow the server sends the certificate to the client and the client verifies the identity of the server but in our case we need ACE to authenticate the client or some form of mutual authentication should be there.

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/vA2_3_0/configuration/ssl/guide/terminat.html#wp1117637

As per the documentation we have enabled the authgroup to enable the client authentication feature, but when we are testing the application it seems that only the front end (client to ACE) connection gets established but not the back end.

We have verified that if client authentication is disabled the application works fine but the ACE sends it the certificate and the client is not authenticated.

crypto authgroup POS

cert cert_client.pem

ssl-proxy service ssl-proxy

key POS

cert cert_server.pem

authgroup POS

ssl advanced-options POS

Our scenario is like given below with client authentication

(Server)---------------clear text----------------(ACE)-----------------SSL--------------------(POS Terminals/client)

Regards,

Akhtar

Akhtar Samo · ‎08-22-2012

Hello Siva,

Just wondering if this would be the right way to check traffic based on source IP before it gets loadbalanced ? These configuration doesn't work. Can you pls. assist.

**************************CONFIGURATION-OPTION-1*********************************************

rserver host PLATTS_APP

ip address 192.168.0.1

inservice

serverfarm host SF_PLATTS

transparent

rserver PLATTS_APP

inservice

class-map match-any SRC-IP-A

2 match source-address 192.168.80.89 255.255.255.255

policy-map type loadbalance first-match PM_L7_BYPASS_SRC_IP

class class-default

serverfarm SF_PLATTS

class-map match-any CM_BYPASS_VIP

2 match virtual-address 10.10.10.10 any

3 match virtual-address 20.20.20.20 any

4 match virtual-address 30.30.30.30 any

5 match virtual-address 40.40.40.40 any

policy-map multi-match PM_BYPASS_SRC_IP

class CM_BYPASS_VIP insert-before SRC-IP-A

loadbalance vip inservice

loadbalance policy PM_L7_BYPASS_SRC_IP

**************************CONFIGURATION-OPTION-2*********************************************

rserver host PLATTS_APP

ip address 192.168.0.1

inservice

serverfarm host SF_PLATTS

transparent

rserver PLATTS_APP

inservice

class-map match-any SRC-IP-A

2 match source-address 192.168.80.89 255.255.255.255

policy-map type loadbalance first-match PM_L7_BYPASS_SRC_IP

class SRC-IP-A

serverfarm SF_PLATTS

class-map match-any CM_VIP

2 match virtual-address 10.10.10.10 any

3 match virtual-address 20.20.20.20 any

4 match virtual-address 30.30.30.30 any

5 match virtual-address 40.40.40.40 any

policy-map multi-match PM_

class CM_BYPASS_VIP

loadbalance vip inservice

loadbalance policy PM_L7_

******************************************************************************

Regards,

Akhtar

sivaksiv · ‎08-22-2012

Hi Akhtar,

Thanks for your question.

If I understand correctly, you want to setup SSL client authencation on ACE to setup mutual authentication between ACE and client. If you implement client authentication on ACE, it will send a client certificate request with all of the certificates in the authgroup. The cert that client sends in response must be signed by one of the certs in the cert group.

Since you mentioned that the front end connection is established but not the backend I would like to understand if the request atleast forwarded to the server in the backend after ssl authentication.

If you have a packet capture we can verify the TCP and SSL handshake between and client and ACE and handshake between ACE and server.

To configure load balancing based on SRC ip address, the below config should work:

The traffic that matches 192.168.80.89 should be loadbalanced to SF_PLATTS

rserver host PLATTS_APP

ip address 192.168.0.1

inservice

serverfarm host SF_PLATTS

transparent

rserver PLATTS_APP

inservice

class-map match-any SRC-IP-A

2 match source-address 192.168.80.89 255.255.255.255

policy-map type loadbalance first-match PM_L7_BYPASS_SRC_IP

class class-default

serverfarm SF_PLATTS

class-map match-any CM_BYPASS_VIP

2 match virtual-address 10.10.10.10 any

3 match virtual-address 20.20.20.20 any

4 match virtual-address 30.30.30.30 any

5 match virtual-address 40.40.40.40 any

policy-map multi-match PM_BYPASS_SRC_IP

class SRC-IP-A

loadbalance vip inservice

loadbalance policy PM_L7_BYPASS_SRC_IP

class CM_BYPASS_VIP

loadbalance vip inservice

loadbalance policy PM_L7_

Let me know if you have any questions.

Regards,
Siva

Akhtar Samo · ‎08-22-2012

Here is the challenge which we are facing,

1. If the traffic comes from SRC-IP-A and their destination is CM_BYPASS_VIP it should be loadbalanced using PM_L7_BYPASS_SRC_IP (using policy PM_BYPASS_SRC_IP)

2. If the traffic comes from SRC-IP-A and their destination is 'any' then it should using policy PM_MAIN_BCPROXY and in turn would be load balanced using PM_LB_SF_BCPROXY / serverfarm SF_BCPR

rserver host RS_BCPR01

ip address 192.168.0.103

inservice

rserver host RS_BCPR02

ip address 192.168.0.104

inservice

serverfarm host SF_PLATTS

transparent

rserver PLATTS_APP

inservice

serverfarm host SF_BCPR

transparent

probe PROBE_TCP

rserver RS_BCPR01

inservice

rserver RS_BCPR02

inservice

sticky ip-netmask 255.255.255.255 address source STICKY-SOURCE

replicate sticky

serverfarm SF_BCPR

class-map match-all CM_SF_BCPR

255 match virtual-address 0.0.0.0 0.0.0.0 tcp eq www

class-map match-any SRC-IP-A

2 match source-address 192.168.80.89 255.255.255.255

class-map match-any CM_BYPASS_VIP

2 match virtual-address 10.10.10.10 any

3 match virtual-address 20.20.20.20 any

4 match virtual-address 30.30.30.30 any

5 match virtual-address 40.40.40.40 any

policy-map type loadbalance http first-match PM_LB_SF_BCPROXY

class class-default

sticky-serverfarm STICKY-SOURCE

policy-map type loadbalance first-match PM_L7_BYPASS_SRC_IP

class class-default

serverfarm SF_PLATTS

policy-map multi-match PM_MAIN_BCPROXY

class CM_SF_BCPR

loadbalance vip inservice

loadbalance policy PM_LB_SF_BCPROXY

loadbalance vip icmp-reply active

appl-parameter http advanced-options PARAMAP_CASE

policy-map multi-match PM_BYPASS_SRC_IP

class CM_BYPASS_VIP insert-before SRC-IP-A

loadbalance vip inservice

loadbalance policy PM_L7_BYPASS_SRC_IP

int vlan 300

service-policy input PM_BYPASS_SRC_IP

service-policy input PM_MAIN_BCPROXY

Regards,

Akhtar

Ask the Expert: Understanding and Troubleshooting ACE Loadbalancer