topic ASK THE EXPERTS:Configuring and troubleshooting Session Persiste in Application Networking

ASK THE EXPERTS:Configuring and troubleshooting Session Persistence on Application Control Engine

ciscomoderator — Fri, 15 Jul 2011 22:48:05 GMT

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to get an update on how to configure and troubleshoot session persistence (stickiness) on ACE with regards to specific protocols with Cisco expert Olivier Hynderick. Olivier has been working for the Cisco Technical Assistance Center for four years. He focuses on the Cisco Application Control Engine (ACE), Cisco Security Manager, and Cisco Wide Area Application Services and related technologies. He initially joined the Security team focusing on the Cisco ASA firewall and VPN on Cisco IOS applications before getting involved in the support of the Cisco ACE load balancer.

Remember to use the rating system to let Olivier know if you have received an adequate response.

Olivier might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Application Networking discussion forum shortly after the event. This event lasts through July 29, 2011. Visit this forum often to view responses to your questions and the questions of other community members.

Re: ASK THE EXPERTS:Configuring and troubleshooting Session Pers

g.eleftheriou — Mon, 18 Jul 2011 07:11:38 GMT

Hello,

I have a cluster of ACE4710 and I use cookies for stickiness.

My question is the following:

I have a farm with server01 and server02. When server01 goes down but the client has already received a cookie for server01, will ACE clear the cookie and send the client a cookie for server02? and are there any special commands to do this?

here is what I have

sticky http-cookie ACE_COOKIE_mobile ebanking_mobile_sticky

cookie insert

replicate sticky

serverfarm SF_Ebanking_Mobile

16 static cookie-value "server01" rserver RS_IAS_1

24 static cookie-value "server02" rserver RS_IAS_2

thanks,

george

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

ohynderi — Mon, 18 Jul 2011 09:28:03 GMT

Hello George,

In case ACE receives a request containing a cookie value but the corresponding server in in the sticky database is down, ACE should loadbalancing the new request to one of the available servers ignoring the cookie.

To illustrate this, see below a quick test that I did in my lab:

sticky http-cookie ACE_COOKIE_mobile ebanking_mobile_sticky

cookie insert

replicate sticky

serverfarm sf1

ACE/ctx# sh sticky cookie-insert group ebanking_mobile_sticky

Cookie | HashKey | rserver-instance

------------+----------------------+----------------------------------------+

R4072271931 | 15598686253581426628 | sf1/server1:0

R4072273020 | 6532832188001237582 | sf1/server2:0

GET / HTTP/1.1

Host: 10.10.170.13

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/5.0.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip, deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Connection: keep-alive

Cookie: ACE_COOKIE_mobile=R4072273020 --> cookie value corresponding to server2

HTTP/1.1 200 OK

Set-Cookie: ACE_COOKIE_mobile=R4072271931; path=/; expires=Tue, 19-Jul-2011 09:22:39 GMT --> server2 is down, so request went to server1 and corresponding cookie was inserted.

...

Small remark, you shouldn't configure both "cookie insert" and static value for a server. What would you like to achieve?

Thanks,

Olivier

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

g.eleftheriou — Mon, 18 Jul 2011 10:15:51 GMT

I thought I need to use both.Can you explain the difference please?

If I use only "cookie insert" it creates the cookies for me and if I use static I name the cookies as I like?

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

ohynderi — Mon, 18 Jul 2011 14:22:33 GMT

Hi again George,

With cookie insertion, ACE generates a cookie value based on the serverfarm and the real server name. This value is permanent and inserted in each server response.

With static values, ACE forwards requests containing cookies according to the static command. This is very similar to cookie insertion accept that ACE no longer inserts cookies...

On ACE 20 (module), you can’t combine cookie insertion with static values. Static command will be ignored. On the other hand, you have this possibility on the ACE appliance as of A3(2.2) and on the ACE30.

--Olivier

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

m.paulo.pires — Sat, 23 Jul 2011 02:19:23 GMT

There are a software to mac ?

Linksys Wireless-G PTZ Internet Camera with Audio - WVC210

Thanks

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

robwu2006 — Tue, 26 Jul 2011 16:54:51 GMT

Hi, is there a list of useful commands for troubleshooting cookie-insert on the ACE?

I will try to get some output, but we have in the past had issues on our ACE modules and cookie insert, so we have resorted to IP based stickiness, but we are finding that we really need to move to the cookie-insert style of stickiness.

Thanks in advance,

Cheers,

Rob

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

Thompso7540_2 — Wed, 27 Jul 2011 16:20:53 GMT

Hey Olivier,

Have you ever seen any issues w/ session persistance and local balancing radius and tacacs? We are trying to put some Cisco ACS servers behind our ACE20, but were not sure if session persistence would be required.

I know that the ACE supports class-maps for radius and tacacs.

Thanks,

Bryan

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

ohynderi — Thu, 28 Jul 2011 07:02:43 GMT

Hi Rob,

When troubleshooting cookie-insertion issues, best is to look at a network capture. With cookie insertion, ACE should insert below header field in the server responses.

Set-Cookie: =R; path=/; expires=”

You will see below counter being incremented.

ACE/ctx# sh stats http | in "Headers inserted"

Headers inserted : 38

Do you then see the client adding the corresponding cookie in subsequent http requests?

Typical error with cookie insertion are:

- Clock on ACE is incorrect causing ACE to insert "Set-Cookie" with the date set in the past. Client will then ignore the Cookie.

- ACE failed to parse a client request because not rfc compliant. Connection is then drop to L4 and the "Static parse errors" counter is incremented.

ACE/ctx# sh stats http | in Static

Static parse errors : 7 , Resource errors : 0

- ACE failed to parse a client request because header is too big. You have the "Max parselen errors" counter for this.

ACE/ctx# sh stats http | in Max

Header insert errors : 0 , Max parselen errors : 0

Let me know if you need more info.

Olivier

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

ohynderi — Thu, 28 Jul 2011 07:34:07 GMT

Hello Bryan,

What kind of issues are you facing?

Common problem with radius load balancing is that the clients often use a very limited number of udp connections to sent requests to the aaa servers. Usually those connections never time out as clients keep reusing them. This can lead to unfair load balancing.

By configuring radius stickiness, we force the ACE to load balance each radius request individually. So yes, it is good proactive to have it! See below a config example with radius stickiness.

http://www.cisco.com/en/US/partner/docs/interfaces_modules/services_modules/ace/vA2_3_0/configuration/slb/guide/classlb.html#wp1112138

Radius class-map is something else. You may for instance want to send requests with some particular radius attributes vaule (eg calling station id, …) to a specific serverfarm. I am not sure you need this.

Finally, make sure that your context is not running out of sticky resources, otherwise ACE will start reusing exiting sticky entires. You can monitor this with following command:

ACE/ctx# sh stats sticky

+------------------------------------------+

+----------- Sticky statistics ------------+

+------------------------------------------+

Total sticky entries reused : 3464 <<<<

prior to expiry

Total active sticky entries : 2

Total active reverse sticky : 0

entries

Total active sticky conns : 0

Total static sticky entries : 2

Regards,

Olivier

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

robwu2006 — Thu, 28 Jul 2011 13:29:04 GMT

HI Olivier,

Thanks for the tips, at the moment our clock on the ACE module is like 1 hour out, as its in UTC, with all the clients are on BST.

The cat6k that we have the ACE modules in are on the correct time (BST), I thought the ACE clock comes from the cat6k, so they should be the same?

Is there a way to explicitly set the ACE module clock, having a quick look but can't find any clock/ntp commands in config t

Also some output below, when we go via our VIP, I do not see any sticky entries for my clients IP or the sticky database group while there are active sessions. Is this normal?

ACE01/DMZ-VRF# sh service-policy TEST_VIP detail

Status : ACTIVE

Description: -

-----------------------------------------

Interface: vlan 849

service-policy: TEST_VIP

class: TEST_7777

nat:

nat dynamic 102 vlan 849

curr conns : 1 , hit count : 28

dropped conns : 0

client pkt count : 880 , client byte count: 222069

server pkt count : 1062 , server byte count: 890955

conn-rate-limit : 0 , drop-count : 0

bandwidth-rate-limit : 0 , drop-count : 0

VIP Address: Protocol: Port:

172.31.XXX.XXX tcp eq 7777

loadbalance:

L7 loadbalance policy: STICKY_COOKIE

VIP Route Metric : 10

VIP Route Advertise : ENABLED-WHEN-ACTIVE

VIP ICMP Reply : ENABLED

VIP State: INSERVICE

curr conns : 1 , hit count : 115

dropped conns : 0

client pkt count : 2783 , client byte count: 720269

server pkt count : 3163 , server byte count: 2530541

conn-rate-limit : 0 , drop-count : 0

bandwidth-rate-limit : 0 , drop-count : 0

L7 Loadbalance policy : STICKY_COOKIE

class/match : class-default

LB action :

sticky group: websphere-sticky-cookie

primary serverfarm: TEST_SFARM

state: UP

backup serverfarm : -

hit count : 983

dropped conns : 9

ACE01/DMZ-VRF# sh sticky database type http-cookie

ACE01/DMZ-VRF# sh sticky database client 172.16.15.7

FYI, we are running a fairly old version of the ACE code, but I'm assuming that this feature has been ok for quite a while, with the sticky cookie insert?

disk0:c6ace-t1k9-mz.A2_1_2.bin

Cheers,

Rob

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

ohynderi — Thu, 28 Jul 2011 17:46:02 GMT

Rob,

ACE can indeed sync with the clock of the chassis if chassis is configured with "clock calendar-valid". Although be aware that ACE and chassis can be configured to be in different time zone…

You can't explicitly set the clock on an ACE module. Like you said, it needs to synchronize with the chassis ;-).

With cookie insertion, ACE assigns statically a cookie to each server. The database doesn't actually contain any information related to the client. This is why "sh sticky database client" doesn't return anything. You have to look at "sh sticky database static".

sticky http-cookie my_cookie cookie1

cookie insert

serverfarm sf1

policy-map type loadbalance first-match lb_policy

class class-default

sticky-serverfarm cookie

ACE/ctx# sh sticky database static

sticky group : cookie1

type : HTTP-COOKIE

timeout : 1440 timeout-activeconns : FALSE

sticky-entry rserver-instance time-to-expire flags

---------------------+--------------------------------+--------------+-------+

R4072271931 server1:0 never -

sticky group : cookie1

type : HTTP-COOKIE

timeout : 1440 timeout-activeconns : FALSE

sticky-entry rserver-instance time-to-expire flags

---------------------+--------------------------------+--------------+-------+

R4072273020 server2:0 never -

If you believe ACE doesn't stick the client connections as it should. You may try to see if configuring "persistence rebalance" does not help.

http://www.cisco.com/en/US/partner/docs/interfaces_modules/services_modules/ace/v3.00_A2/configuration/slb/guide/classlb.html#wpxref30971

Thanks,

Olivier

Re: ASK THE EXPERTS:Configuring and troubleshooting Session Pers

Surya ARBY — Thu, 28 Jul 2011 19:50:03 GMT

Hello.

Is there any real use case for persistance based on SSL session-id ?

It used to be used in the past (let's say, maybe, 10 years ago) but since we discovered issues with some old IE versions restarting the negotiation of SSL session every 2 minutes it was not used anymore, replaced by cookie insertion when possible or source-IP based persistance.

I've always wondered why all vendors still provide this feature which seems to be useless (SSL sessions should not be related to the applications sessions according to the OSI model).

Any clue ?

Other question, I usually set the cookie insertion with a timeout of 0 (cookie only available until the browser is closed) for web-based (browser-based) applications. Have you ever seen any case where the cookie insert method doesn't work and you had to use cookie learning or other stuff like that ? or timeout different of 0 ?

Is there any official guide of best practices (even outside Cisco's authorized material) available ? The only book I've found is the one published by Cisco Press (written by Maurizzio Portolani) and it's old.

Thanks in advance.

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

ohynderi — Fri, 29 Jul 2011 12:53:05 GMT

Hello Surya,

The SSL/TLS protocol allows clients to re-use same key materials for multiple TCP connections. When this happens, client sends a hello message containing already an SSL Session-ID. From there, there are 2 possibilities. Either the server has or still has the corresponding key materials. Or it responds with a different SSL Session-ID forcing the client to generate new key materials.

I guess you knew this already. The point of stickiness based on Session-ID is just to make sure that the client hello message is sent to the server that has to correct key materials. This to make the SSL negotiation faster. In case you have to maintain, multiple connections with different Session-ID, then indeed, it won't help. That really depends on your application and what you are looking for...

Anyway, ACE actually uses the Generic Protocol Parsing (GPP) to achieve this. So I would say that stickiness based on SSL Session-ID is one of the many thing you can achieve with GPP than a real option...

See below another link on this, which explains more or less the same...

http://docwiki.cisco.com/wiki/Secure_Sockets_Layer_Persistence_Configuration_Example#Limitations_of_SSL_stickiness

About cookie insertion with 0 timeout, i don't thing i have ever seen any problems with that. Actually doing so, you no longer have to rely on the load balancer clock which may, for whatever reasons, be/become incorrect.

About learning cookie, again, that depends on your application. I have seen some customer applications where client was only taken into account a specific cookie inserted by the ACE.

Thanks,

Olivier

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

robwu2006 — Fri, 29 Jul 2011 16:26:12 GMT

thanks for that tip, it looks like my clocks are synced, so I will have another go at this next week.

FYI, the link you sent :

http://www.cisco.com/en/US/partner/docs/interfaces_modules/services_modules/ace/v3.00_A2/configuration/slb/guide/classlb.html#wpxref30971

it doesnt work for me I get a Forbidden File or Application, are you able to provide another link?

Cheers,

Rob

ASK THE EXPERTS:Configuring and troubleshooting Session Persiste

robwu2006 — Fri, 29 Jul 2011 16:48:10 GMT

I think I've found the link, is it this one your talking about?

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A1/configuration/slb/guide/classlb.html#wp1062907