Connection reset on ACE

[turfsniffer0]

Hi,

We have a VIP configured for a server-farm which hosts a specific web service that is mostly used for internal users. When the users is browsing and clicks an option on the browser the get a blank white screen (instead of data). The server-farm hosts two real servers, so I marked one out of service and issue still happens. A wireshark on the client shows a TCP ACK/RST coming from the VIP and the server wire-shark shows the same coming from client. It looks as if its the ACE is resetting the connection. I did a capture on the ACE itself and again there is no indication of an event that causes the reset, both ends appear to get sent a reset. To verify, we got the client to talk directly to the server (bypassing the load-balancer) and it works with no issue. We upgraded the software for the ACE context and same behaviour so starting to suspect that it's the configuration of the farm. The only problem is the configuration is not standard and looks a bit messy and I'm having problems trying to figure out the flow. (I've inherited the problem and the configuration). I don't fully understand the setup regarding the web service, but I understand that it should be accessed externally as well. Any help deciphering the config would be appreciated

rserver host DC-W01
  ip address 10.3.3.11
  inservice
rserver host DC-W02
  ip address 10.3.3.12
  inservice

serverfarm host DC-W01-FARM
  probe HTTP
  rserver DC-W01 80
    inservice

serverfarm host DC-W02-FARM
  probe HTTP
  rserver DC-W02 80
    inservice

serverfarm host DC-WSMQ-FARM
  predictor response syn-to-synack
  probe HTTP
  rserver DC-W01 80
    inservice
  rserver DC-W02 80
    inservice

sticky ip-netmask 255.255.255.255 address source DC-WSMQ-FARM-STICKY
  serverfarm DC-WSMQ-FARM

class-map match-any L4-DC-WSMQ-WEXT
  2 match virtual-address 10.1.253.18 tcp eq https

class-map match-any L4-DC-WSMQ
  2 match virtual-address 10.1.253.17 tcp eq https

class-map type http loadbalance match-all L7-TP-DC-W01-FARM
  2 match http header Host header-value ".*-wsm01.*
class-map type http loadbalance match-all L7-TP-DC-W02-FARM
  2 match http header Host header-value ".*-wsm02.*"

policy-map type loadbalance http first-match L7-POLICY-DC-WSMQ-WEXT
  class L7-TP-DC-W01-FARM
    serverfarm DC-W01-FARM
    insert-http AppStoreNetworkLoc header-value "External"
  class L7-TP-DC-W02-FARM
    serverfarm DC-W02-FARM
    insert-http AppStoreNetworkLoc header-value "External"
  class class-default
    sticky-serverfarm DC-WSMQ-FARM
    insert-http AppStoreNetworkLoc header-value "External"

policy-map type loadbalance http first-match L7-POLICY-DC-WSMQ
  class L7-TP-DC-W01-FARM
    serverfarm DC-W01-FARM
    insert-http AppStoreNetworkLoc header-value "Internal"
  class L7-TP-DC-W02-FARM
    serverfarm DC-W02-FARM
    insert-http AppStoreNetworkLoc header-value "Internal"
  class class-default
    sticky-serverfarm DC-WSMQ-FARM
    insert-http AppStoreNetworkLoc header-value "Internal"

policy-map multi-match L4-SSL-VIP-POLICY
class L4-DC-WSMQ
    loadbalance vip inservice
    loadbalance policy L7-POLICY-DC-WSMQ
    loadbalance vip icmp-reply
    loadbalance vip advertise active
    ssl-proxy server SSL-PROXY
 
class L4-SDC-SWS-WSMEXTQ
    loadbalance vip inservice
    loadbalance policy L7-POLICY-DC-WSMQ-WEXT
    loadbalance vip icmp-reply
    loadbalance vip advertise active
    ssl-proxy server SSL-PROXY

loader:    Version 12.2[125]
system:    Version A5(3.3) [build 3.0(0)A5(3.3)

[turfsniffer0]

Just an update on this issue. Had to get ssl certs installed on both client and server to allow ssl offloading to be turned off so that the client talks to server directly over port 443. The issue disappears with offloading disabled and reappears when it's enabled. Apart from wiresharks is there a debug I can do on the ACE to identify the trigger that is causing the ACE to send out a reset to both client and server at the same time? The problem is fairly consistent and seems to occur when accessing a link that sends a GET request to server.

[Richard Bradfield]

Have you seen this link, a good example

http://docwiki.cisco.com/wiki/SSL_Termination_on_the_Cisco_Application_Control_Engine_Using_an_Existing_Chained_Certificate_and_Key_in_Routed_Mode_Configuration_Example

[turfsniffer0]

Thanks Richard, haven't seen it before but the ssl cert and key already exists and work fine for other server farms, so I don't think it's relating to the configuration of the ssl offloading itself - but more to do with the load balancing configuration.

[Jorge Bejarano]

Hi,

You may try these changes:

parameter-map type http PARAMETER-HTTP
case-insensitive
persistence-rebalance
set header-maxparse-length 65535
set content-maxparse-length 65535
length-exceed continue
parsing non-strict

policy-map multi-match L4-SSL-VIP-POLICY
class L4-DC-WSMQ
loadbalance vip inservice
loadbalance policy TEST_POLICY
loadbalance vip icmp-reply active
appl-parameter http advanced-options PARAMETER-HTTP ---> apply it like this
class L4-SDC-SWS-WSMEXTQ
loadbalance vip inservice
loadbalance policy L7-POLICY-DC-WSMQ-WEXT
loadbalance vip icmp-reply
loadbalance vip advertise active
ssl-proxy server SSL-PROXY
appl-parameter http advanced-options PARAMETER-HTTP ---> apply it like this

serverfarm host DC-W01-FARM
failaction purge ---> add this line
probe HTTP
rserver DC-W01 80
inservice

serverfarm host DC-W02-FARM
failaction purge ---> add this line
probe HTTP
rserver DC-W02 80
inservice

serverfarm host DC-WSMQ-FARM
failaction purge ---> add this line
predictor response syn-to-synack--> remove this line
probe HTTP
rserver DC-W01 80
inservice
rserver DC-W02 80
inservice

References.
failaction purge: http://www.cisco.com/c/en/us/td/docs/interfaces_modules/services_modules/ace/vA5_1_0/command/reference/ACE_cr/servfarm.html#wp1107574

http parameter:
http://www.cisco.com/c/en/us/td/docs/interfaces_modules/services_modules/ace/vA5_1_0/command/reference/ACE_cr/parammap.html#wp1650471

Hope this helps!

Jorge

Don´t forget to rate the answers.