The short of the long story is this. I have four rservers. I have found that if the first listed server in my serverfarm is off line, the entire farm quits working. How did I come to this conclusion? You see as part of "serverfarm host PORTAL-FARM" rservers "SISPOAS1 through 4". I can shut down any server except SISPOAS1 and all is well. The load balancer sees the probes have failed to that given server and continuses to load balance to the others. However, If I shut down SISPOAS1, nothing works. I confirmed this by eliminating SISPOAS1 from the configuration completely. After doing so, I could reproduce the exact same problem using SISPOAS2 since it is now the first rserver in the list after I removed SISPOAS1. I'm stumped! Looking at the configuration below, what am I missing???
access-list TRAFFIC line 8 extended permit ip any any
access-list TRAFFIC line 16 extended permit icmp any any
probe tcp 389
port 389
interval 2
passdetect interval 2
passdetect count 1
open 1
probe tcp 636
port 636
interval 2
passdetect interval 2
passdetect count 1
open 1
probe tcp 7777
port 7777
interval 2
passdetect interval 2
passdetect count 1
open 1
probe tcp 7778
port 7778
interval 2
passdetect interval 2
passdetect count 1
open 1
probe tcp 7780
port 7780
interval 2
passdetect interval 2
passdetect count 1
open 1
probe tcp 7782
port 7782
interval 2
passdetect interval 2
passdetect count 1
open 1
probe tcp 9401
port 9401
interval 2
passdetect interval 2
passdetect count 1
open 1
rserver host FORMS1
ip address 165.199.5.151
inservice
rserver host FORMS2
ip address 165.199.5.152
inservice
rserver host SISPOAS1
ip address 165.199.5.81
inservice
rserver host SISPOAS2
ip address 165.199.5.82
inservice
rserver host SISPOAS3
ip address 165.199.5.83
inservice
rserver host SISPOAS4
ip address 165.199.5.84
inservice
serverfarm host BIS-FARM-SSL
predictor leastconns
probe 7780
rserver SISPOAS1 7780
inservice
rserver SISPOAS2 7780
inservice
rserver SISPOAS3 7780
inservice
rserver SISPOAS4 7780
inservice
serverfarm host FORMS-FARM-SSL
predictor leastconns
probe 7777
rserver FORMS1 7777
inservice
rserver FORMS2 7777
inservice
serverfarm host PORTAL-FARM
predictor leastconns
probe 389
probe 636
probe 7777
probe 7778
probe 7780
probe 7782
probe 9401
fail-on-all
rserver SISPOAS1
inservice
rserver SISPOAS2
inservice
rserver SISPOAS3
inservice
rserver SISPOAS4
inservice
serverfarm host PORTAL-FARM-SSL
predictor leastconns
probe 7778
rserver SISPOAS1 7778
inservice
rserver SISPOAS2 7778
inservice
rserver SISPOAS3 7778
inservice
rserver SISPOAS4 7778
inservice
serverfarm host RPORTAL-FARM-SSL
predictor leastconns
probe 7777
rserver SISPOAS1 7777
inservice
rserver SISPOAS2 7777
inservice
rserver SISPOAS3 7777
inservice
rserver SISPOAS4 7777
inservice
sticky ip-netmask 255.255.255.255 address source STICKY-PORTAL-CLIENT
serverfarm PORTAL-FARM
sticky ip-netmask 255.255.255.255 address source STICKY-PORTAL-CLIENT-SSL
serverfarm PORTAL-FARM-SSL
sticky ip-netmask 255.255.255.255 address source STICKY-RPORTAL-CLIENT-SSL
serverfarm RPORTAL-FARM-SSL
sticky ip-netmask 255.255.255.255 address source STICKY-BIS-CLIENT-SSL
serverfarm BIS-FARM-SSL
sticky ip-netmask 255.255.255.255 address source STICKY-FORMS-CLIENT-SSL
serverfarm FORMS-FARM-SSL
ssl-proxy service BIS-SSL-PROXY
key bisrsakey20120522.pem
cert bis20120522CA.pem
ssl-proxy service FORMS-SSL-PROXY
key forms0rsakey20130103.pem
cert forms020130103CA.pem
ssl-proxy service PORTAL-SSL-PROXY
key portalrsakey20120522.pem
cert portal20120522CA.pem
ssl-proxy service RPORTAL-SSL-PROXY
key rportalrsakey20120522.pem
cert rportal20120522CA.pem
class-map match-all BIS-CLASS-HTTPS
2 match virtual-address 165.199.5.159 tcp eq https
class-map match-all FORMS-CLASS-HTTPS
2 match virtual-address 165.199.5.164 tcp eq https
class-map match-any PORTAL-CLASS-HTTP
2 match virtual-address 165.199.5.158 tcp any
4 match virtual-address 165.199.5.80 tcp any
6 match virtual-address 165.199.5.159 tcp any
8 match virtual-address 165.199.5.79 tcp any
class-map match-all PORTAL-CLASS-HTTPS
2 match virtual-address 165.199.5.158 tcp eq https
class-map match-all RPORTAL-CLASS-HTTPS
2 match virtual-address 165.199.5.80 tcp eq https
policy-map type loadbalance first-match BIS-POLICY-HTTPS
class class-default
sticky-serverfarm STICKY-BIS-CLIENT-SSL
policy-map type loadbalance first-match FORMS-POLICY-HTTPS
class class-default
sticky-serverfarm STICKY-FORMS-CLIENT-SSL
policy-map type loadbalance first-match PORTAL-POLICY-HTTP
class class-default
sticky-serverfarm STICKY-PORTAL-CLIENT
policy-map type loadbalance first-match PORTAL-POLICY-HTTPS
class class-default
sticky-serverfarm STICKY-PORTAL-CLIENT-SSL
policy-map type loadbalance first-match RPORTAL-POLICY-HTTPS
class class-default
sticky-serverfarm STICKY-RPORTAL-CLIENT-SSL
policy-map multi-match PPE-MULTI-MATCH
class BIS-CLASS-HTTPS
loadbalance vip inservice
loadbalance policy BIS-POLICY-HTTPS
loadbalance vip icmp-reply
ssl-proxy server BIS-SSL-PROXY
class PORTAL-CLASS-HTTPS
loadbalance vip inservice
loadbalance policy PORTAL-POLICY-HTTPS
loadbalance vip icmp-reply
ssl-proxy server PORTAL-SSL-PROXY
class RPORTAL-CLASS-HTTPS
loadbalance vip inservice
loadbalance policy RPORTAL-POLICY-HTTPS
loadbalance vip icmp-reply
ssl-proxy server RPORTAL-SSL-PROXY
class FORMS-CLASS-HTTPS
loadbalance vip inservice
loadbalance policy FORMS-POLICY-HTTPS
loadbalance vip icmp-reply
ssl-proxy server FORMS-SSL-PROXY
class PORTAL-CLASS-HTTP
loadbalance vip inservice
loadbalance policy PORTAL-POLICY-HTTP
loadbalance vip icmp-reply
interface vlan 5
description Client-Side
bridge-group 1
access-group input TRAFFIC
service-policy input PPE-MULTI-MATCH
no shutdown
interface vlan 105
description Server-Side
bridge-group 1
access-group input TRAFFIC
no shutdown
interface bvi 1
ip address 165.199.5.9 255.255.255.0
description Client and Server Bridge Group 1
no shutdown
ip route 0.0.0.0 0.0.0.0 165.199.5.1
