ACE-30 - hash address source predictor vs dead real servers

[Andreas Rosengart]

HI Everyone,

did not find anything with the search (if you know some other thread just let me know :-D ).

I have a general question that I was not able to find in the cisco documentation about the predictors.

We do have a serverfarm which is experiencing very high unequal loadbalancing between realserver after new server migration. The customer did not complain before that adding of the new realservers (maybe he didn't know it anyway but does not matter right now)

Serverfarm config and show output: (Names and IPs are of course changed

serverfarm host FARMXXX
  predictor hash address source
  rserver 1-1-1-1 36000
    probe PROBE_TCP_36000
    inservice
  rserver 1-1-1-1 36003
    probe PROBE_TCP_36003
    inservice
  rserver 2-2-2-2 36000
    probe PROBE_TCP_36000
    inservice
  rserver 2-2-2-2 36003
    probe PROBE_TCP_36003
    inservice
  rserver 3-3-3-3 36000
    probe PROBE_TCP_36000
    inservice
  rserver 3-3-3-3 36003
    probe PROBE_TCP_36003
    inservice
  rserver 4-4-4-4 36000
    probe PROBE_TCP_36000
    inservice
  rserver 4-4-4-4 36003
    probe PROBE_TCP_36003
    inservice
  rserver 5-5-5-5 36000
    probe PROBE_TCP_36000
    inservice
  rserver 5-5-5-5 36003
    probe PROBE_TCP_36003
    inservice
  rserver 6-6-6-6 36000
    probe PROBE_TCP_36000
    inservice
  rserver 6-6-6-6 36003
    probe PROBE_TCP_36003
    inservice
  rserver 7-7-7-7 36000
    probe PROBE_TCP_36000
    inservice
  rserver 8-8-8-8 36000
    probe PROBE_TCP_36000
    inservice
  rserver 9-9-9-9 36000
    probe PROBE_TCP_36000
    inservice
  rserver 10-10-10-10 36000
    probe PROBE_TCP_36000
    inservice
  rserver 11-11-11-11 36000
    probe PROBE_TCP_36000
    inservice
  rserver 12-12-12-12 36000
    probe PROBE_TCP_36000
    inservice
  rserver 13-13-13-13 36000
    probe PROBE_TCP_36000
    inservice
  rserver 14-14-14-14 36000
    probe PROBE_TCP_36000
    inservice

                                                ----------connections-----------
       real                  weight state        current    total      failures
   ---+---------------------+-----+------------+----------+----------+---------
   rserver: 1-1-1-1
       1.1.1.1:36000  8   PROBE-FAILED    0          3206682    836339
       1.1.1.1:36003  8   PROBE-FAILED    0          3549784    968705
   rserver: 2-2-2-2
       2.2.2.2:36000  8   PROBE-FAILED    0          2976501    699274
       2.2.2.2:36003  8   PROBE-FAILED    0          3255863    798423
   rserver: 3-3-3-3
       3.3.3.3:36000   8   PROBE-FAILED    0          3423583    839763
       3.3.3.3:36003   8   PROBE-FAILED    0          3199739    737812
   rserver: 4-4-4-4
       4.4.4.4:36000   8   PROBE-FAILED    0          2958373    692051
       4.4.4.4:36003   8   PROBE-FAILED    0          3226513    766236
   rserver: 5-5-5-5
       5.5.5.5:36000   8   PROBE-FAILED    0          3368001    758481
       5.5.5.5:36003   8   PROBE-FAILED    0          3534090    780365
   rserver: 6-6-6-6
       6.6.6.6:36000    8   PROBE-FAILED    0          3450448    894396
       6.6.6.6:36003    8   PROBE-FAILED    0          3333548    772662
   rserver: 7-7-7-7
       7.7.7.7:36000    8   OPERATIONAL     975        2664203    138837
   rserver: 8-8-8-8
       8.8.8.8:36000    8   OPERATIONAL     85         254861     17010
   rserver: 9-9-9-9
       9.9.9.9:36000   8   OPERATIONAL     69         266892     11825
   rserver: 10-10-10-10
       10.10.10.10:36000   8   OPERATIONAL     200        520065     20135
   rserver: 11-11-11-11
       11.11.11.11:36000   8   OPERATIONAL     141        489838     23293
   rserver: 12-12-12-12
       12.12.12.12:36000   8   OPERATIONAL     164        504719     18928
   rserver: 13-13-13-13
       13.13.13.13:36000   8   OPERATIONAL     111        248177     11419
   rserver: 14-14-14-14
       14.14.14.14:36000   8   OPERATIONAL     105        243694     7693

You can see the old servers are still configured (for fast rollback of the customer). You also can see that the first server after the dead old servers getting a huge load. I now wonder if the Servers which are still configured and just probe failed are considered to be used in the hashing algorithm and because they are offline the next active server gets those connections which are calculated (by the algorithm)?

Did not find any documentation which states that only operational servers are used for the calculation, hope someone has some information about that.

We already recommended the customer to request a change of the predictor to be leastconns with slowstart option because he already has stickyfarm configured in this setup as well (we already checked if this behavior is coming form old stickiness but all sessions are cleared overnight so stickiness is unequal too, but because of the decisions of the hashing algorithm)

Regards Andy