ACE balancing problems

p.diaz · ‎12-23-2010

Hi everyone.

We have a customer who has a server farm formed by 3 servers with the following real ip address:

10.10.24.5-6-7 and a virtual 10.10.24.3 as configured in the ace module.

We found the following behavior in the session number of the servers. We can conclude that there is a server with much more sessions than the others (10.10.24.6):

Can sombody help me telling why can happen that?

I am attaching the ACE config as a reference

Thanks

-----------------------------------------------------------------------------------------------------------------

ACE-DIGENERAL/OCS# sh serverfarm Herramientas_Col

serverfarm : Herramientas_Col, type: HOST

total rservers : 3

---------------------------------

----------connections-----------

real weight state current total

---+---------------------+------+------------+----------+--------------------

rserver: SP1

10.10.24.5:0 8 OPERATIONAL 390 296043280

rserver: SP2

10.10.24.6:0 8 OPERATIONAL 1003 3371471400

rserver: SP3

10.10.24.7:0 8 OPERATIONAL 354 164816790

Como se puede observar el sever 10.10.24.6 posee mas del doble de conexiones que los otros 2.

5. En el siguiente pantallazo también se observan conexiones detalladas y los puertos por donde habla:

ACE-DIGENERAL/OCS# sh conn serverfarm Herramientas_Col

conn-id np dir proto vlan source destination state

----------+--+---+-----+----+---------------------+---------------------+------+

70 1 in TCP 951 10.10.22.13:3837 10.10.24.3:80 ESTAB

17239 1 out TCP 324 10.10.24.7:80 10.10.22.13:3837 ESTAB

76 1 in TCP 951 10.83.21.32:1419 10.10.24.3:80 ESTAB

5531 1 out TCP 324 10.10.24.6:80 10.83.21.32:1419 ESTAB

95 1 in TCP 951 10.20.7.51:1702 10.10.24.3:80 ESTAB

16237 1 out TCP 324 10.10.24.6:80 10.20.7.51:1702 ESTAB

98 1 in TCP 951 10.80.31.55:3188 10.10.24.3:80 ESTAB

11995 1 out TCP 324 10.10.24.6:80 10.80.31.55:3188 ESTAB

32749 1 in TCP 951 10.80.21.23:1926 10.10.24.3:80 ESTAB

108 1 out TCP 324 10.10.24.7:80 10.80.21.23:1926 ESTAB

110 1 in TCP 951 10.25.14.231:1705 10.10.24.3:80 ESTAB

37994 1 out TCP 324 10.10.24.6:80 10.25.14.231:1705 ESTAB

7438 1 in TCP 951 10.31.102.32:2329 10.10.24.3:80 ESTAB

141 1 out TCP 324 10.10.24.7:80 10.31.102.32:2329 ESTAB

31247 1 in TCP 951 10.81.36.32:1650 10.10.24.3:80 ESTAB

151 1 out TCP 324 10.10.24.5:80 10.81.36.32:1650 ESTAB

176 1 in TCP 951 10.20.208.124:2598 10.10.24.3:80 ESTAB

13219 1 out TCP 324 10.10.24.7:80 10.20.208.124:2598 ESTAB

32576 1 in TCP 951 10.233.9.40:1577 10.10.24.3:80 ESTAB

233 1 out TCP 324 10.10.24.6:80 10.233.9.40:1577 ESTAB

27499 1 in TCP 951 10.218.16.28:2902 10.10.24.3:80 ESTAB

244 1 out TCP 324 10.10.24.5:80 10.218.16.28:2902 ESTAB

248 1 in TCP 951 10.85.19.55:1540 10.10.24.3:80 ESTAB

14014 1 out TCP 324 10.10.24.7:80 10.85.19.55:1540 ESTAB

27166 1 in TCP 951 10.25.22.90:1766 10.10.24.3:80 ESTAB

254 1 out TCP 324 10.10.24.6:80 10.25.22.90:1766 ESTAB

380 1 in TCP 951 10.23.22.62:1855 10.10.24.3:80 ESTAB

11563 1 out TCP 324 10.10.24.6:80 10.23.22.62:1855 ESTAB

397 1 in TCP 951 10.212.35.30:1540 10.10.24.3:80 ESTAB

15491 1 out TCP 324 10.10.24.7:80 10.212.35.30:1540 ESTAB

35588 1 in TCP 951 10.100.30.5:1773 10.10.24.3:80 ESTAB

405 1 out TCP 324 10.10.24.6:80 10.100.30.5:1773 ESTAB

31392 1 in TCP 951 10.216.27.41:1524 10.10.24.3:80 ESTAB

449 1 out TCP 324 10.10.24.6:80 10.216.27.41:1524 ESTAB

592 1 in TCP 951 10.25.21.219:1364 10.10.24.3:80 ESTAB

2988 1 out TCP 324 10.10.24.5:80 10.25.21.219:1364 ESTAB

614 1 in TCP 951 10.25.42.221:1517 10.10.24.3:80 ESTAB

18877 1 out TCP 324 10.10.24.6:80 10.25.42.221:1517 ESTAB

21553 1 in TCP 951 10.80.39.123:1634 10.10.24.3:80 ESTAB

652 1 out TCP 324 10.10.24.6:80 10.80.39.123:1634 ESTAB

13640 1 in TCP 951 10.206.2.34:1385 10.10.24.3:80 ESTAB

708 1 out TCP 324 10.10.24.6:80 10.206.2.34:1385 ESTAB

26959 1 in TCP 951 10.100.30.7:1289 10.10.24.3:80 ESTAB

719 1 out TCP 324 10.10.24.5:80 10.100.30.7:1289 ESTAB

29277 1 in TCP 951 10.100.202.50:1248 10.10.24.3:80 ESTAB

758 1 out TCP 324 10.10.24.5:80 10.100.202.50:1248 ESTAB

6185 1 in TCP 951 10.25.27.222:1497 10.10.24.3:80 ESTAB

760 1 out TCP 324 10.10.24.6:80 10.25.27.222:1497 ESTAB

767 1 in TCP 951 10.97.21.28:1821 10.10.24.3:80 ESTAB

23511 1 out TCP 324 10.10.24.7:80 10.97.21.28:1821 ESTAB

826 1 in TCP 951 10.31.105.140:3810 10.10.24.3:80 ESTAB

13460 1 out TCP 324 10.10.24.6:80 10.31.105.140:3810 ESTAB

21987 1 in TCP 951 10.25.31.213:1855 10.10.24.3:80 ESTAB

839 1 out TCP 324 10.10.24.5:80 10.25.31.213:1855 ESTAB

874 1 in TCP 951 10.88.29.27:1503 10.10.24.3:80 ESTAB

29839 1 out TCP 324 10.10.24.6:80 10.88.29.27:1503 ESTAB

945 1 in TCP 951 10.27.122.13:1286 10.10.24.3:80 ESTAB

32298 1 out TCP 324 10.10.24.6:80 10.27.122.13:1286 ESTAB

24330 1 in TCP 951 10.40.21.50:2368 10.10.24.3:80 ESTAB

954 1 out TCP 324 10.10.24.6:80 10.40.21.50:2368 ESTAB

961 1 in TCP 951 10.80.26.76:1414 10.10.24.3:80 ESTAB

11176 1 out TCP 324 10.10.24.5:80 10.80.26.76:1414 ESTAB

28989 1 in TCP 951 10.91.22.38:1408 10.10.24.3:80 ESTAB

985 1 out TCP 324 10.10.24.5:80 10.91.22.38:1408 ESTAB

1006 1 in TCP 951 10.217.4.20:1522 10.10.24.3:80 ESTAB

26946 1 out TCP 324 10.10.24.5:80 10.217.4.20:1522 ESTAB

8360 1 in TCP 951 10.11.3.28:1679 10.10.24.3:80 ESTAB

1020 1 out TCP 324 10.10.24.6:80 10.11.3.28:1679 ESTAB

9498 1 in TCP 951 10.25.42.221:1519 10.10.24.3:80 ESTAB

1031 1 out TCP 324 10.10.24.6:80 10.25.42.221:1519 ESTAB

18510 1 in TCP 951 10.165.55.51:1232 10.10.24.3:80 ESTAB

1072 1 out TCP 324 10.10.24.7:80 10.165.55.51:1232 ESTAB

5583 1 in TCP 951 10.25.14.12:2086 10.10.24.3:80 ESTAB

1142 1 out TCP 324 10.10.24.6:80 10.25.14.12:2086 ESTAB

39713 1 in TCP 951 10.25.36.58:1663 10.10.24.3:80 ESTAB

1144 1 out TCP 324 10.10.24.7:80 10.25.36.58:1663 ESTAB

8601 1 in TCP 951 10.217.26.34:1677 10.10.24.3:80 ESTAB

1167 1 out TCP 324 10.10.24.6:80 10.217.26.34:1677 ESTAB

17209 1 in TCP 951 10.165.40.45:1526 10.10.24.3:80 ESTAB

1173 1 out TCP 324 10.10.24.5:80 10.165.40.45:1526 ESTAB

18708 1 in TCP 951 10.31.105.137:3714 10.10.24.3:80 ESTAB

1175 1 out TCP 324 10.10.24.6:80 10.31.105.137:3714 ESTAB

1180 1 in TCP 951 10.201.18.40:4777 10.10.24.3:80 ESTAB

6528 1 out TCP 324 10.10.24.6:80 10.201.18.40:4777 ESTAB

1214 1 in TCP 951 10.31.104.46:1501 10.10.24.3:80 ESTAB

5924 1 out TCP 324 10.10.24.6:80 10.31.104.46:1501 ESTAB

1228 1 in TCP 951 10.231.37.32:1161 10.10.24.3:80 ESTAB

15171 1 out TCP 324 10.10.24.6:80 10.231.37.32:1161 ESTAB

28431 1 in TCP 951 10.25.5.76:2317 10.10.24.3:80 ESTAB

1293 1 out TCP 324 10.10.24.5:80 10.25.5.76:2317 ESTAB

1328 1 in TCP 951 10.201.2.26:1293 10.10.24.3:80 ESTAB

19276 1 out TCP 324 10.10.24.7:80 10.201.2.26:1293 ESTAB

1356 1 in TCP 951 10.80.23.27:1396 10.10.24.3:80 ESTAB

4141 1 out TCP 324 10.10.24.6:80 10.80.23.27:1396 ESTAB

1368 1 in TCP 951 10.80.36.124:1428 10.10.24.3:80 ESTAB

19905 1 out TCP 324 10.10.24.6:80 10.80.36.124:1428 ESTAB

30280 1 in TCP 951 10.25.8.11:4836 10.10.24.3:80 ESTAB

1438 1 out TCP 324 10.10.24.6:80 10.25.8.11:4836 ESTAB

1478 1 in TCP 951 10.216.6.46:4153 10.10.24.3:80 ESTAB

12312 1 out TCP 324 10.10.24.6:80 10.216.6.46:4153 ESTAB

23389 1 in TCP 951 10.211.30.38:1593 10.10.24.3:80 ESTAB

1527 1 out TCP 324 10.10.24.6:80 10.211.30.38:1593 ESTAB

1562 1 in TCP 951 10.90.21.58:2889 10.10.24.3:80 ESTAB

36398 1 out TCP 324 10.10.24.7:80 10.90.21.58:2889 ESTAB

1587 1 in TCP 951 10.84.22.29:2121 10.10.24.3:80 ESTAB

37031 1 out TCP 324 10.10.24.6:80 10.84.22.29:2121 ESTAB

1624 1 in TCP 951 10.25.21.218:1465 10.10.24.3:80 ESTAB

4941 1 out TCP 324 10.10.24.6:80 10.25.21.218:1465 ESTAB

Nelson Galimba · ‎01-05-2011

Hi,

I see you have not gotten any replies to this post. Please try moving this post to content networking. If moving it does not work ( I did not have permissions to move it), try re-posting it in content networking.

Thanks,

Nelson

chrhiggi · ‎01-26-2011

Hello!

A "show connection serverfarm Herramientas_Col detail" and "show sticky database group POOL3" would be handy in this situation. You have sticky configured which will intentionally throw off the loadbalancing predictor. My guess at this point is that rserver SP2 might not close connections in the same manner that SP1 and SP3 do. If that was true, that would result in a longer connection time, which means the sticky database would not idle out as fast, hence more connection for SP2.

Regards,

Chris