ACE Error Message

12pratham · ‎09-21-2011

I have 2 servers under a server farm ,the servers are up and operation and even the probe is successfull.But in the logging I keep getting this error message ,when I did my research ,cisco website states if we continuosuly get this error ,please contact Cisco TAC.

Experts can you please let me know ,why we are getting this error message

%ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket

%ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket

%ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket

%ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket

%ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket

%ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket

%ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket

%ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket

%ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket

Admin# sh probe PROBE_TCP_HTTPS

probe : PROBE_TCP_HTTPS

type : TCP

state : ACTIVE

----------------------------------------------

port : 443 address : 0.0.0.0 addr type : -

interval : 5 pass intvl : 5 pass count : 3

fail count: 3 recv timeout: 10

------------------ probe results ------------------

associations ip-address port porttype probes failed passed health

------------ ---------------+-----+--------+--------+--------+--------+------

serverfarm : devservers

real : web2[0]

10.12.13.83 443 PROBE 37181 34207 2974 SUCCESS

real : jweb1[0]

10.12.13.82 443 PROBE 37183 34209 2974 SUCCESS

Admin# sh serverfarm devjumpservers

Codes: L - local, R - remote

serverfarm : devservers, type: HOST

total rservers : 2

state : ACTIVE

DWS state : DISABLED

---------------------------------

----------connections-----------

real weight state current total failures

---+---------------------+-----+------------+----------+----------+---------

rserver: web2

10.12.13.83:0 8 OPERATIONAL 0 75 0

rserver: web1

10.12.13.82:0 8 OPERATIONAL 0 40 1

chrhiggi · ‎09-26-2011

Hello Pratham-

You have an interval of 5 seconds with a recieve timeout of 10 seconds, which means you have an opportunity for overlapping probes. If this occurs, you will have skipped probes, thus, the error that you cannot open a new port if an existing one is open. There are 2 possibilities here, you are trying to send over 2000 probes at once, or your probes are overlapping. Grab a show probe PROBE_TCP_HTTPS detail and paste it in for me - we can verify it. As well, grab show system resource usage all.

Regards,

Chris Higgins

12pratham · ‎09-26-2011

Chris

below are the outputs

Admin# sh probe PROBE_TCP_HTTPS detail

probe : PROBE_TCP_HTTPS

type : TCP

state : ACTIVE

description : " This probe is for HTTPS 443 "

----------------------------------------------

port : 443 address : 0.0.0.0 addr type : -

interval : 5 pass intvl : 5 pass count : 3

fail count: 3 recv timeout: 10

conn termination : GRACEFUL

expect offset : 0 , open timeout : 10

expect regex : -

send data : -

------------------ probe results ------------------

associations ip-address port porttype probes failed passed health

------------ ---------------+-----+--------+--------+--------+--------+------

serverfarm : devservers

real : web2[0]

10.12.13.83 443 PROBE 117554 44929 72625 SUCCESS

Socket state : CLOSED

No. Passed states : 4 No. Failed states : 4

No. Probes skipped : 10 Last status code : 0

No. Out of Sockets : 0 No. Internal error: 0

Last disconnect err : -

Last probe time : Mon Sep 26 11:16:28 2011

Last fail time : Wed Sep 21 20:56:40 2011

Last active time : Thu Sep 22 11:45:55 2011

real : web1[0]

10.12.13.82 443 PROBE 117555 44943 72612 SUCCESS

Socket state : CLOSED

No. Passed states : 5 No. Failed states : 5

No. Probes skipped : 9 Last status code : 0

No. Out of Sockets : 0 No. Internal error: 0

Last disconnect err : -

Last probe time : Mon Sep 26 11:16:29 2011

Last fail time : Sat Sep 24 22:02:36 2011

Last active time : Sat Sep 24 22:03:31 2011

Admin# sh resource usage all

Allocation

Resource Current Peak Min Max Denied

-------------------------------------------------------------------------------

Context: Admin

conc-connections 19 57 100 2000000 0

mgmt-connections 4 26 100 100000 0

proxy-connections 0 4 0 262143 0

xlates 0 0 0 65535 0

acc-connections 0 0 0 105 0

bandwidth 17985 123221 2500000 250000000 0

throughput 17756 110508 1250000 125000000 0

mgmt-traffic rate 229 12713 1250000 125000000 0

connection rate 12 75 100 120000 0

ssl-connections rate 0 0 0 7500 0

mac-miss rate 0 1 0 2000 0

inspect-conn rate 0 0 0 40000 0

http-comp rate 0 0 0 262144000 0

acl-memory 34544 34544 0 48822192 0

sticky 0 3 81920 737280 0

regexp 607 607 0 1048576 0

syslog buffer 4194304 4194304 0 4194304 0

syslog rate 0 31 0 100000 0

Context: production

conc-connections 0 0 0 1999900 0

mgmt-connections 0 0 0 99900 0

proxy-connections 0 0 0 262143 0

xlates 0 0 0 65535 0

acc-connections 0 0 0 105 0

bandwidth 0 0 0 247500000 0

throughput 0 0 0 123750000 0

mgmt-traffic rate 0 0 0 123750000 0

connection rate 0 0 0 119900 0

ssl-connections rate 0 0 0 7500 0

mac-miss rate 0 0 0 2000 0

inspect-conn rate 0 0 0 40000 0

http-comp rate 0 0 0 262144000 0

acl-memory 6976 7040 0 48822192 0

sticky 0 0 81920 737280 0

regexp 457 457 0 1048576 0

syslog buffer 0 0 0 4194304 0

syslog rate 0 0 0 100000 0

chrhiggi · ‎09-26-2011

Hello Pratham-

You are fine on probes/second (the management peak should not add up to over 2000 across all contexts - If it did, it would be possible that was probe traffic and more investigation would need to be done.)

You are overlapping probes. As you can see, "No. Probes Skipped" have been incrimented. What you want to do is decrease the recieve timeout under the probe to be lower than 5 seconds (4 seconds should be fine).

Regards,

Chris Higgins