topic ACE Error Message in Application Networking

topic ACE Error Message in Application Networking https://community.cisco.com/t5/application-networking/ace-error-message/m-p/1786918#M35463 <P>I have 2 servers under a server farm ,the servers are up and operation and  even the probe is successfull.But in the logging I keep getting this error message ,when I did my research ,cisco website states if we continuosuly get this error ,please contact Cisco TAC.</P><P>Experts can you please let me know ,why we are getting this error message</P><P></P><P></P><P></P><P>%ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P></P><P></P><P>Admin# sh probe PROBE_TCP_HTTPS </P><P></P><P> probe       : PROBE_TCP_HTTPS</P><P> type        : TCP</P><P> state       : ACTIVE</P><P>----------------------------------------------</P><P>   port      : 443     address     : 0.0.0.0         addr type  : -           </P><P>   interval  : 5       pass intvl  : 5               pass count : 3    </P><P>   fail count: 3       recv timeout: 10   </P><P>                ------------------ probe results ------------------</P><P>   associations ip-address      port  porttype probes   failed   passed   health </P><P>   ------------ ---------------+-----+--------+--------+--------+--------+------ </P><P>   serverfarm  : devservers</P><P>     real      : web2[0]</P><P>                10.12.13.83    443   PROBE    37181    34207    2974     SUCCESS</P><P></P><P>     real      : jweb1[0]</P><P>                10.12.13.82    443   PROBE    37183    34209    2974     SUCCESS</P><P></P><P>Admin# sh serverfarm devjumpservers </P><P></P><P>   Codes: L - local,   R - remote</P><P></P><P> serverfarm     : devservers, type: HOST</P><P> total rservers : 2</P><P> state          : ACTIVE</P><P> DWS state      : DISABLED</P><P> ---------------------------------</P><P>                                                ----------connections-----------</P><P>       real                  weight state        current    total      failures </P><P>   ---+---------------------+-----+------------+----------+----------+---------</P><P>   rserver: web2</P><P>       10.12.13.83:0        8   OPERATIONAL     0          75         0</P><P>   rserver: web1</P><P>       10.12.13.82:0        8   OPERATIONAL     0          40         1</P> Thu, 22 Sep 2011 02:48:44 GMT 12pratham 2011-09-22T02:48:44Z ACE Error Message https://community.cisco.com/t5/application-networking/ace-error-message/m-p/1786918#M35463 <P>I have 2 servers under a server farm ,the servers are up and operation and  even the probe is successfull.But in the logging I keep getting this error message ,when I did my research ,cisco website states if we continuosuly get this error ,please contact Cisco TAC.</P><P>Experts can you please let me know ,why we are getting this error message</P><P></P><P></P><P></P><P>%ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.83 on port 443, internal error: failed to setup a socket</P><P> %ACE-3-251006: Health probe failed for server 10.12.13.82 on port 443, internal error: failed to setup a socket</P><P></P><P></P><P>Admin# sh probe PROBE_TCP_HTTPS </P><P></P><P> probe       : PROBE_TCP_HTTPS</P><P> type        : TCP</P><P> state       : ACTIVE</P><P>----------------------------------------------</P><P>   port      : 443     address     : 0.0.0.0         addr type  : -           </P><P>   interval  : 5       pass intvl  : 5               pass count : 3    </P><P>   fail count: 3       recv timeout: 10   </P><P>                ------------------ probe results ------------------</P><P>   associations ip-address      port  porttype probes   failed   passed   health </P><P>   ------------ ---------------+-----+--------+--------+--------+--------+------ </P><P>   serverfarm  : devservers</P><P>     real      : web2[0]</P><P>                10.12.13.83    443   PROBE    37181    34207    2974     SUCCESS</P><P></P><P>     real      : jweb1[0]</P><P>                10.12.13.82    443   PROBE    37183    34209    2974     SUCCESS</P><P></P><P>Admin# sh serverfarm devjumpservers </P><P></P><P>   Codes: L - local,   R - remote</P><P></P><P> serverfarm     : devservers, type: HOST</P><P> total rservers : 2</P><P> state          : ACTIVE</P><P> DWS state      : DISABLED</P><P> ---------------------------------</P><P>                                                ----------connections-----------</P><P>       real                  weight state        current    total      failures </P><P>   ---+---------------------+-----+------------+----------+----------+---------</P><P>   rserver: web2</P><P>       10.12.13.83:0        8   OPERATIONAL     0          75         0</P><P>   rserver: web1</P><P>       10.12.13.82:0        8   OPERATIONAL     0          40         1</P> Thu, 22 Sep 2011 02:48:44 GMT https://community.cisco.com/t5/application-networking/ace-error-message/m-p/1786918#M35463 12pratham 2011-09-22T02:48:44Z ACE Error Message https://community.cisco.com/t5/application-networking/ace-error-message/m-p/1786919#M35464 <HTML><HEAD></HEAD><BODY><P> Hello Pratham-</P><P></P><P> You have an interval of 5 seconds with a recieve timeout of 10 seconds, which means you have an opportunity for overlapping probes.  If this occurs, you will have skipped probes, thus, the error that you cannot open a new port if an existing one is open.  There are 2 possibilities here, you are trying to send over 2000 probes at once, or your probes are overlapping.  Grab a show probe PROBE_TCP_HTTPS detail and paste it in for me - we can verify it.  As well, grab show system resource usage all.</P><P></P><P>Regards,</P><P>Chris Higgins</P></BODY></HTML> Mon, 26 Sep 2011 17:58:32 GMT https://community.cisco.com/t5/application-networking/ace-error-message/m-p/1786919#M35464 chrhiggi 2011-09-26T17:58:32Z ACE Error Message https://community.cisco.com/t5/application-networking/ace-error-message/m-p/1786920#M35465 <HTML><HEAD></HEAD><BODY><P>Chris </P><P></P><P>below are the outputs</P><P></P><P>Admin# sh probe PROBE_TCP_HTTPS detail </P><P></P><P> probe       : PROBE_TCP_HTTPS</P><P> type        : TCP</P><P> state       : ACTIVE</P><P> description : " This probe is for HTTPS 443 "</P><P>----------------------------------------------</P><P>   port      : 443     address     : 0.0.0.0         addr type  : -           </P><P>   interval  : 5       pass intvl  : 5               pass count : 3    </P><P>   fail count: 3       recv timeout: 10   </P><P>   conn termination : GRACEFUL  </P><P>   expect offset    : 0         , open timeout     : 10        </P><P>   expect regex     : -</P><P>   send data        : -</P><P>                ------------------ probe results ------------------</P><P>   associations ip-address      port  porttype probes   failed   passed   health </P><P>   ------------ ---------------+-----+--------+--------+--------+--------+------ </P><P>   serverfarm  : devservers</P><P>     real      : web2[0]</P><P>                10.12.13.83    443   PROBE    117554   44929    72625    SUCCESS</P><P></P><P>   Socket state        : CLOSED</P><P>   No. Passed states   : 4         No. Failed states : 4</P><P>   No. Probes skipped  : 10         Last status code  : 0</P><P>   No. Out of Sockets  : 0         No. Internal error: 0</P><P>   Last disconnect err :  - </P><P>   Last probe time     : Mon Sep 26 11:16:28 2011</P><P>   Last fail time      : Wed Sep 21 20:56:40 2011</P><P>   Last active time    : Thu Sep 22 11:45:55 2011</P><P></P><P>     real      : web1[0]</P><P>                10.12.13.82    443   PROBE    117555   44943    72612    SUCCESS</P><P></P><P>   Socket state        : CLOSED</P><P>   No. Passed states   : 5         No. Failed states : 5</P><P>   No. Probes skipped  : 9         Last status code  : 0</P><P>   No. Out of Sockets  : 0         No. Internal error: 0</P><P>   Last disconnect err :  - </P><P>   Last probe time     : Mon Sep 26 11:16:29 2011</P><P>   Last fail time      : Sat Sep 24 22:02:36 2011</P><P>   Last active time    : Sat Sep 24 22:03:31 2011</P><P></P><P></P><P></P><P>Admin# sh resource usage all </P><P>                                                     Allocation</P><P>        Resource         Current       Peak        Min        Max       Denied</P><P>-------------------------------------------------------------------------------</P><P>Context: Admin</P><P>  conc-connections             19         57        100    2000000          0</P><P>  mgmt-connections              4         26        100     100000          0</P><P>  proxy-connections             0          4          0     262143          0</P><P>  xlates                        0          0          0      65535          0</P><P>  acc-connections               0          0          0        105          0</P><P>  bandwidth                 17985     123221    2500000  250000000          0</P><P>    throughput              17756     110508    1250000  125000000          0</P><P>    mgmt-traffic rate         229      12713    1250000  125000000          0</P><P>  connection rate              12         75        100     120000          0</P><P>  ssl-connections rate          0          0          0       7500          0</P><P>  mac-miss rate                 0          1          0       2000          0</P><P>  inspect-conn rate             0          0          0      40000          0</P><P>  http-comp rate                0          0          0  262144000          0</P><P>  acl-memory                34544      34544          0   48822192          0</P><P>  sticky                        0          3      81920     737280          0</P><P>  regexp                      607        607          0    1048576          0</P><P>  syslog buffer           4194304    4194304          0    4194304          0</P><P>  syslog rate                   0         31          0     100000          0</P><P>Context: production</P><P>  conc-connections              0          0          0    1999900          0</P><P>  mgmt-connections              0          0          0      99900          0</P><P>  proxy-connections             0          0          0     262143          0</P><P>  xlates                        0          0          0      65535          0</P><P>  acc-connections               0          0          0        105          0</P><P>  bandwidth                     0          0          0  247500000          0</P><P>    throughput                  0          0          0  123750000          0</P><P>    mgmt-traffic rate           0          0          0  123750000          0</P><P>  connection rate               0          0          0     119900          0</P><P>  ssl-connections rate          0          0          0       7500          0</P><P>  mac-miss rate                 0          0          0       2000          0</P><P>  inspect-conn rate             0          0          0      40000          0</P><P>  http-comp rate                0          0          0  262144000          0</P><P>  acl-memory                 6976       7040          0   48822192          0</P><P>  sticky                        0          0      81920     737280          0</P><P>  regexp                      457        457          0    1048576          0</P><P>  syslog buffer                 0          0          0    4194304          0</P><P>  syslog rate                   0          0          0     100000          0</P></BODY></HTML> Mon, 26 Sep 2011 18:25:22 GMT https://community.cisco.com/t5/application-networking/ace-error-message/m-p/1786920#M35465 12pratham 2011-09-26T18:25:22Z ACE Error Message https://community.cisco.com/t5/application-networking/ace-error-message/m-p/1786921#M35466 <HTML><HEAD></HEAD><BODY><P>Hello Pratham-</P><P></P><P>  You are fine on probes/second (the management peak should not add up to over 2000 across all contexts - If it did, it would be possible that was probe traffic and more investigation would need to be done.)</P><P></P><P> You are overlapping probes. As you can see, "No. Probes Skipped" have been incrimented.  What you want to do is decrease the recieve timeout under the probe to be lower than 5 seconds (4 seconds should be fine).</P><P></P><P>Regards,</P><P>Chris Higgins</P></BODY></HTML> Mon, 26 Sep 2011 18:52:57 GMT https://community.cisco.com/t5/application-networking/ace-error-message/m-p/1786921#M35466 chrhiggi 2011-09-26T18:52:57Z