cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
7454
Views
0
Helpful
10
Replies

Issue with regexes in http health probes on ACE 4710

akshay2196
Level 1
Level 1

Folks,

We're currently experiencing fairly bizarre behavior when attempting to set up http probes that expect a regexp.  Namely, if we specify a regexp, the probe *always* passes, regardless of status code and regardless of whether or not the message actually matches the pattern.  Doing 'no expect regexp' fixes this behavior (by which I mean that the 'expect status' rules work again). 

We haven't noticed until now because this is the first time we've tried to set up a probe that does this.  Are we missing something?  Is this a known issue with our current firmware version?

Sincerely,

Patrick T. Ramsey

# show run probe | begin HTTP-nfscheck | end regex

Generating configuration....

probe http HTTP-nfscheck

  description Simple HTTP probe to check nfs mount health

  port 80

  interval 15

  passdetect interval 20

  request method head url /nfs-health-check/

  open 1

  expect regex "^ureytgraeuikghfdjg$"

# sh ver
Cisco Application Control Software (ACSW)
Copyright (c) 1985-2009 by Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
Software
  loader:    Version 0.95.1
  system:    Version A3(2.4) [build 3.0(0)A3(2.4) adbuild_11:46:02-2009/09/27_/auto/adbu-rel2/rel_a3_2_3_throttle/REL_3_0_0_A3_2
_4]
  system image file: (hd0,1)/c4710ace-mz.A3_2_4.bin
  Device Manager version 1.2 (0) 20090925:1550
  installed license: no feature license is installed
Hardware
  cpu info:
    Motherboard:
        number of cpu(s): 2
    Daughtercard:
        number of cpu(s): 16
  memory info:
    total: 6226388 kB, free: 3972668 kB
    shared: 0 kB, buffers: 22020 kB, cached 0 kB
  cf info:
    filesystem: /dev/hdb2
    total: 861668 kB, used: 728656 kB, available: 89240 kB
last boot reason:  Unknown
configuration register:  0x1
ldbottom kernel uptime is 325 days 3 hours 46 minute(s) 43 second(s)
10 Replies 10

Ivan Kovacevic
Cisco Employee
Cisco Employee

Are you trying to parse a large page? There were some problems with expect regex on ACE like CSCtg13892, so we introduced a new way of parsing content on A3(2.6). You should upgrade to this version and check ACE behavior on that version.

No.  This page is 35 characters long, and the problem persists even if the "HEAD" method is used.

Can you check if the server response contains the Content-Length header filed? Without it the ACE will not attempt to parse the content.

The server response did not contain Content-Length.  I was briefly hopeful, but I've altered the health check to return said header and the behavior of the probe has not changed: while both rservers in the server farm are reporting an error state, the probe is being marked successful.

Also, the documentation at http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA1_7_/command/reference/probe.html does not seem to say anything about the requirement that the Content-Length header be present (except for the hash option, which we're not using).  Is this a new requirement?  Also, why is it that the probe succeeds if the content isn't parsed?  If there's no usable content, shouldn't the regex fail?

Many thanks,

Patrick T. Ramsey

It is documented in the config guide and it is not a new requirement:

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA3_1_0/configuration/slb/guide/probe.html#wp1075645

"For HTTP or HTTPS probes, the server response must include the Content-Length header for the expect regex command to function. Otherwise, the probe does not attempt to parse the regex."

So, if the Content-Length is missing, this functionality is not active and no decision is made based on it.

BTW, I am not sure what are you trying to do with HEAD method, as the server will not return message-body with it. For this type of probe you need to use method GET.

Hi Ivan,

    What if the size is large? I experienced that my HTTP probe, the server return content around 6K bytes, but the packets are split to 4 ethernet payload due to 1500 MTU. And the regexp is split to first and second packets. the ACE never match the regex and failed. I changed to only world in the first packet, it worked. So, the ACE seem to not assembly entire HTTP payload or having fixed buffer i.e. 2048 or 4096 bytes. It just grab the first TCP packet which has HTTP header (return code), then parse for regex.

Note: My OS is A2 (3.4) ACE20 module.

parry_worley
Level 1
Level 1

I have seen similar behavior where an HTTP probe still failed even when properly setting the Content-Length header using the probe config below:

probe http HA-script-second-15101
  port 15101
  interval 10
  faildetect 2
  passdetect interval 30
  passdetect count 2
  receive 5
  request method get url /status.html
  expect regex "responding"

I was able to get this to work by setting up a TCP probe using the "send-data" command to specify my GET.

probe tcp HA-script-15101
  port 15101
  interval 2
  faildetect 1
  passdetect interval 2
  passdetect count 1
  receive 5
  send-data "GET /status.html"
  expect regex "responding"

My suspicion is there are some incompatibility between HTTP servers (possibly HTTP/1.1 vs. HTTP/1.0?) where the HTTP probes don't always work. My suggestion is to always use the TCP with the send-data to guarantee compatibility.

Also - Cisco's documentation seems to differ a bit on whether the Content-Length is actually needed. One says it is needed when using the hash function:


http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA1_7_/command/reference/probe.html#wp1005411


The server response must include the Content-Length header for the hash command to function. Otherwise, the probe does not attempt to parse the hash value.

Another suggests in general this is required:

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA3_1_0/configuration/slb/guide/probe.html

For HTTP or HTTPS probes, the server response must include the Content-Length header for the expect regex command to function. Otherwise, the probe does not attempt to parse the regex.

I would like to test this out in a lab to determine the actual definitive behavior.

I hope this helps.

Jorge Bejarano
Level 4
Level 4

Hi Johhny,

Your applications need to enter the Content-Length.

Also you are missing the expect status, please try with "expect status 0 999"

Here you have an old thread which might help:

https://supportforums.cisco.com/thread/130727

Hope this helps a little bit.

Jorge

songtong-cisco
Level 1
Level 1

Hi All,

    Does the "expect regex" in HTTP probe parse HTTP header & HTTP content? Or just HTTP content?

Vishal Babrekar
Level 1
Level 1

I also went through a similar issue in which we need to probe the real server PESERVER01 and if the real server replies with the keyword "PE Server" in the HTTP content then the probe should be passed successful.

In my case the real server was listening on port 32776 for HTTP service so we configured the serverfarm as below,

serverfarm host SF-TEST-32776

  description SF-TEST-32776

  failaction purge

  probe PE-SERVER-STRING

  rserver PESERVER01 32776

    inservice

And the TCP probe as below,

probe tcp PE-SERVER-STRING

  port 32776

  send-data GET /IOR/ping HTTP/1.1      <<== command should not be in inverted  commas

  expect regex "PE Server"

The above probe worked really well and when we checked the probe status it was marking as success. I also tried changing the regex from "PE Server" to "Vishal12345" and it was failing as expected because there was no such keyword in the HTTP content.

==================================================================================

T2-LB02# sh probe PE-SERVER-STRING

probe       : PE-SERVER-STRING

type        : TCP

state       : ACTIVE

----------------------------------------------

   port      : 32776   address     : 0.0.0.0         addr type  : -

   interval  : 15      pass intvl  : 60              pass count : 3

   fail count: 3       recv timeout: 10

                ------------------ probe results ------------------

   associations ip-address      port  porttype probes   failed   passed   health

   ------------ ---------------+-----+--------+--------+--------+--------+------

   serverfarm  : SF-TEST-32776

     real      : PESERVER01[32776]

                10.10.10.1    32776 PROBE    105      0        105      SUCCESS

==================================================================================

I was struggling with this issue from long time. Even raised couple of Cisco TAC cases with no luck. The most important thing here is to identify the exact command to be send to real server like GET /IOR/ping HTTP/1.1 that we used here.

To collect this command I did packet capture on one of the client machine and then tried to open the URL from real server which can return the string "PE Server". Then analyzed the captures in Wireshark and checked the HTTP data with follow the TCP stream option in which I seen the below data, which gives the command to be send in probe as well as the string we should expect.

==================================================================================

GET /IOR/ping HTTP/1.1

User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2

Host: 10.144.70.85:32776

Accept: */*

HTTP/1.0 200 OK

Content-type: text/html

Ping

PE Server

WRVFKO11 [Win32 Server Production (3 silos) (Oracle Blob 512 MB) -- {dap451.007.028 dap451.004.002 pe451.003.010x pui451.003.010  pui451.001.004} Mar  9 2012 15:07:53 en ]

===================================================================================

Please try this and see if it helps you.

Thanks,

Vishal Babrekar

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: