ACE makes all PROBES failed when 2000 users testing happened

[Yahshanulla S]

Hi,

We have ACE Appliance containing more than 10 applications. When one of the application team did 2000 user testing ACE dropping all PROBES and made them PROBE_FAILED.  2000 users testing carried 2 Gig traffic (approx) but we have license only for 1 Gig.

Licensed Feature                            Count

------------------------------------        -----

Compression Performance in Mbps             100

Web Optimization Concurrent Conns.          50

SSL transactions per second                 1000

Virtualized contexts                        5

Module bandwidth in Gbps                    1.0

But we dont see any dropping due to this license limit.

                                                     Allocation

        Resource         Current       Peak        Min        Max       Denied

-------------------------------------------------------------------------------

Context: Admin

  conc-connections             19         21          0    2000000          0

  mgmt-connections              4         16          0     100000          0

  proxy-connections             0          2          0     262143          0

  xlates                        0          0          0      65535          0

  acc-connections               0          0          0         50          0

  bandwidth                   145      24466          0  250000000          0

    throughput                  0      18220          0  125000000          0

    mgmt-traffic rate         145       6246          0  125000000          0

  connection rate               0          7          0     120000          0

  ssl-connections rate          0          3          0       1000          0

  mac-miss rate                 0          0          0       2000          0

  inspect-conn rate             0          0          0      40000          0

  http-comp rate                0          0          0   13107200          0

  acl-memory                33824      33824          0   48824320          0

  sticky                        0          0          0     737280          0

  regexp                      607        607          0    1048576          0

  syslog buffer              2048       2048          0    4194304          0

  syslog rate                   0        980          0     100000          0

Context: PROD-01

  conc-connections           3113      96313          0    2000000          0

  mgmt-connections              8         44          0     100000          0

  proxy-connections          2559      12975          0     262143          0

  xlates                        0          0          0      65535          0

  acc-connections               0          0          0         50          0

  bandwidth                443778   83969871          0  250000000          0

    throughput             442984   83964177          0  125000000          0

    mgmt-traffic rate         794       5694          0  125000000          0

  connection rate              13       1015          0     120000          0

  ssl-connections rate         18         94          0       1000          0

  mac-miss rate                 0          0          0       2000          0

  inspect-conn rate             0          0          0      40000          0

  http-comp rate                0          0          0   13107200          0

  acl-memory                37168      37168          0   48824320          0

  sticky                     1112       1153      81920      81920          0

  regexp                     1633       1633          0    1048576          0

  syslog buffer           4192256    4192256          0    4194304          0

  syslog rate                  43       3712          0     100000          0

The Mystery is this ACE is in DMZ , so all traffic we are seeing only thriugh Internet, But we have only 300Megs internet pipe. However we see around 2 Gigs traffic on ACE Uplinks (On Layer2 device where ACE connected).

Any advice or how can we conclude we are hitting license limit. First of all is it a license issue?

[Cesar Roque]

Hi Yahshanulla,

The best way to know if you are hitting license limits is looking the denies counter.  In your case is all zeros, so it does not look that your ACE was over limits at any point.

[Yahshanulla S]

Then do we have any idea why all PROBES were failed during the load test.

[Cesar Roque]

Hi Yahshanulla,

Do you have a showtech of the ACE after and before the tests?

By any chance did you take captures on the servers when the probe was failing?

Without this information will be really hard to give a root cause.   We know that trafffic generators can cause some odd behaviors,  but we need to see these outputs to know what is happening.

[Yahshanulla S]

well. We dont have packet captures, but i did capture the tech support during the issue happened with Admin context.

Attaching that

[Jorge Bejarano]

Hello, Yahsh

What is the result of these outputs?

show license usage

show license internal event-history

Did you experience this with the device which you requested to have replaced?

Same suggestions have been given to you before, sniffer captures are needed to determine the root cause of the behavior or at least have a better idea about what is happening at that moment

Jorge

[Yahshanulla S]

Jorge,

We havent repleced the ACE , but we are running Secondary ACE now. And we have faced the same issue. There are 3 outages happened and all the time matches the load testing timings.

And it will be very hard to do the sniffers as we have around 100+ servers involved in this outage.

I attached the outputs of above asked.

We are planning to do load test again under supervision, Can you Suggest is there any specific info we need to cloeect during that time.