cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5566
Views
0
Helpful
3
Replies

ACE Total Connections Failed stats

KarloskiG
Level 1
Level 1

I have been monitoring connection stats on an ACE Module context (show stats connection).

During recent load testing I observer that about 1/3 of all connections were failing and some timing out.

How can I find out more details about the connections that are failing and timing out?

ie which specific connection are failing, what type of failures increment this counter, etc

Cheers,

Karl

# sh stats conn

+------------------------------------------+

+------- Connection statistics ------------+

+------------------------------------------+

Total Connections Created : 5725

Total Connections Current : 2382

Total Connections Destroyed: 3590

Total Connections Timed-out: 11

Total Connections Failed : 2112

3 Replies 3

kitanaka
Level 1
Level 1

Hello Karl,

As far as I have tested in my lab using  A232.

For the "Total Connections Failed", this increments when you access a  vip where associated

service-policy  is out of service, with L7 rule you access a vip where associated  service-policy is inservice and ACE

sends SYN out toward one of  rservers but it never respond back so retry timed-out,

and also  packets failed for route lookup etc.

For the "Total  Connections Timed-out",  this increment with L3, L4 rule when you  access a vip where associated

service-policy is inservice but used rserver never respond back so  retry  timed-out.

In order to isolate your problem, I guess you configure access-list  as specific as possible to rule out "packets  failed for route lookup".

The, check out "failure" counters of rserver and serverfarm also  dropped counter of service-policy.

Pls note that  both counter get incremented "2" for one access to vip  if the access failured due to reasons mention above.

Thanks and regards,

Kim.

Hi Kim,

Thanks for responding.

A load test was run earlier today. I cleared the counters and collected some stats (see below).

I also had a look at various other stats for drops.

As far as I could see all relevant, rservers, serverfarms and service-policies were:

  • in service BUT
  • drop stats were not incrementing at anywhere near the same rate as the Total Connections Failed stat.

Another load test is scheduled for tomorrow.

Any thoughts on what to check next?

labcorelb/DZ1ENV# sh stats conn

+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
Total Connections Created  : 131698
Total Connections Current  : 3660
Total Connections Destroyed: 67558
Total Connections Timed-out: 70
Total Connections Failed   : 64090

+------------------------------------------+
+-------------- HTTP statistics -----------+
+------------------------------------------+
LB parse result msgs sent : 91904      , TCP data msgs sent       : 187614   
Inspect parse result msgs : 0          , SSL data msgs sent       : 0        
                      sent
TCP fin msgs sent         : 75         , TCP rst msgs sent:       : 3        
Bounced fin msgs sent     : 0          , Bounced rst msgs sent:   : 0        
SSL fin msgs sent         : 0          , SSL rst msgs sent:       : 0        
Drain msgs sent           : 31831      , Particles read           : 397303   
Reuse msgs sent           : 0          , HTTP requests            : 64390    
Reproxied requests        : 58314      , Headers removed          : 0        
Headers inserted          : 0          , HTTP redirects           : 0        
HTTP chunks               : 741        , Pipelined requests       : 0        
HTTP unproxy conns        : 64360      , Pipeline flushes         : 0        
Whitespace appends        : 0          , Second pass parsing      : 0        
Response entries recycled : 0          , Analysis errors          : 0        
Header insert errors      : 0          , Max parselen errors      : 0        
Static parse errors       : 0          , Resource errors          : 0        
Invalid path errors       : 0          , Bad HTTP version errors  : 0        
Headers rewritten         : 0          , Header rewrite errors    : 0        
SSL headers inserted      : 0          , SSL header insert errors : 0        
SSL spoof headers deleted : 0        
Unproxy msgs sent         : 64362    


+------------------------------------------+
+--------- HTTP Inspect statistics --------+
+------------------------------------------+
Total request/response   : 0
Total allow decisions    : 0
Total drop decisions     : 0
Total logging decisions  : 0


+------------------------------------------+
+------- Loadbalance statistics -----------+
+------------------------------------------+
Total version mismatch              : 0
Total Layer4 decisions              : 142
Total Layer4 rejections             : 0
Total Layer7 decisions              : 64401
Total Layer7 rejections             : 0
Total Layer4 LB policy misses       : 0
Total Layer7 LB policy misses       : 0
Total times rserver was unavailable : 0
Total ACL denied                    : 0
Total IDMap Lookup Failures         : 0


+------------------------------------------+
+----------- Sticky statistics ------------+
+------------------------------------------+
Total sticky entries reused    : 0
prior to expiry
Total active sticky entries    : 0
Total active reverse sticky    : 0
entries
Total active sticky conns      : 0
Total static sticky entries    : 0


+-----------------------------------------------------+
+---------------- KAL-AP(UDP) statistics -------------+
+-----------------------------------------------------+

Total bytes received                         : 0
Total bytes sent                             : 0
Total requests received                      : 0
Total responses sent                         : 0
Total requests successfully received         : 0
Total queries successfully received          : 0
Total responses successfully sent            : 0
Total secure requests received               : 0
Total secure responses sent                  : 0
Total requests with errors                   : 0
Total requests with parse errors             : 0
Total requests dropped due to queue overflow : 0
Total response transfer errors               : 0
-----------------------------------------------------

labcorelb/DZ1ENV#sh service-policy Vip_POLICY summary

service-policy: Vip_POLICY
Class                            VIP       Prot  Port        VLAN          State    Curr Conns   Hit Count  Conns Drop
xxxx34_HTTP_CLASS                xxxx.34   tcp   eq 80       ALL           IN-SRVC           0           0          0
WCMST-WWW_HTTP_CLASS             xxxx.50   tcp   eq 80       ALL           IN-SRVC           7          48          0
WCMST-FORUM_HTTP_CLASS           xxxx.111  tcp   eq 80       ALL           IN-SRVC           8         577          0
WCMST-SEARCH_HTTP_CLASS          xxxx.51   tcp   eq 80       ALL           IN-SRVC           0         174          0
WCMST-ENGINEADM_HTTP_CLASS       xxxx.112  tcp   eq 80       ALL           IN-SRVC           0           0          0
INTERNET-WWW_HTTP_CLASS          xxxx.110  tcp   eq 80       ALL           IN-SRVC        1794       33919         18

Hello, Karl

There were no increment of "dropped" counter which means service-policy was INSERVICE during the test.

How was "failure" counter in "show rserver detail" or "show serverfarm detail" before and after the test ?

Is there certain amount of increments with that ?

I see you are testing with L7 rule. As I mentioned, with L7 rule the "Total Connections Failed" counter increments

in the event of that rservers never responds for SYN from ACE.

The other possibility I mentioned are route lookup failure including an access to vip with wrong port number.

If route lookup failure or access to vip with wrong port are happening, following counter will be incremented.

"show np [1 | 2] me-stats -socm"

Drop [route lookup fail]:

Would it be possible for you to gather following outputs twice at before and after the load test ?

Those would help to know what is happening.

"show stats connection"

"show rservers detail"

"show serverfarm detail"

"show service-policy detail"

"show np [1 | 2] me-stats -socm"

"show np [1 | 2] me-stats -sicm"

"show np [1 | 2] me-stats -snorm"

"show np [1 | 2] me-stats -stcp"


Thanks and regards,

Kim.