03-14-2011 04:15 PM
I have been monitoring connection stats on an ACE Module context (show stats connection).
During recent load testing I observer that about 1/3 of all connections were failing and some timing out.
How can I find out more details about the connections that are failing and timing out?
ie which specific connection are failing, what type of failures increment this counter, etc
Cheers,
Karl
# sh stats conn
+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
Total Connections Created : 5725
Total Connections Current : 2382
Total Connections Destroyed: 3590
Total Connections Timed-out: 11
Total Connections Failed : 2112
03-14-2011 09:51 PM
Hello Karl,
As far as I have tested in my lab using A232.
For the "Total Connections Failed", this increments when you access a vip where associated
service-policy is out of service, with L7 rule you access a vip where associated service-policy is inservice and ACE
sends SYN out toward one of rservers but it never respond back so retry timed-out,
and also packets failed for route lookup etc.
For the "Total Connections Timed-out", this increment with L3, L4 rule when you access a vip where associated
service-policy is inservice but used rserver never respond back so retry timed-out.
In order to isolate your problem, I guess you configure access-list as specific as possible to rule out "packets failed for route lookup".
The, check out "failure" counters of rserver and serverfarm also dropped counter of service-policy.
Pls note that both counter get incremented "2" for one access to vip if the access failured due to reasons mention above.
Thanks and regards,
Kim.
03-16-2011 12:55 AM
Hi Kim,
Thanks for responding.
A load test was run earlier today. I cleared the counters and collected some stats (see below).
I also had a look at various other stats for drops.
As far as I could see all relevant, rservers, serverfarms and service-policies were:
Another load test is scheduled for tomorrow.
Any thoughts on what to check next?
labcorelb/DZ1ENV# sh stats conn
+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
Total Connections Created : 131698
Total Connections Current : 3660
Total Connections Destroyed: 67558
Total Connections Timed-out: 70
Total Connections Failed : 64090
+------------------------------------------+
+-------------- HTTP statistics -----------+
+------------------------------------------+
LB parse result msgs sent : 91904 , TCP data msgs sent : 187614
Inspect parse result msgs : 0 , SSL data msgs sent : 0
sent
TCP fin msgs sent : 75 , TCP rst msgs sent: : 3
Bounced fin msgs sent : 0 , Bounced rst msgs sent: : 0
SSL fin msgs sent : 0 , SSL rst msgs sent: : 0
Drain msgs sent : 31831 , Particles read : 397303
Reuse msgs sent : 0 , HTTP requests : 64390
Reproxied requests : 58314 , Headers removed : 0
Headers inserted : 0 , HTTP redirects : 0
HTTP chunks : 741 , Pipelined requests : 0
HTTP unproxy conns : 64360 , Pipeline flushes : 0
Whitespace appends : 0 , Second pass parsing : 0
Response entries recycled : 0 , Analysis errors : 0
Header insert errors : 0 , Max parselen errors : 0
Static parse errors : 0 , Resource errors : 0
Invalid path errors : 0 , Bad HTTP version errors : 0
Headers rewritten : 0 , Header rewrite errors : 0
SSL headers inserted : 0 , SSL header insert errors : 0
SSL spoof headers deleted : 0
Unproxy msgs sent : 64362
+------------------------------------------+
+--------- HTTP Inspect statistics --------+
+------------------------------------------+
Total request/response : 0
Total allow decisions : 0
Total drop decisions : 0
Total logging decisions : 0
+------------------------------------------+
+------- Loadbalance statistics -----------+
+------------------------------------------+
Total version mismatch : 0
Total Layer4 decisions : 142
Total Layer4 rejections : 0
Total Layer7 decisions : 64401
Total Layer7 rejections : 0
Total Layer4 LB policy misses : 0
Total Layer7 LB policy misses : 0
Total times rserver was unavailable : 0
Total ACL denied : 0
Total IDMap Lookup Failures : 0
+------------------------------------------+
+----------- Sticky statistics ------------+
+------------------------------------------+
Total sticky entries reused : 0
prior to expiry
Total active sticky entries : 0
Total active reverse sticky : 0
entries
Total active sticky conns : 0
Total static sticky entries : 0
+-----------------------------------------------------+
+---------------- KAL-AP(UDP) statistics -------------+
+-----------------------------------------------------+
Total bytes received : 0
Total bytes sent : 0
Total requests received : 0
Total responses sent : 0
Total requests successfully received : 0
Total queries successfully received : 0
Total responses successfully sent : 0
Total secure requests received : 0
Total secure responses sent : 0
Total requests with errors : 0
Total requests with parse errors : 0
Total requests dropped due to queue overflow : 0
Total response transfer errors : 0
-----------------------------------------------------
labcorelb/DZ1ENV#sh service-policy Vip_POLICY summary
service-policy: Vip_POLICY
Class VIP Prot Port VLAN State Curr Conns Hit Count Conns Drop
xxxx34_HTTP_CLASS xxxx.34 tcp eq 80 ALL IN-SRVC 0 0 0
WCMST-WWW_HTTP_CLASS xxxx.50 tcp eq 80 ALL IN-SRVC 7 48 0
WCMST-FORUM_HTTP_CLASS xxxx.111 tcp eq 80 ALL IN-SRVC 8 577 0
WCMST-SEARCH_HTTP_CLASS xxxx.51 tcp eq 80 ALL IN-SRVC 0 174 0
WCMST-ENGINEADM_HTTP_CLASS xxxx.112 tcp eq 80 ALL IN-SRVC 0 0 0
INTERNET-WWW_HTTP_CLASS xxxx.110 tcp eq 80 ALL IN-SRVC 1794 33919 18
03-16-2011 03:43 AM
Hello, Karl
There were no increment of "dropped" counter which means service-policy was INSERVICE during the test.
How was "failure" counter in "show rserver detail" or "show serverfarm detail" before and after the test ?
Is there certain amount of increments with that ?
I see you are testing with L7 rule. As I mentioned, with L7 rule the "Total Connections Failed" counter increments
in the event of that rservers never responds for SYN from ACE.
The other possibility I mentioned are route lookup failure including an access to vip with wrong port number.
If route lookup failure or access to vip with wrong port are happening, following counter will be incremented.
"show np [1 | 2] me-stats -socm"
Drop [route lookup fail]:
Would it be possible for you to gather following outputs twice at before and after the load test ?
Those would help to know what is happening.
"show stats connection"
"show rservers detail"
"show serverfarm detail"
"show service-policy detail"
"show np [1 | 2] me-stats -socm"
"show np [1 | 2] me-stats -sicm"
"show np [1 | 2] me-stats -snorm"
"show np [1 | 2] me-stats -stcp"
Thanks and regards,
Kim.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide