03-31-2011 09:24 PM
Hello,
We've not long implemented a pair of ACE 4710's on our network, configuring them with multiple contexts to support different applications. We are now want to look at the performance of these devices to see if they are being "stressed" in any way.
To my surprise, when I issue the "show resource usage all" command, there are a large number "denied" counters against the bandwidth Resource. The ACE is connected to the network with a 4Gb/sec link, and to each "real server" by a 1Gb/sec link.
The context in question (RC_Exchange2010) is configured to recieve a minimum of 26% of all resources, and an unlimited maximum.
I have attached a file showing the output of the show resource usage all, and show resource allocation commands.
Can anyone tell me what these counters mean, and what would be causing them to increment?
Thanks in advance,
Jeremy Keen
04-04-2011 09:05 PM
Hi Jeremy
I have reviewed the output and looks a bit weird.
PCN-ACE4710-03/Admin# show resource usage all
Allocation
Resource Current Peak Min Max Denied
-------------------------------------------------------------------------------
Context: Admin
bandwidth 134635 11583241 0 479334802 0
throughput 3447 48905 0 354334802 0
Context: VC_Exchange2010
bandwidth 1552028 134931123 139586437 479334802 79503
throughput 1486492 133816911 139586437 354334802 79503
Context: VC_Sharepoint
bandwidth 444 13430223 21474836 479334802 0
throughput 444 12840399 21474836 354334802 0
Context: VC_APH
bandwidth 90 262234 21474836 479334802 0
throughput 90 90 21474836 354334802 0
Can you reply with a following info?
sh ver
sh license usage
sh license status
regards
Andrew
04-06-2011 09:40 PM
04-08-2011 07:13 AM
Hi Jeremy,
Here are some following commands to judge performance of ACE.
Before you begin to troubleshoot ACE performance issues, check and record the following items:
1. Be sure that the correct licenses are installed in your ACE.
2. Record the number of flows that you are sending to the ACE.
3. Record the performance of a single flow.
4. Identify the type of traffic: unidirectional (UDP, management) or bidirectional (TCP, HTTP, SSL, and so on)
5. Identify the ACE context that is receiving the traffic.
6. Enter the following Exec mode commands and save the output to a file:
7. Be familiar with your application setup.
To troubleshoot performance issues with your ACE, follow these steps:
1. Display the resources allocated to each resource class in the ACE by entering the following command:
ACE_module5/Admin# show resource allocation --------------------------------------------------------------------------- Parameter Min Max Class --------------------------------------------------------------------------- acl-memory 0.00% 100.00% default 0.00% 100.00% RC1 syslog buffer 0.00% 100.00% default 0.00% 100.00% RC1 conc-connections 0.00% 100.00% default 0.00% 100.00% RC1 mgmt-connections 0.00% 100.00% default 0.00% 100.00% RC1 proxy-connections 0.00% 100.00% default 0.00% 100.00% RC1 bandwidth 0.00% 100.00% default 0.00% 100.00% RC1 connection rate 0.00% 100.00% default 0.00% 100.00% RC1 inspect-conn rate 0.00% 100.00% default 0.00% 100.00% RC1 syslog rate 0.00% 100.00% default 0.00% 100.00% RC1 regexp 0.00% 100.00% default 0.00% 100.00% RC1 sticky 0.00% 100.00% default 5.00% 5.00% RC1 xlates 0.00% 100.00% default 0.00% 100.00% RC1 ssl-connections rate 0.00% 100.00% default 0.00% 100.00% RC1 mgmt-traffic rate 0.00% 100.00% default 0.00% 100.00% RC1 mac-miss rate 0.00% 100.00% default 0.00% 100.00% RC1 throughput 0.00% 100.00% default 0.00% 100.00% RC1
2. Display the resources allocated to the context in question by entering the following command
ACE_module5/Admin# show resource usage context C1 Allocation Resource Current Peak Min Max Denied ------------------------------------------------------------------------------- Context: C1 conc-connections 0 0 0 8000000 0 mgmt-connections 0 0 0 100000 0 proxy-connections 0 0 0 1048574 0 xlates 0 0 0 1048574 0 bandwidth 0 0 0 625000000 0 throughput 0 0 0 500000000 0 mgmt-traffic rate 0 0 0 125000000 0 <------- 1 GBps bandwidth reserved for management traffic connection rate 0 0 0 1000000 0 ssl-connections rate 0 0 0 5000 0 mac-miss rate 0 0 0 2000 0 inspect-conn rate 0 0 0 6000 0 acl-memory 0 0 0 78610432 0 sticky 0 0 209714 0 0 regexp 0 0 0 1048576 0 syslog buffer 0 0 0 4194304 0 syslog rate 0 0 0 100000 0
3. From the supervisor CLI, check the connectivity to the back plane by entering the following command:
cat6k# show fabric status slot channel speed module fabric status status 2 0 8G OK OK 3 0 8G OK OK 4 0 8G OK OK 5 0 8G OK OK <-------Shows 8 Gbps connectivity to the chassis back plane 6 0 20G OK OK 8 0 8G OK OK
4. Check the fabric utilization by entering the following command:
cat6k# show fabric utilization slot channel speed Ingress % Egress % 2 0 8G 3 2 3 0 8G 0 0 4 0 8G 0 0 5 0 8G 0 0 6 0 20G 0 0 8 0 8G 2 3
5. Display the load of the network processors (NPs) in terms of packets and connection processing for each microengine (ME) by entering the following command:
ACE_module5/Admin# show np 1 me-stats -cpu 0 proxies open. ME Utilization Statistics -------------- RECEIVE: 7 FASTPATH: 44 SLOWTX: 0 TCP_RX: 0 HTTP: 0 IH_RX 0 SSL_ME: 0 CM_CLOSE: 36 X_TO_ME: 0 FIXUP: 0 REASSEMBLY: 0 OCM: 0 TCP_TX: 0 ICM: 39
ACE/Admin# show np 2 me-stats -cpu 0 proxies open. ME Utilization Statistics -------------- RECEIVE: 9 FASTPATH: 46 SLOWTX: 2 TCP_RX: 0 HTTP: 0 IH_RX 0 SSL_ME: 0 CM_CLOSE: 43 X_TO_ME: 0 FIXUP: 0 REASSEMBLY: 0 OCM: 0 TCP_TX: 0 ICM: 46
6. Monitor the CDE queues and ensure that the Fifo Full drop count counter is not incrementing by entering the following command:
ACE_module5/Admin# show cde health | include Fifo Fifo Full drop count 0
Backpressure is the mechanism that the ACE uses to slow the system down if queues start to fill up internally. Queues that can be affected and create backpressure are as follows:
It is possible that some packets that are received by the ACE could be dropped internally if backpressure is applied.
7. Monitor the Fastpath micro engine queues and ensure that the FastQ Transmit Backpressure, the SlowQ Transmit Backpressure, the Drop: Transmit Backpressure, and the Drop: Next-Hop queue full counters are not incrementing by entering the following command:
ACE_module5/Admin# show np 1 me-stats "-s fp" | include Backpressure FastQ Transmit Backpressure: 0 SlowQ Transmit Backpressure: 0 Drop: Transmit Backpressure: 0
ACE/Admin# show np 1 me-stats "-s fp" | include queue Drop: Next-Hop queue full: 0
8. Monitor the TCP micro engine queues and ensure the Drops due to FastTX queue full, Drops due to Fastpath queue full, Drops due to HTTP queue full, Drops due to SSL queue full, Drops due to AI queue full, and Drops due to Fixup queue full are not incrementing by entering the following command. If TCP receives backpressure, it can drop packets, fail to ACK packets, and fail to properly track the next packet in the TCP connection.
ACE/Admin# show np 1 me-stats "-s tcp" | include queue Drop reproxy msg queue full: 0 Drops due to FastTX queue full: 0 Drops due to Fastpath queue full: 0 Drops due to HTTP queue full: 0 Drops due to SSL queue full: 0 Drops due to AI queue full: 0 Drops due to Fixup queue full: 0
The control plane (CP) processor processes all CP traffic (ARP, HSRP, ICMP to VIPs, routing, syslogs, SNMP, probes, and so on) and handles configuration management to parse the CLI for syntactical errors and enforce configuration dependencies and requirements before pushing the configuration to the data plane.
9. Display a three-way moving average of the CP processor utilization (updated every five seconds) by entering the following command:
ACE_module5/Admin# show processes cpu | inc util CPU utilization for five seconds: 81%; one minute: 15%; five minutes: 10%
The ACE allocates data-plane memory to guarantee concurrent connection support for basic Layer 4 connections (such as TCP, UDP, IPsec), Layer 7 connections (proxied flows, typically for application aware load balancing or inspection, and SSL connection when using SSL acceleration). The ACE can support the maximum bidirectional concurrent connection limit regardless of the features enabled.
Table 1. Concurrent Connection Support
Connection Type | ACE Module Limit |
Layer 4 | 4,000,000 |
Layer 7 | 512,000 |
The state for both directions (client-to-VIP/ACE and server-to-ACE) of a TCP connection is maintained with distinct connection objects.
10. Display the connection table by entering the following command:
ACE_module5/Admin# show conn total current connections : 6 conn-id np dir proto vlan source destination state ----------+--+---+-----+----+---------------------+---------------------+------+ 1 1 in TCP 130 161.44.67.242:2856 10.86.215.134:23 ESTAB 2 1 out TCP 130 10.86.215.134:23 161.44.67.242:2856 ESTAB 4 1 in TCP 130 161.44.67.242:2837 10.86.215.134:23 ESTAB 3 1 out TCP 130 10.86.215.134:23 161.44.67.242:2837 ESTAB 4 2 in TCP 130 161.44.67.242:2857 10.86.215.134:23 ESTAB 3 2 out TCP 130 10.86.215.134:23 161.44.67.242:2857 ESTAB
Note: | You can add the detail command option to provide the following additional fields: connection idle time, elapsed time of the connection, byte count, and packet count for each connection object. |
The total current connections counter is also maintained in the output of the following command:
switch/Admin# show stats connection +------------------------------------------+ +------- Connection statistics ------------+ +------------------------------------------+ Total Connections Created : 124 Total Connections Current : 6 Total Connections Destroyed: 62 Total Connections Timed-out: 58 Total Connections Failed : 0
Because of the Cisco ACE Module’s architecture, with distinct paths for new and established connections, the number of existing concurrent connections does not heavily impact the rate at which new connections can be set up. Nevertheless, a very large number of concurrent connections will eventually affect the performance of the system in setting up new connections.
11. Use the command "tcp wan-optimization rtt 0" for slow connections.
The ACE module architecture includes a mechanism where connections can be moved to the fastpath in order to increase performance for a given connection. The LB decision is made in the software (proxy) and then moved to the fastpath (unproxy). In a persistence rebalance scenario, the proxy/unproxy can occur Many times on a given connection. It is possible that if a packet enters the system during the transition Between the proxy and unproxy states, a packet may not be forwarded as expected and a retransmission may be relied upon. This can affect performance. As a workaround, it is possible to configure the ACE such that fastpath forwarding is prohibited This can be accomplished by configuring a parameter map with the following:
"tcp wan-optimization rtt 0"
Find Below the link containg al the other troubleshooting Guides. Please rate.
HTH
Sachin Garg
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide