04-27-2011 08:50 AM
Just inherited support of our Cisco Waas devices, one devie inparticular dosent seem to be working with local file sharing, while web caching appeas to be working as normal.
In this device IS see the following: (I assume this is a problem as cache size and usage are almost the same and seeing the file evictions)
On a related not what exactly is an evicted resource? Could not seem to find the answer in the Cisco PDF manual.
81.75488 GB | |
81.75411 GB | |
5000000 | |
159219 | |
939372 | |
Wed Apr 27 10:40:10 CDT 2011 | |
96 | |
94 | |
95 | |
94 | |
5.7103443 week | |
Wed Apr 27 10:40:10 CDT 2011 |
04-29-2011 04:08 AM
Hi Eric,
When the cache of your device is almost full (and it seems to be the case on your setup), the device is going to start deleting some of the cached objects to make room for new ones that comes in, the ones which are selected are the ones with the oldest time-last-accessed values in WAAS.
This deletion process is also called eviction, the number you see in your output is simply the number of cached objects which have been deleted to make room for new ones.
Regards,
Nicolas
05-01-2011 09:11 AM
Hi,
Priority is given to client requests
Priority given to eviction
No client requests handled
05-01-2011 12:28 PM
Peer relationships time out after 10 minutes of inactivity (that is, no active connections are established between two peers for 10 minutes). When the peer relationship is timed out, it becomes reusable by another peer. Data stored in the DRE compression history remains intact even if a peer becomes disconnected due to inactivity, unless the DRE compression history becomes full. In cases where the DRE compression history becomes full, an eviction process is initiated to remove the oldest set of data in the DRE compression history to make room for new data.
In the navigation area, click Monitoring under the WAFS Edge menu.
The WAFS Edge Monitoring window appears.
The Cache tab displays the following information:
•Maximum cache disk size—Maximum amount of disk space (in gigabytes) allocated to the WAFS Edge device cache.
•Current cache disk usage—Current amount of disk space (in kilobytes) used by the WAFS Edge device cache. The name of this statistic is a link that you can use to display its historical graphs (without first going to the Graphs tab).
•Maximum cache resources—Maximum number of resources (files and directories) allowed in the WAFS Edge device cache.
•Current cache resources—Current number of resources contained in the WAFS Edge device cache. The name of this statistic is a link that you can use to display its historical graphs (without first going to the Graphs tab).
•Evicted resources count—Number of resources that have been evicted from the cache since the Edge WAE was started.
•Last eviction time—Time when a cache eviction last occurred.
•Cache size high watermark—Percentage of disk usage that causes the WAFS Edge device to begin evicting resources.
•Cache size low watermark—Percentage of disk usage that causes the WAFS Edge device to stop evicting resources.
•Cache resources high watermark—Percentage of total cache resources that causes the WAFS Edge device to begin evicting resources.
•Cache resources low watermark—Percentage of total cache resources that causes the WAFS Edge device to stop evicting resources.
•Last evicted resource age—Amount of time that the last-evicted resource spent in the WAFS Edge device cache.
•Last evicted resource access time—Last time that the last-evicted resource was accessed.
Along with providing the user-side proxy functionality necessary for accelerating CIFS, the edge services also provide termination of the CIFS proxy connections that are established between edge and core WAAS devices. WAAS devices configured with edge services provide a client-side object cache and metadata cache for CIFS, which is independent of the DRE compression history. By keeping the application cache independent of the compression history, application performance can be maintained even in the event of massive compression history eviction, which could happen if a user were to, for instance, back up their entire home movie or music library over the WAN. Furthermore, having a separate application cache ensures that failures and cache-clearing events are isolated from one another. As an example, an application cache may be used to hold numerous gigabytes of distributed software or patches, which, if evicted, may need to be transferred over the WAN again. Keeping the cache and compression history isolated from one another provides numerous benefits in terms of operational efficiency.
In essence, the core acts as a "staging area," because it is unaware of the status of the edge cache. The core, once it receives the manifest from the origin server (based on the job configuration), passes the manifest to the edge. Then, the core begins fetching files from the origin server based on the manifest. These files are then stored in a temporary staging area, which is ~1 GB in capacity. As the edge fetches files, they are removed from the staging area and the core continues to fetch additional files from the origin server. If the staging area becomes full of files, and the edge requests a file that is not in the staging area, the core will know that the edge does not need any of the files found in the staging area. In such a case, the core will empty its staging area and begin fetching files from the origin server starting with the file requested by the edge. Because the core and edge are working from the same manifest, the core can adapt and evict files from the staging area based on the requests coming from the edge.
HTH
Sachin
06-24-2011 07:48 PM
Can you explain why the last eviction time is in the future? I need to understand when was the last time an eviction happened and how to correlate it to the oldest DRE cache. Is the last eviction time shown below only for the CIFS AO? And if so, how can I see the last eviction time for the DRE cache? Also, when you do a show disk detail, the obj1 volume partition doesn't match the Maximum cache disk size.
/obj1 CONTENT /dev/data1/obj 204338MB 142589MB 61749MB 69% CIFS AO? Doesn't match 175GB.
/dre1 CONTENT /dev/data1/dre 991932MB 973878MB 18054MB 98%
DRE Cache Stats
Cache:
Status: Usable, Oldest Data (age):6d14h
Total usable disk size: 972799 MB, Used: 99.57%
Hash table RAM size: 3588 MB, Used: 93.00%
175.55469 GB | |
118.42443 GB | |
5000000 | |
4500003 | |
2018623 | |
Sat Jun 25 00:44:22 UTC 2011 | |
96 | |
94 | |
95 | |
94 | |
19.03269 week | |
Sat Jun 25 00:44:22 UTC 2011 |
06-24-2011 09:36 PM
Hi John,
Please check have you set the correct time zone and time on each WAE. Domain integration requires that each domain member, including WAEs, be no more than 5 minutes askew from the relative time on the domain controller. That is, the time difference on the WAE, after accounting for time zone differences, must be within 5 minutes of the relative time on the domain controller.
Kindly provide the output of the commands I have requested to look further into your configuration:
WAE# show clock
1.
The CIFS accelerator transparently optimizes CIFS traffic on ports 139 and 445.
You can verify the general AO configuration and status with the
# show accelerator
and
#show license
command
-------------------------------------------------------------------------------------------
2.
Next, verify the status that is specific to the CIFS AO by using the
#show accelerator cifs command,
You want to see that the CIFS AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is Enabled but the Operational State is Shutdown, it indicates a licensing problem.
------------------------------------------------------------------------------------------
3.
Use the show running-config command to verify that the CIFS traffic policy is properly configured. You want to see accelerate cifs for the WAFS application action and you want to see appropriate match conditions listed for the CIFS classifier, as follows:
WAE674# sh run | include CIFS
classifier CIFS
name WAFS classifier CIFS action optimize full accelerate cifs
WAE674# sh run | begin CIFS
...skipping
classifier CIFS
match dst port eq 139
match dst port eq 445
exit
---------------------------------------------------------------------------------------------------------
4.
Use the show statistics connection optimized cifs command to check that the WAAS device is establishing optimized CIFS connections. Verify that "TCDL" appears in the Accel column for a connection. A "C" indicates that the CIFS AO was used.
WAE674# sh stat conn opt cifs
Current Active Optimized Flows: 3
Current Active Optimized TCP Plus Flows: 3
Current Active Optimized TCP Only Flows: 0
Current Active Optimized TCP Preposition Flows: 1
Current Active Auto-Discovery Flows: 0
Current Active Pass-Through Flows: 0
Historical Flows: 100
D:DRE,L:LZ,T:TCP Optimization,
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
ConnID Source IP:Port Dest IP:Port PeerID Accel
1074 10.10.10.10:2704 10.10.100.100:445 00:14:5e:84:24:5f TCDL <------Look for "C"
If you see "TDL" in the Accel column, the connection was optimized by transport optimizations only and was not inspected by the CIFS AO. This situation can happen if the CIFS AO is disabled, the Enterprise license is not configured, or if the maximum connection limit is reached.
If you see a "G" instead of a "C" in the Accel column, then the connection was pushed down from the CIFS AO to the generic AO and was optimized with transport optimizations only. This situation can happen if the connection requires SMB2 or a digital signature and an error message is logged for it
-----------------------------------------------------------------------------
5.
You can view the CIFS connection statistics by using the show statistics connection optimized cifs detail command as follows:
WAE674# sh stat connection optimized cifs detail
Connection Id: 1801
Peer Id: 00:14:5e:84:24:5f
Connection Type: EXTERNAL CLIENT
Start Time: Thu Jun 25 06:15:58 2009
Source IP Address: 10.10.10.10
Source Port Number: 3707
Destination IP Address: 10.10.100.100
Destination Port Number: 139
Application Name: WAFS <-----Should see WAFS
Classifier Name: CIFS <-----Should see CIFS
Map Name: basic
Directed Mode: FALSE
Preposition Flow: FALSE
Policy Details:
Configured: TCP_OPTIMIZE + DRE + LZ
Derived: TCP_OPTIMIZE + DRE + LZ
Peer: TCP_OPTIMIZE + DRE + LZ
Negotiated: TCP_OPTIMIZE + DRE + LZ
Applied: TCP_OPTIMIZE + DRE + LZ
Accelerator Details:
Configured: CIFS <-----Should see CIFS configured
Derived: CIFS
Applied: CIFS <-----Should see CIFS applied
Hist: None
Original Optimized
-------------------- --------------------
Bytes Read: 189314 10352510
Bytes Written: 91649704 28512
. . .
Connection details:
Chunks: encoded 3, decoded 49922, anchor(forced) 0(1)
Total number of processed messges: 1820
num_used_block per msg: 0.140659
Ack: msg 1609, size 7066 B
Encode bypass due to:
last partial chunk: chunks: 1, size: 142 B
skipped frame header: messages: 138, size: 27202 B
Nacks: total 0
R-tx: total 0
Encode LZ latency: 0.060 ms per msg
Decode LZ latency: 0.071 ms per msg
Aggregation encode: Retransmissions: 0 <-----Packets lost between peers
level 0: chunks: 3 hits: 0 miss: 3
level 1: chunks: 0 hits: 0 miss: 0
level 2: chunks: 0 hits: 0 miss: 0
level 3: chunks: 0 hits: 0 miss: 0
Aggregation decode: Collisions: 0
level 0: chunks: 174093 hits: 128716 miss: 0
level 1: chunks: 0 hits: 0 miss: 0
level 2: chunks: 0 hits: 0 miss: 0
level 3: chunks: 0 hits: 0 miss: 0
Aggregation stack memory usage: Sender: 452 B Receiver: 9119 B
Noise filter: Chunks: 0, Bytes: 0 B
. . .
If the Retransmissions counter increases, it means that packets are getting lost in the middle, between the two peer WAEs. This situation will result in lower throughput. You should investigate possible causes for packet lose in the network between the two peer WAEs.
You can view the CIFS request statistics by using the show statistics cifs requests command
------------------------------------------------------------------------------------------------------------
6. CIFS AO Logging
The following log files are available for troubleshooting CIFS AO issues:
Transaction log files: /local1/logs/tfo/working.log (and /local1/logs/tfo/tfo_log_*.txt)
CIFS internal log file: /local1/errorlog/cifs/cifs_err.log
Debug log files: /local1/errorlog/cifsao-errorlog.current (and cifsao-errorlog.*)
For easier debugging, you should first set up an ACL to restrict packets to one host.
WAE674(config)# ip access-list extended 150 permit tcp host 10.10.10.10 any
WAE674(config)# ip access-list extended 150 permit tcp any host 10.10.10.10
To enable transaction logging, use the transaction-logs configuration command as follows:
wae(config)# transaction-logs flow enable
wae(config)# transaction-logs flow access-list 150
You can view the end of a transaction log file by using the type-tail command as follows:
wae# type-tail tfo_log_10.10.11.230_20090715_130000.txt
:EXTERNAL CLIENT :00.14.5e.84.24.5f :basic :WAFS :CIFS :F :(DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO)
(DRE,LZ,TFO) :
Wed Jul 15 15:48:45 2009 :1725 :10.10.10.10 :2289 :10.10.100.100 :139 :OT :START :EXTERNAL CLIENT :00.14.5e.84.24.5f :basic :WAFS
:CIFS :F :(DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) :
Wed Jul 15 15:48:55 2009 :1725 :10.10.10.10 :2289 :10.10.100.100 :139 :OT :END : EXTERNAL CLIENT :(CIFS) :159 :221
To set up and enable debug logging of the CIFS AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a production environment.
You can enable detailed logging to the disk as follows:
WAE674(config)# logging disk enable
WAE674(config)# logging disk priority detail
You can enable debug logging for connections in the ACL:
WAE674# debug connection access-list 150
The options for CIFS AO debugging are as follows:
WAE674# debug accelerator cifs ?
all enable all CIFS accelerator debugs
shell enable CIFS shell debugs
You can enable debug logging for CIFS connections and then display the end of the debug error log as follows:
WAE674# debug accelerator cifs all
WAE674# type-tail errorlog/cifsao-errorlog.current follow
---------------------------------------------------------------------------------------
7.
Send the output of the following:
WAE# show cifs cache entry-count
WAE# show cifs cache disk-use
WAE# show clock
Please check have you set the correct time zone and time on each WAE. Domain integration requires that each domain member, including WAEs, be no more than 5 minutes askew from the relative time on the domain controller. That is, the time difference on the WAE, after accounting for time zone differences, must be within 5 minutes of the relative time on the domain controller.
The simplest way to align time on the WAE device with the domain is to first set the time zone on each WAE and then configure each WAE to use a Network Time Protocol (NTP) server. Although this can be easily accomplished through the device CLI, recommended practice dictates that you create unique device groups, where each device group represents a separate time zone. After you add the appropriate WAE devices into each timezone-based device group, you can apply the NTP server settings to the device group.
Also send me the output of the following command:
WAE# show running-config
................................................................
8. Regarding more infor for your disks
It is also useful to check the Predictive Failure Analysis (PFA) flag for RAID-5 disks by using the show disks tech-support command. You will find the PFA flag at the end of the output. If the PFA flag is set to Yes, it indicates a predicted drive failure and you should replace the disk. A critical alarm is also raised on the WAE.
Also send the output of show disks tech-support command as follows:
wae# show disks tech-support
Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
Controller Status : Okay
Channel description : SAS/SATA
Controller Model : IBM ServeRAID 8k
Controller Serial Number : 40453F0
Physical Slot : 0
Installed memory : 256 MB
Copyback : Disabled
Data scrubbing : Disabled
Defunct disk drive count : 0
Logical drives/Offline/Critical : 1/0/0
---------------------------------------------------
Controller Version Information
---------------------------------------------------
BIOS : 5.2-0 (15418)
Firmware : 5.2-0 (15418) <-----Firmware version
Driver : 1.1-5 (2449)
Boot Flash : 5.1-0 (15418)
---------------------------------------------------
................................................................
9. Send the output of the
#show license
# show tech
----
Once you send the output of the above commands I can check further.
Thanks
sachin
06-27-2011 09:14 AM
I think I understand it now, the eviction is occurring quite frequently on our core WAEs, so the time is almost daily. The clock is set to using our NTP servers, standard across the network. How can I view the last eviction time for the DRE cache? It only shows it for the CFS AO when logging into the GUI for the WAE itself. Is it normal for the eviction to be occurring daily on the CIFS AO? Is there a log to see how many times it occurs?
06-27-2011 10:36 AM
You have not send the output of any of the commands I have requested in my prevous post as it can help me in seeing more in details about your device and maybe your sample output can be helpful for others in future when they refer this post. But anyways if you dont want to publish that.
Please send me the output of following if possible:
WAE674# show statistics dre cache
sh statistics dre
Cache:
Status: Usable, Oldest Data (age): 21m39s
Total usable disk size: 116735 MB, Used: 0.01%
Hash table RAM size: 417 MB, Used: 0.00%
Connections: Total (cumulative): 533 Active: 12
Encode:
Overall: msg: 3984, in: 2003 KB, out: 1073 KB, ratio: 46.41%
DRE: msg: 3644, in: 1942 KB, out: 1741 KB, ratio: 10.32%
DRE Bypass: msg: 1036, in: 62427 B
LZ: msg: 2955, in: 1293 KB, out: 553 KB, ratio: 57.20%
LZ Bypass: msg: 1029, in: 508 KB
Avg latency: 0.155 ms Delayed msg: 599
Encode th-put: 3244 KB/s
Message size distribution:
0-1K=87% 1K-5K=12% 5K-15K=0% 15K-25K=0% 25K-40K=0% >40K=0%
Decode:
Overall: msg: 4338, in: 1456 KB, out: 2685 KB, ratio: 45.76%
DRE: msg: 3873, in: 2249 KB, out: 2547 KB, ratio: 11.70%
DRE Bypass: msg: 1495, in: 137 KB
LZ: msg: 3326, in: 894 KB, out: 1837 KB, ratio: 51.33%
LZ Bypass: msg: 1012, in: 562 KB
Avg latency: 0.084 ms
Decode th-put: 7394 KB/s
Message size distribution:
0-1K=84% 1K-5K=13% 5K-15K=1% 15K-25K=0% 25K-40K=0% >40K=0%
Will monitor for a few days.
Using the system message log feature of the WAAS Central Manager GUI, you can view information about events that have occurred in your WAAS network.
From the WAAS Central Manager GUI, choose System > Logs > System Messages. The System Message Log window appears.
From the WAAS Central Manager GUI, choose System > Logs > Audit Trail Logs
The following directories are used by Cisco WAAS for log
files:
–/local1—Root directory for all log files
–/local1/logs—Service log files (aka ―admin‖ logs)
–/local1/errorlog—Service log files (aka ―debug‖ logs)
–/local1/core_dir—Process core dump files
File system navigation commands:
–cd
–pwd
–dir
–type-tail
–find-pattern
----
Packets can be captured on all WAAS interfacesusing one of the following CLI tools:
–tethereal
–tcpdump
----------------
for Transaction Logs
Every transaction generates log
Multiple transaction attributes recorded
–TCP connection start time
–TCP connection end time
–Optimization done (AO, DRE, LZ, TFO, or PT)
–Flow identification information (L3/L4/L5)
–Bytes
• Origin received/sent
• Optimized received/sent
-----------
Transaction Logs – CM
- Enable transaction logging on branch WAEs
- /local1/logs/tfo
- Archive log schedule
- The archive filenamesuse this format: tfo_log_IPADDRESS_YYYYMMDD_HHMMSS.txt
-Export log schedule
-----------
to see TFO Transaction Logs
WAE# cd logs/tfo
WAE#
WAE#ls
ftp_export.statustfo_log_22.1.43.10_20090508_190000.txt
tfo_log_22.1.43.10_20090508_200000.txt
tfo_log_22.1.43.10_20090508_210000.txt
working.log
WAE# type-tail working.log
-----------------------
you can use Sawmill Transaction Log Analysis
- Sawmill understands WAAS transaction logs
- Syslog or FTP/SFTP transfer
- Extensive reports
--------------------------
Verify Global DRE Status
WAE674# sh stat dre
Cache:
Status: Usable, Oldest Data (age): 17h
Total usable disk size: 116735 MB, Used: 0.03%
Hash table RAM size: 442 MB, Used:0.00%
Connections: Total (cumulative): 13 Active:0
Encode:
Overall: msg: 11166, in: 67555 KB, out: 5686 KB, ratio:91.58%
DRE: msg: 2264, in: 64697 KB, out: 22962 KB, ratio: 64.51%
DRE Bypass: msg: 12556, in: 2858 KBLZ: msg: 11158, in: 25906 KB, out: 5685 KB, ratio: 78.06%
LZ Bypass: msg: 8, in: 0 B
Avg latency: 0.429 ms Delayed msg: 2706
Encode th-put: 14087 KB/s
Message size distribution:
0-1K=27% 1K-5K=4% 5K-15K=13% 15K-25K=7% 25K-40K=9% >40K=36%
Decode:Overall: msg: 13558, in: 6764 KB, out: 28336 KB, ratio: 76.13%
DRE: msg: 2018, in: 13192 KB, out: 25729 KB, ratio: 48.73%DRE Bypass: msg: 25565, in: 2606 KBLZ: msg: 13514, in: 6288 KB, out: 15419 KB, ratio: 59.22%LZ Bypass: msg: 44, in: 476 KBAvg latency: 0.063 msDecode th-put: 33241 KB/sMessage size distribution:0-1K=17% 1K-5K=45% 5K-15K=10% 15K-25K=5% 25K-40K=8% >40K=12%
------to show statistics connection
WAE674# sh stat conn
and once you see the different connection ID as output of the above commands you can see the details of that connection
WAE674# sh stat conn conn-id 2516
--------------
WAE-SiteA#sh stati
WAE-SiteA#sh statistics dre
Cache:
Status: Usable, Oldest Data (age): 4m14s
Total usable disk size: 47527 MB, Used: 0.00%
Hash table RAM size: 189 MB, Used: 0.00%
Connections: Total (cumulative): 1 Active: 1
Encode:
Overall: msg: 426, in: 2120 KB, out: 1558 KB, ratio: 26.47%
DRE: msg: 426, in: 2120 KB, out: 1843 KB, ratio: 13.04%
DRE Bypass: msg: 0, in: 0 B
LZ: msg: 374, in: 749 KB, out: 464 KB, ratio: 38.00%
LZ Bypass: msg: 52, in: 1094 KB
Avg latency: 0.808 ms
Message size distribution:
0-1K=69% 1K-5K=8% 5K-15K=7% 15K-25K=6% 25K-40K=5% >40K=1%
Decode:
Overall: msg: 397, in: 101158 B, out: 2196 KB, ratio: 95.50%
DRE: msg: 396, in: 162 KB, out: 2195 KB, ratio: 92.61%
DRE Bypass: msg: 1, in: 89 B
LZ: msg: 338, in: 67992 B, out: 130 KB, ratio: 48.95%
LZ Bypass: msg: 59, in: 33166 B
Avg latency: 0.045 ms
Message size distribution:
0-1K=65% 1K-5K=9% 5K-15K=8% 15K-25K=4% 25K-40K=11% >40K=0%
-------------------------------
Following are the details which can be helpful for you to troubleshoot DRE:
For DRE Troubleshooting you can use the following steps:
When application acceleration is expected but not being observed, verify that the appropriate optimizations are being applied to the traffic flow and that the DRE cache is reducing the size of the optimized traffic appropriately.
Policy engine maps for DRE and LZ optimization include the following:
1. DRE + LZ (full): policy-engine application map other optimize full
2. DRE only: policy-engine application map other optimize DRE yes compression none
3. LZ only: policy-engine application map other optimize DRE no compression LZ
4. TFO pass-through: policy-engine application map other pass-through
Various conditions could cause DRE and/or LZ not to be applied to a connection, even though it is configured:
1. Cache initialization is in progress
2. Disk I/O errors
3. Low memory
4. Data is not compressible or gain is too small
5. Data is encrypted such that it does not contain repeated byte sequences
6. Messages are too small to benefit from compression
Note: In all of the above conditions, the show statistics connection command will report Acceleration of "TDL" for connections where this was the negotiated policy.
Looking at the amount of DRE or LZ bypass traffic will tell you whether DRE or LZ optimizations were actually applied. Use the show statistics connection conn-id command, as described later, and look at the DRE encode numbers to see if the DRE or LZ ratio is near 0% and most of the traffic is bypassed.
The first three conditions will be reported by the "Encode bypass due to" field and the last three conditions result from the traffic data pattern and are accounted for in the reported DRE and LZ ratios.
You can view the statistics for a specific connection to determine what basic optimizations were configured, negotiated with the peer, and applied by using the show statistics connection conn-id command. First you will need to determine the connection ID for a particular connection by using the show statistics connection command, as follows:
WAE#show stat conn
Current Active Optimized Flows: 1
Current Active Optimized TCP Plus Flows: 0
Current Active Optimized TCP Only Flows: 1
Current Active Optimized TCP Preposition Flows: 0
Current Active Auto-Discovery Flows: 0
Current Reserved Flows: 10
Current Active Pass-Through Flows: 0
Historical Flows: 375
D:DRE,L:LZ,T:TCP Optimization RR:Total Reduction Ratio
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
ConnID Source IP:Port Dest IP:Port PeerID Accel RR
343 10.10.10.10:3300 10.10.100.100:80 00:14:5e:84:24:5f T 00.0% <------
You will find the connection IDs for each connection listed at the end of the output. To view the statistics for a specific connection, use the show statistics connection conn-id command, as follows:
WAE# sh stat connection conn-id 343
Connection Id: 343
Peer Id: 00:14:5e:84:24:5f
Connection Type: EXTERNAL CLIENT
Start Time: Tue Jul 14 16:00:30 2009
Source IP Address: 10.10.10.10
Source Port Number: 3300
Destination IP Address: 10.10.100.100
Destination Port Number: 80
Application Name: Web <-----Application name
Classifier Name: HTTP <-----Classifier name
Map Name: basic
Directed Mode: FALSE
Preposition Flow: FALSE
Policy Details:
Configured: TCP_OPTIMIZE + DRE + LZ <-----Configured policy
Derived: TCP_OPTIMIZE + DRE + LZ
Peer: TCP_OPTIMIZE + DRE + LZ
Negotiated: TCP_OPTIMIZE + DRE + LZ <-----Policy negotiated with peer
Applied: TCP_OPTIMIZE + DRE + LZ <-----Applied policy
. . .
The Application Name and Classifier Name fields tell you the application and classifier applied to this connection.
The optimization policies are listed in the Policy Details section. If the Configured and Applied policies do not match, it means that you configured one policy for this type of connection but a different policy was applied. This could result from the peer being down, misconfigured, or overloaded. Check the peer WAE and its configuration.
The following section of output shows DRE encode/decode-related statistics including the number of messages, how many had DRE applied, LZ applied, or bypassed DRE and LZ:
. . .
DRE: 353
Conn-ID: 353 10.10.10.10:3304 -- 10.10.100.100:139 Peer No: 0 Status: Active
------------------------------------------------------------------------------
Open at 07/14/2009 16:04:30, Still active
Encode:
Overall: msg: 178, in: 36520 B, out: 8142 B, ratio: 77.71% <-----Overall compression
DRE: msg: 1, in: 356 B, out: 379 B, ratio: 0.00% <-----DRE compression ratio
DRE Bypass: msg: 178, in: 36164 B <-----DRE bypass
LZ: msg: 178, in: 37869 B, out: 8142 B, ratio: 78.50% <-----LZ compression ratio
LZ Bypass: msg: 0, in: 0 B <-----LZ bypass
Avg latency: 0.335 ms Delayed msg: 0 <-----Avg latency
Encode th-put: 598 KB/s <-----In 4.3.3 and earlier only
Message size distribution:
0-1K=0% 1K-5K=0% 5K-15K=0% 15K-25K=0% 25K-40K=0% >40K=0% <-----In 4.3.3 and earlier only
Decode:
Overall: msg: 14448, in: 5511 KB, out: 420 MB, ratio: 98.72% <-----Overall compression
DRE: msg: 14372, in: 5344 KB, out: 419 MB, ratio: 98.76% <-----DRE compression ratio
DRE Bypass: msg: 14548, in: 882 KB <-----DRE bypass
LZ: msg: 14369, in: 4891 KB, out: 5691 KB, ratio: 14.07% <-----LZ compression ratio
LZ Bypass: msg: 79, in: 620 KB <-----LZ bypass
Avg latency: 4.291 ms <-----Avg latency
Decode th-put: 6946 KB/s <-----In 4.3.3 and earlier only
Message size distribution:
0-1K=4% 1K-5K=12% 5K-15K=18% 15K-25K=9% 25K-40K=13% >40K=40% <-----Output from here in 4.3.3 and earlier only
. . .
The following statistics are highlighted in the above example for both encoding and decoding:
Overall ratio - the overall compression ratio for the data including both DRE and LZ
DRE ratio - the compression ratio due to DRE alone
DRE Bypass - the number of messages and bytes that bypassed DRE
LZ ratio - the compression ratio due to LZ alone
LZ Bypass - the number of messages and bytes that bypassed LZ
Avg latency - the average latency for the encode or decode operation
If you see a large amount of bypass traffic, the DRE compression ratio will be smaller than expected. It could be due to encrypted traffic, small messages, or otherwise uncompressible data. Consider contacting TAC for further troubleshooting help.
If you see a large amount of LZ Bypass traffic, this could be due to a large amount of encrypted traffic, which is not generally compressible.
The Average latency numbers can be useful for debugging a throughput issue. Both the encode and decode average latency should generally be less than 30 ms. If users experience low throughput and one or both of these numbers are higher, it indicates an issue with encoding or decoding, generally on the side with the higher latency.
It may be useful to look at the DRE statistics data such as the oldest usable data, cache size, percent of cache used, hash table RAM used, and so on by using the show statistics dre detail command, as follows:
WAE# sh stat dre detail
Cache:
Status: Usable, Oldest Data (age): 10h <-----Cache age
Total usable disk size: 311295 MB, Used: 0.32% <-----Percent cache used
Hash table RAM size: 1204 MB, Used: 0.00% <-----Output from here is in 4.3.3 and earlier only
. . .
If you are not seeing significant DRE compression, it could be because the DRE cache is not populated with enough data. Check if the cache age is short and less than 100 percent of the cache is used, which would indicate this situation. The compression ratio should improve as the cache fills with more data. If 100% of the cache is used and the cache age is short, it indicates that the WAE may be undersized and not able to handle the traffic volume.
If you are not seeing significant DRE compression, look at the Nack/R-tx counters in the following section of command output:
Connection details:
Chunks: encoded 398832, decoded 269475, anchor(forced) 43917(9407) <-----In 4.3.3 and earlier only
Total number of processed messges: 28229 <-----In 4.3.3 and earlier only
num_used_block per msg: 0.053597 <-----In 4.3.3 and earlier only
Ack: msg 18088, size 92509 B <-----In 4.3.3 and earlier only
Encode bypass due to: <-----Encode bypass reasons
remote cache initialization: messages: 1, size: 120 B
last partial chunk: chunks: 482, size: 97011 B
skipped frame header: messages: 5692, size: 703 KB
Nacks: total 0 <-----Nacks
R-tx: total 0 <-----Retransmits
Encode LZ latency: 0.133 ms per msg
Decode LZ latency: 0.096 ms per msg
. . .
The Nacks and R-tx counters should generally be low relative to the traffic volume. For example, about 1 per 100 MB of original (unoptimized) traffic. If you see significantly higher counts, it could indicate a DRE cache synchronization problem. Use the clear cache dre command to clear the DRE cache on all devices, or contact TAC.
The encode bypass reasons counters report the number of bytes bypassed due to various reasons. This can help you determine what is causing bypass traffic (other than an unoptimizable data pattern).
It is sometimes helpful to identify the connected and active peer WAEs and look at peer statistics, which you can do with the show statistics peer dre command as follows:
WAE# sh stat peer dre
Current number of connected peers: 1
Current number of active peers: 1
Current number of degrade peers: 0
Maximum number of connected peers: 1
Maximum number of active peers: 1
Maximum number of degraded peers: 0
Active peer details:
Peer-No : 0 Context: 65027
Peer-ID : 00:14:5e:95:4a:b5
Hostname: wae7.example.com <-----Peer hostname
------------------------------------------------------------------------------
Cache: Used disk: 544 MB, Age: 14d23h <-----Peer cache details in 4.3.3 and earlier only
Cache: Used disk: 544 MB <-----Peer cache details in 4.4.1 and later only
Peer version: 0.4 <-----
Ack-queue size: 38867 KB |
Buffer surge control: |<---In 4.3.3 and earlier only
Delay: avg-size 0 B, conn: 0, flush: 0 |
Agg-ft: avg-size 20902 B, conn: 388, flush: 0 |
remote low-buff: 0, received flush: 0 <-----
Connections: Total (cumulative): 3226861, Active: 597
Concurrent Connections (Last 2 min): max 593, avg 575
. . .
Other output from this command shows the encode and decode statistics similar to an individual connection.
------------------------------------------------
For TFO Troubleshooting you can use the following troubleshooting steps:
The number of TCP connections, their status, and disposition can give an indication of the health of the WAAS system in a specific location. A healthy system will show a large number of connections, with a significantly large percentage of these closed normally. The show statistics tfo detail command provides an indication of the volume, status, and disposition of connections between a particular WAAS device and other devices in the network.
You can view global TFO statistics by using the show statistics tfo detail command as follows:
WAE# show statistics tfo detail
Total number of connections : 2852
No. of active connections : 3 <-----Active connections
No. of pending (to be accepted) connections : 0
No. of bypass connections : 711
No. of normal closed conns : 2702
No. of reset connections : 147
Socket write failure : 0
Socket read failure : 0
WAN socket close while waiting to write : 0
AO socket close while waiting to write : 2
WAN socket error close while waiting to read : 0
AO socket error close while waiting to read : 64
DRE decode failure : 0
DRE encode failure : 0
Connection init failure : 0
WAN socket unexpected close while waiting to read : 32
Exceeded maximum number of supported connections : 0
Buffer allocation or manipulation failed : 0
Peer received reset from end host : 49
DRE connection state out of sync : 0
Memory allocation failed for buffer heads : 0
Unoptimized packet received on optimized side : 0
Data buffer usages:
Used size: 0 B, B-size: 0 B, B-num: 0
Cloned size: 0 B, B-size: 0 B, B-num: 0
Buffer Control:
Encode size: 0 B, slow: 0, stop: 0
Decode size: 0 B, slow: 0, stop: 0
Scheduler:
Queue Size: IO: 0, Semi-IO: 0, Non-IO: 0
Total Jobs: IO: 1151608, Semi-IO: 5511278, Non-IO: 3690931
Policy Engine Statistics
-------------------------
Session timeouts: 0, Total timeouts: 0
Last keepalive received 00.5 Secs ago
Last registration occurred 15:00:17:46.0 Days:Hours:Mins:Secs ago
Hits: 7766, Update Released: 1088
Active Connections: 3, Completed Connections: 7183
Drops: 0
Rejected Connection Counts Due To: (Total: 0)
Not Registered : 0, Keepalive Timeout : 0
No License : 0, Load Level : 0
Connection Limit : 0, Rate Limit : 0 <-----Connection limit overload
Minimum TFO : 0, Resource Manager : 0
Global Config : 0, TFO Overload : 0
Server-Side : 0, DM Deny : 0
No DM Accept : 0
. . .
The No. of active connections field reports the number of connections that are currently being optimized.
In the Policy Engine Statistics section of the output, the Rejected Connection Counts section show various reasons why connections have been rejected. The Connection Limit counter reports the number of times that a connection has been rejected because the maximum number of optimized connections has been exceeded. If you see a high number here, you should look into overload conditions. See the article Troubleshooting Overload Conditions for more information.
Additionally, TFO optimization for connections that are pushed down from other AOs because they cannot optimize the traffic is handled by the generic AO, which is covered in the article Troubleshooting the Generic AO.
You can view TFO connection statistics by using the show statistics connection command.
------------------------------------------
How to Monitor TFO Flows and Overload Conditions
When a WAAS accelerator device is overloaded, you typically see the following Central Manager alarm: Entering overload state due to Max connections (nnn). The number nnn is the number of times the WAE has become overloaded since the last reboot.
The device also logs a syslog error message similar to the following: Sysmon: %WAAS-SYSMON-3-445015: Fault detected: The TFO accelerator is overloaded (connection limit)
You can use various show commands at the CLI to determine the number of allowed and actual connections and gather more information.
Checking the TCP Connection Limit
The first useful command is show tfo detail, which can tell you how many optimized TFO connections that the device can handle, as follows:
wae-7341# show tfo detail
Policy Engine Config Item Value
------------------------- -----
State Registered
Default Action Use Policy
Connection Limit 12000 <-----Maximum number of TFO optimized connections
Effective Limit 11988
Keepalive timeout 3.0 seconds
The Connection Limit value tells you that this WAAS device can support 12000 TFO optimized connections.
The Effective Limit may be lower than the Connection Limit if the MAPI AO has reserved some connections. The reserved connections are subtracted from the Connection Limit to get the Effective Limit.
For Checking the Optimized TCP Connections use the following steps:
To understand the TCP flows on the device, you can use the show statistics connection command (in version 4.1.1, use the show statistics connection all command). This command displays the currently handled TFO/DRE/LZ flows, pass-through flows, and flows that are being handled by a specific application accelerator. An example of this command follows:
wae# show statistics connection
Current Active Optimized Flows: 5
Current Active Optimized TCP Plus Flows: 5
Current Active Optimized TCP Only Flows: 0
Current Active Optimized TCP Preposition Flows: 0
Current Active Auto-Discovery Flows: 0
Current Reserved Flows: 12 <---------- Added in 4.1.5
Current Active Pass-Through Flows: 0
Historical Flows: 143
D:DRE,L:LZ,T:TCP Optimization,
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
ConnID Source IP:Port Dest IP:Port PeerID Accel
92917 10.86.232.131:41197 70.70.7.11:3268 00:1a:64:69:19:fc TDL
92918 10.86.232.131:41198 70.70.7.11:3268 00:1a:64:69:19:fc TDL
92921 10.86.232.131:41216 70.70.7.11:3268 00:1a:64:69:19:fc TDL
94458 10.86.232.131:45354 70.70.7.11:1026 00:1a:64:69:19:fc TDL
36883 10.86.232.136:1857 10.86.232.131:1026 00:1a:64:69:19:fc TDL
From the first line in the output (Current Active Optimized Flows), you can see that the device currently has five active optimized flows. From the second counter (Current Active Optimized TCP Plus Flows), you can see that they are all being handled with TFO/DRE/LZ optimization (TFO Plus means that DRE and/or LZ optimization is being used in addition to TFO). The third counter (Current Active Optimized TCP Only Flows) shows flows that are optimized by TFO only.
Another useful counter is the Current Active Auto-discovery Flows, which displays flows that have not been fully set up to become optimized flows or pass-through flows.
To become fully set up, the connection must see the SYN, SYN ACK, ACK handshake, which is useful to note when dealing with an overload condition.
The Current Active Pass-Through Flows counter shows connections that the device has determined to be pass-through or where the device did not see the SYN, SYN ACK, ACK setup. These flows will not be counted as optimized flows. For pass-through flows, a device should be able to handle up to 10 times the number of optimized flows for which it is rated.
The Current Reserved Flows counter shows the number of connections reserved for the MAPI accelerator. For more details about reserved MAPI connections and their impact on device overload, see the section MAPI Application Accelerator Reserved Connections Impact on Overload.
The sum of the following three counters tells you how close the WAE device is to its connection limit:
Another command that you can use to see the number of TFO flows currently on a device is the show statistics tfo detail command. Two of the most useful counters in the output are "No. of active connections" and under the Policy Engine Statistics the "Active connections", as follows:
wae# show statistics tfo detail
Total number of connections : 22915
No. of active connections : 3 <-----Current optimized connections
No. of pending (to be accepted) connections : 0
No. of bypass connections : 113
No. of normal closed conns : 19124
No. of reset connections : 3788
Socket write failure : 2520
Socket read failure : 0
WAN socket close while waiting to write : 1
AO socket close while waiting to write : 86
WAN socket error close while waiting to read : 0
AO socket error close while waiting to read : 80
DRE decode failure : 0
DRE encode failure : 0
Connection init failure : 0
WAN socket unexpected close while waiting to read : 1048
Exceeded maximum number of supported connections : 0
Buffer allocation or manipulation failed : 0
Peer received reset from end host : 53
DRE connection state out of sync : 0
Memory allocation failed for buffer heads : 0
Unoptimized packet received on optimized side : 0
Data buffer usages:
Used size: 0 B, B-size: 0 B, B-num: 0
Cloned size: 54584 B, B-size: 73472 B, B-num: 111
Buffer Control:
Encode size: 0 B, slow: 0, stop: 0
Decode size: 0 B, slow: 0, stop: 0
AckQ Control:
Total: 0, Current: 0
Scheduler:
Queue Size: IO: 0, Semi-IO: 0, Non-IO: 0
Total Jobs: IO: 219110, Semi-IO: 186629, Non-IO: 49227
Policy Engine Statistics
-------------------------
Session timeouts: 0, Total timeouts: 0
Last keepalive received 00.0 Secs ago
Last registration occurred 8:03:54:38.7 Days:Hours:Mins:Secs ago
Hits: 52125, Update Released: 17945
Active Connections: 3, Completed Connections: 37257 <-----Active Connections
Drops: 0
Rejected Connection Counts Due To: (Total: 12)
Not Registered : 12, Keepalive Timeout : 0
No License : 0, Load Level : 0
Connection Limit : 0, Rate Limit : 0 <-----Connection Limit
Minimum TFO : 0, Resource Manager : 0
Global Config : 0, Server-Side : 0
DM Deny : 0, No DM Accept : 0
Auto-Discovery Statistics
-------------------------
Total Connections queued for accept: 22907
Connections queuing failures: 0
Socket pairs queued for accept: 0
Socket pairs queuing failures: 0
AO discovery successful: 0
AO discovery failure: 0
In some cases, the two counters will differ and the reason is that the “No. of active connections” displays all the current flows that being optimized by TFO, TFO/DRE, TFO/DRE/LZ, and TFO/DRE/LZ and an application accelerator. The “Active Connections” under the policy engine statistics includes all the flows in the state above plus the connections that are optimized only by TFO and an application accelerator. This situation means that a TCP flow has come in and matched an application accelerator classifier but the SYN, SYN ACK, ACK handshake has not been completed.
In many TFO overload cases, if the problem is still occurring, you can look at these commands and determine if the number of optimized flows is around the rated number of optimized TCP connections for the hardware. If it is, then you can view the flow details and see what is using up all the flows to determine if this traffic is legitimate and overloading the device or is a virus, security scanner, or something else occurring on the network.
The "Connection Limit" counter under the policy engine statistics reports the number of connections rejected and passed through because the WAE has exceeded its rated number of optimized TCP connections. If this counter is high, it means the WAE is frequently getting more connections than it can handle.
If the number of optimized connections is not close to the rated number of optimized TCP connections and you are still getting an overload alarm, then you should look at the Current active auto-discovery flows from the show statistics connection command or the “Active Connections” under Policy
Engine Statistics from the show statistics tfo detail command. In some cases, the number of optimized connections can be very low but the Active Connections under the Policy Engine Statistics are roughly equal to the rated number of optimized flows for the hardware. This situation means that there are many flows that match a classifier but they are not fully established. When a TCP SYN matches a classifier, it will reserve an optimized connection. This connection will not appear in the optimized TCP connections count until the TCP handshake is finished and optimization starts. If the device determines that the flow should not be optimized, it will be removed from the count of active connections under the Policy Engine Statistics.
To further troubleshoot cases where TFO overload is occurring and the Policy Engine Statistics Active Connections seem to be using up all the optimized TCP connections on the device, use the show statistics accelerator detail command. In the output of this command, look at the Active Connections under the Policy Engine Statistics for each application accelerator to determine which application accelerator is receiving these connections that are not fully established.
Next, look at what state these flows might be in by using the show statistics filtering command, which provides you with the number of filtering tuples on the device, as follows:
wae# show statistics filtering
Number of filtering tuples: 18
Number of filtering tuple collisions: 0
Packets dropped due to filtering tuple collisions: 0
Number of transparent packets locally delivered: 965106
Number of transparent packets dropped: 0
Packets dropped due to ttl expiry: 0
Packets dropped due to bad route: 10
Syn packets dropped with our own id in the options: 0
Syn-Ack packets dropped with our own id in the options: 0
Internal client syn packets dropped: 0
Syn packets received and dropped on estab. conn: 0
Syn-Ack packets received and dropped on estab. conn: 0
Syn packets dropped due to peer connection alive: 525
Syn-Ack packets dropped due to peer connection alive: 0
Packets recvd on in progress conn. and not handled: 0
Packets dropped due to peer connection alive: 1614
Packets dropped due to invalid TCP flags: 0
Packets dropped by FB packet input notifier: 0
Packets dropped by FB packet output notifier: 0
Number of errors by FB tuple create notifier: 0
Number of errors by FB tuple delete notifier: 0
Dropped WCCP GRE packets due to invalid WCCP service: 0
Dropped WCCP L2 packets due to invalid WCCP service: 0
Number of deleted tuple refresh events: 0
Number of times valid tuples found on refresh list: 0
The number of filtering tuples is the number of flows on the device that are optimized, in pass-through, in FIN WAIT state, in setup state, and so on. Each established flow appears as two tuples, one for each side of the flow, so the number that you see in this output may be much larger than the number of flows that you are seeing in the other commands.
To get more information on the flows in the filtering list, you can use the show filtering list command as follows:
wae# show filtering list
E: Established, S: Syn, A: Ack, F: Fin, R: Reset
s: sent, r: received, O: Options, P: Passthrough
B: Bypass, L: Last Ack, W: Time Wait, D: Done
T: Timedout, C: Closed
Local-IP:Port Remote-IP:Port Tuple(Mate) State
10.86.232.82:23 10.86.232.134:41784 0xbc1ae980(0x0 ) E
10.86.232.131:58775 70.70.7.11:3268 0x570b2900(0x570b2b80) EW
70.70.7.11:3268 10.86.232.131:58775 0x570b2b80(0x570b2900) EDL
70.70.7.11:3268 10.86.232.131:57920 0x570b2d80(0x570b2800) E
10.86.232.131:57920 70.70.7.11:3268 0x570b2800(0x570b2d80) E
10.86.232.82:23 161.44.67.102:4752 0xbc1aee00(0x0 ) E
10.86.232.131:58787 70.70.7.11:1026 0x570b2080(0x570b2e80) EW
70.70.7.11:1026 10.86.232.131:58787 0x570b2e80(0x570b2080) EDL
10.86.232.131:48698 70.70.7.11:1026 0x570b2f00(0x570b2880) PE
10.86.232.131:58774 70.70.7.11:389 0x570b2300(0x570b2180) EW
70.70.7.11:389 10.86.232.131:58774 0x570b2180(0x570b2300) EDL
10.86.232.131:58728 70.70.7.11:1026 0x570b2380(0x570b2a00) E
10.86.232.131:58784 70.70.7.11:1026 0x570b2e00(0x570b2980) EW
70.70.7.11:1026 10.86.232.131:58784 0x570b2980(0x570b2e00) EDL
70.70.7.11:1026 10.86.232.131:48698 0x570b2880(0x570b2f00) PE
10.86.232.131:58790 70.70.7.11:3268 0x570b2100(0x570b2c80) EW
70.70.7.11:3268 10.86.232.131:58790 0x570b2c80(0x570b2100) EDL
If the show statistics accelerator all command shows you which application accelerator is using up all the optimized TFO connections, you can filter on that port or traffic. For example, if you want to filter on port 80 traffic, use the show filtering list | I :80 command.
Look at the legend in the State column. In the case where the flows are in the SYN state, you may see a lot of flows with a state of S. If the WAE has sent back the SYN ACK with options set you may see the state SAsO. This indication may help you determine the state of the flow and from there, you can determine if there is a routing problem, virus, or a problem with the WAE not releasing connections. You may need traces to determine exactly what is happening to the flows but the commands above should give you an idea of what to look for.
MAPI Application Accelerator Reserved Connections Impact on Overload
Often, a TFO overload can be caused by the MAPI application accelerator reserved connections, so it is helpful to understand the process of how the MAPI application accelerator reserves connections.
The MAPI application accelerator reserves TFO connections to ensure that it will have enough connections available to it to accelerate all current and future connections that the clients will make to the Exchange servers. It is normal for a MAPI client to make multiple connections. If a client makes the initial connection through the MAPI application accelerator, but the subsequent connections fail in the MAPI application accelerator, there is a risk that the client's connection might fail.
In order to avoid these potential connection failures, the MAPI application accelerator reserves connection resources as follows:
Before any client connections begin, it reserves 10 connections for itself, as a buffer for anticipated new connections.
For each client connection to the server, it reserves three TFO connections for that client-server pair and one of the three is used as an active connection for this first connection. If the same client makes a second or third connection to the same server, those are handled out of the reserved connection pool. If a client only ever makes a single connection to the server, those two reserved connections will be unused and remain in the reserved pool. If the client makes a connection to a different server, three new connections are again reserved for that client-server pair.
All of these reserved connections are designed to improve performance and to reduce the possibility of a client connection failing because of the inability to make additional connections through the MAPI application accelerator.
Overload occurs when Current Active Optimized Flows + Current Active Auto-Discovery Flows + Current Reserved Flows is greater than the device's fixed Connection Limit.
In general, new connections would then be passed through. But some new MAPI connections may still be optimized. When the device is at the overload point, if a client makes an additional request to a MAPI server it already has connected to, then reserved connections are used. But if there are not enough reserved connections (for example, if a client makes a fourth connection to the same MAPI server and the WAE is already in overload) then an escaped connection condition might occur, and this could lead to erroneous behavior such as a client receiving many duplicate copies of the same single mail message.
If the system did not forward the connection to the MAPI application accelerator, you should see "PT Rjct Resources" or "PT in progress", depending on whether there is activity on the connection. If the connection was forwarded to the MAPI application accelerator and then the reservation failed, the connection will be marked with a "G" for the Accelerator, instead of an "M" (in the show statistics connection optimized mapi command output). For an example of this command, see the article Troubleshooting the MAPI AO.
If you are experiencing frequent overload conditions, it is important to understand how the Outlook clients are making connections (how many connections to how many Exchange servers). With Outlook running on a client, hold the Ctrl key while you right-click on the Outlook icon in the system tray on the task bar. Choose Connection Status to display the list of servers to which the Outlook client has connected. From that you can see how many connections the client is making and to how many different Exchange servers. If the client is making connections to several different servers, it would be helpful to investigate ways to consolidate mail so a user only opens MAPI connections to a single Exchange server, and makes use of multiple connections to that server.
It is also useful to investigate whether there are any other applications that might be making MAPI connections.
Solutions for Overload Conditions
Examine optimized connections to see if they are legitimate. In many cases, a Denial of Service (DoS) attack encountered in the network may be causing the WAE to attempt to optimize connections. If so, employ a DoS protection mechanism in the network to proactively close the connections.
In cases where the connections are legitimate, the WAE deployed in the location is undersized and may need to be upgraded, or an additional WAE can be deployed to increase scalability within that site.
..................................................................
-------------------------------------------------------------
Use the following URL for for monitoring :
http://www.cisco.com/en/US/docs/app_ntwk_services/waas/waas/v401/configuration/guide/monitor.html
-----------------------------------------
Please rate if it help you any.
HTH
Sachin
06-27-2011 10:54 AM
I don't have a problem with the CIFS acceleration or any other application acceleration service. Can you tell me how to view the eviction for DRE cache if it exists, not just for CIFS AO. I want to compare the oldest data age from the show stat dre cache command to the last eviction. How can I determine how many times a day is the purging happening?????? I need to understand if we need to add additional WAE-7371s to the core environment if it can't keep up with the caching.
show stat dre
Cache:
Status: Usable, Oldest Data (age): 6d22h
Total usable disk size: 972799 MB, Used: 99.75%
Hash table RAM size: 3588 MB, Used: 93.00%
Connections: Total (cumulative): 887721158 Active: 5115
Encode:
Overall: msg: 15713080635, in: 30026812 MB, out: 8980472 MB, ratio: 70.09%
DRE: msg: 14542620806, in: 29719427 MB, out: 13830638 MB, ratio: 53.46%
DRE Bypass: msg: 2153943270, in: 307247 MB
LZ: msg: 13387089320, in: 8289595 MB, out: 3116934 MB, ratio: 62.40%
LZ Bypass: msg: 2325991315, in: 5848290 MB
Avg latency: 0.300 ms Delayed msg: 6522988262
Encode th-put: 6517 KB/s
Message size distribution:
0-1K=79% 1K-5K=12% 5K-15K=4% 15K-25K=2% 25K-40K=1% >40K=0%
Decode:
Overall: msg: 15546149095, in: 5384756 MB, out: 15545432 MB, ratio: 65.36%
DRE: msg: 13905506768, in: 8660775 MB, out: 14644398 MB, ratio: 40.86%
DRE Bypass: msg: 3639425455, in: 901057 MB
LZ: msg: 13602881560, in: 3116432 MB, out: 7311753 MB, ratio: 57.38%
LZ Bypass: msg: 1943267535, in: 2268323 MB
Avg latency: 0.188 ms
Decode th-put: 5445 KB/s
Message size distribution:
06-27-2011 11:43 AM
Hi John,
Kindly send the output of the following two commands as well:
WAE# show statistics dre connection
Conn Peer Client-ip:port Server-ip:port Encode-in/ Status
Id No Decode-in (A-Active)
(C-Closed)
99674 1 10.10.13.100:1559 10.10.10.100:445 737B/ 1KB C(6m46s)
99673 1 10.10.13.100:1560 10.10.10.100:139 0B/ 0B C(6m57s)
99645 1 10.10.13.100:1558 10.10.10.100:80 487MB/ 0B C(7m49s)
WAE# show statistics dre connection id 99645
Conn-ID: 99645 10.10.13.100:1558 -- 10.10.10.100:80 Peer No: 1 Status: Closed
------------------------------------------------------------------------------
Open at 10/20/2007 17:37:52, Close at 10/20/2007 17:38:32, Duration: 40 secs
Encode:
Overall: msg: 15427, in: 487 MB, out: 5416 KB, ratio: 98.92%
DRE: msg: 15427, in: 487 MB, out: 5691 KB, ratio: 98.86%
DRE Bypass: msg: 0, in: 0 B
LZ: msg: 1562, in: 932 KB, out: 657 KB, ratio: 29.49%
LZ Bypass: msg: 13865, in: 4759 KB
Avg latency: 1.405 ms
Message size distribution:
0-1K=1% 1K-5K=4% 5K-15K=13% 15K-25K=16% 25K-40K=30% >40K=33%
Decode:
Overall: msg: 0, in: 0 B, out: 0 B, ratio: 0.00%
DRE: msg: 0, in: 0 B, out: 0 B, ratio: 0.00%
DRE Bypass: msg: 0, in: 0 B
LZ: msg: 0, in: 0 B, out: 0 B, ratio: 0.00%
LZ Bypass: msg: 0, in: 0 B
Avg latency: 0.000 ms
Message size distribution:
0-1K=0% 1K-5K=0% 5K-15K=0% 15K-25K=0% 25K-40K=0% >40K=0%
HTH
sachin
06-27-2011 11:46 AM
I'm running version 4.2.1. I don't have those options.
ryewaas1#show statistics dre ?
detail Display detailed DRE statistics
| Output Modifiers
06-27-2011 01:56 PM
kindly attach the output of the following commands
#show stat dre detail
and
#show tech
for suggesting more.
Sachin
06-28-2011 07:11 AM
I've logged a case with TAC - SR 618217149. You can view the show tech log from there and the notes. I'm looking for a way to see how frequent the purging is occurring on the DRE cache, without logging into the CLI all the time and do a show stat dre.
Thanks for the help.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: