cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2658
Views
0
Helpful
12
Replies

What is an evicted resource?

erictgunn
Level 1
Level 1

Just inherited support of our Cisco Waas devices, one devie inparticular dosent seem to be working with local file sharing, while web caching appeas to be working as normal.

In this device IS see the following: (I assume this is a problem as cache size and usage are almost the same and seeing the file evictions)

On a related not what exactly is an evicted resource? Could not seem to find the answer in the Cisco PDF manual.

81.75488 GB
81.75411 GB
5000000
159219
939372
Wed Apr 27 10:40:10 CDT 2011
96
94
95
94
5.7103443 week
Wed Apr 27 10:40:10 CDT 2011

12 Replies 12

Nicolas Fournier
Cisco Employee
Cisco Employee

Hi Eric,

When the cache of your device is almost full (and it seems to be the case on your setup), the device is going to start deleting some of the cached objects to make room for new ones that comes in, the ones which are selected are the ones with the oldest time-last-accessed values in WAAS.

This deletion process is also called eviction, the number you see in your output is simply the number of cached objects which have been deleted to make room for new ones.

Regards,

Nicolas

ktunugun
Cisco Employee
Cisco Employee

Hi,

Eviction is the automated method for preventing cache from filling. It is done for both disk cache and resource (metadata) cache
Eviction occurs BEFORE the disk and resource limits are reached, and becomes more aggressive with higher usage
Disk and resource eviction occurs independently.
Eviction starts when high-watermark of 95% is reached

Priority is given to client requests

Aggressive eviction starts above 95%

Priority given to eviction

Urgent eviction starts at 98%

No client requests handled

Eviction stops at low-watermark threshold of 94%
It seems your device is doing urgent eviction which is why no client requests are handled.
Regards
Kiran.

Peer relationships time out after 10 minutes of inactivity (that is, no active connections are established between two peers for 10 minutes). When the peer relationship is timed out, it becomes reusable by another peer. Data stored in the DRE compression history remains intact even if a peer becomes disconnected due to inactivity, unless the DRE compression history becomes full. In cases where the DRE compression history becomes full, an eviction process is initiated to remove the oldest set of data in the DRE compression history to make room for new data.

In the navigation area, click Monitoring under the WAFS Edge menu.

The WAFS Edge Monitoring window appears.

The Cache tab displays the following information:

•Maximum cache disk size—Maximum amount of disk space (in gigabytes) allocated to the WAFS Edge device cache.

•Current cache disk usage—Current amount of disk space (in kilobytes) used by the WAFS Edge device cache. The name of this statistic is a link that you can use to display its historical graphs (without first going to the Graphs tab).

•Maximum cache resources—Maximum number of resources (files and directories) allowed in the WAFS Edge device cache.

•Current cache resources—Current number of resources contained in the WAFS Edge device cache. The name of this statistic is a link that you can use to display its historical graphs (without first going to the Graphs tab).

Evicted resources count—Number of resources that have been evicted from the cache since the Edge WAE was started.

Last eviction time—Time when a cache eviction last occurred.

•Cache size high watermark—Percentage of disk usage that causes the WAFS Edge device to begin evicting resources.

•Cache size low watermark—Percentage of disk usage that causes the WAFS Edge device to stop evicting resources.

•Cache resources high watermark—Percentage of total cache resources that causes the WAFS Edge device to begin evicting resources.

•Cache resources low watermark—Percentage of total cache resources that causes the WAFS Edge device to stop evicting resources.

Last evicted resource age—Amount of time that the last-evicted resource spent in the WAFS Edge device cache.

Last evicted resource access time—Last time that the last-evicted resource was accessed.

Along with providing the user-side proxy functionality necessary for accelerating CIFS, the edge services also provide termination of the CIFS proxy connections that are established between edge and core WAAS devices. WAAS devices configured with edge services provide a client-side object cache and metadata cache for CIFS, which is independent of the DRE compression history. By keeping the application cache independent of the compression history, application performance can be maintained even in the event of massive compression history eviction, which could happen if a user were to, for instance, back up their entire home movie or music library over the WAN. Furthermore, having a separate application cache ensures that failures and cache-clearing events are isolated from one another. As an example, an application cache may be used to hold numerous gigabytes of distributed software or patches, which, if evicted, may need to be transferred over the WAN again. Keeping the cache and compression history isolated from one another provides numerous benefits in terms of operational efficiency.

In essence, the core acts as a "staging area," because it is unaware of the status of the edge cache. The core, once it receives the manifest from the origin server (based on the job configuration), passes the manifest to the edge. Then, the core begins fetching files from the origin server based on the manifest. These files are then stored in a temporary staging area, which is ~1 GB in capacity. As the edge fetches files, they are removed from the staging area and the core continues to fetch additional files from the origin server. If the staging area becomes full of files, and the edge requests a file that is not in the staging area, the core will know that the edge does not need any of the files found in the staging area. In such a case, the core will empty its staging area and begin fetching files from the origin server starting with the file requested by the edge. Because the core and edge are working from the same manifest, the core can adapt and evict files from the staging area based on the requests coming from the edge.

HTH

Sachin

Can you explain why the last eviction time is in the future? I need to understand when was the last time an eviction happened and how to correlate it to the oldest DRE cache.  Is the last eviction time shown below only for the CIFS AO? And if so, how can I see the last eviction time for the DRE cache? Also, when you do a show disk detail, the obj1 volume partition doesn't match the Maximum cache disk size.

/obj1            CONTENT    /dev/data1/obj    204338MB  142589MB   61749MB  69%  CIFS AO? Doesn't match 175GB.

/dre1            CONTENT    /dev/data1/dre    991932MB  973878MB   18054MB  98%  

DRE Cache Stats

Cache:

    Status: Usable, Oldest Data (age):6d14h

    Total usable disk size: 972799 MB,  Used: 99.57%

      Hash table RAM  size:   3588 MB,  Used: 93.00%

175.55469 GB
118.42443 GB
5000000
4500003
2018623
Sat Jun 25 00:44:22 UTC 2011
96
94
95
94
19.03269 week
Sat Jun 25 00:44:22 UTC 2011

Hi John,

Please check have you set the correct time zone and time on each WAE. Domain integration requires that each domain member, including WAEs, be no more than 5 minutes askew from the relative time on the domain controller. That is, the time difference on the WAE, after accounting for time zone differences, must be within 5 minutes of the relative time on the domain controller.

Kindly provide the output of the commands I have requested to look further into your configuration:

WAE# show clock

1.

The CIFS accelerator transparently optimizes CIFS traffic on ports 139 and 445.

You can verify the general AO configuration and status with the

# show accelerator

and

#show license

command

-------------------------------------------------------------------------------------------

2.

Next, verify the status that is specific to the CIFS AO by using the

#show accelerator cifs command,

You want to see that the CIFS AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is Enabled but the Operational State is Shutdown, it indicates a licensing problem.

------------------------------------------------------------------------------------------

3.

Use the show running-config command to verify that the CIFS traffic policy is properly configured. You want to see accelerate cifs for the WAFS application action and you want to see appropriate match conditions listed for the CIFS classifier, as follows:

WAE674# sh run | include CIFS

      classifier CIFS

    name WAFS classifier CIFS action optimize full accelerate cifs

WAE674# sh run | begin CIFS

...skipping

   classifier CIFS

      match dst port eq 139

      match dst port eq 445

   exit

---------------------------------------------------------------------------------------------------------

4.

Use the show statistics connection optimized cifs command to check that the WAAS device is establishing optimized CIFS connections. Verify that "TCDL" appears in the Accel column for a connection. A "C" indicates that the CIFS AO was used.

WAE674# sh stat conn opt cifs

Current Active Optimized Flows:                      3

   Current Active Optimized TCP Plus Flows:          3

   Current Active Optimized TCP Only Flows:          0

   Current Active Optimized TCP Preposition Flows:   1

Current Active Auto-Discovery Flows:                 0

Current Active Pass-Through Flows:                   0

Historical Flows:                                    100

D:DRE,L:LZ,T:TCP Optimization,

A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO

ConnID  Source IP:Port        Dest IP:Port          PeerID             Accel

1074    10.10.10.10:2704      10.10.100.100:445     00:14:5e:84:24:5f  TCDL       <------Look for "C"

If you see "TDL" in the Accel column, the connection was optimized by transport optimizations only and was not inspected by the CIFS AO. This situation can happen if the CIFS AO is disabled, the Enterprise license is not configured, or if the maximum connection limit is reached.

If you see a "G" instead of a "C" in the Accel column, then the connection was pushed down from the CIFS AO to the generic AO and was optimized with transport optimizations only. This situation can happen if the connection requires SMB2 or a digital signature and an error message is logged for it

-----------------------------------------------------------------------------

5.

You can view the CIFS connection statistics by using the show statistics connection optimized cifs detail command as follows:

WAE674# sh stat connection optimized cifs detail

Connection Id:            1801

   Peer Id:                  00:14:5e:84:24:5f

   Connection Type:          EXTERNAL CLIENT

   Start Time:               Thu Jun 25 06:15:58 2009

   Source IP Address:        10.10.10.10

   Source Port Number:       3707

   Destination IP Address:   10.10.100.100

   Destination Port Number:  139

   Application Name:         WAFS                                      <-----Should see WAFS

   Classifier Name:          CIFS                                      <-----Should see CIFS

   Map Name:                 basic

   Directed Mode:            FALSE

   Preposition Flow:         FALSE

   Policy Details:

          Configured:        TCP_OPTIMIZE + DRE + LZ

             Derived:        TCP_OPTIMIZE + DRE + LZ

                Peer:        TCP_OPTIMIZE + DRE + LZ

          Negotiated:        TCP_OPTIMIZE + DRE + LZ

             Applied:        TCP_OPTIMIZE + DRE + LZ

   Accelerator Details:

               Configured:   CIFS                                      <-----Should see CIFS configured

                  Derived:   CIFS

                  Applied:   CIFS                                      <-----Should see CIFS applied

                     Hist:   None

                                               Original            Optimized

                                   -------------------- --------------------

   Bytes Read:                                   189314             10352510

   Bytes Written:                              91649704                28512

. . .

Connection details:

Chunks: encoded 3,  decoded 49922,  anchor(forced) 0(1)

Total number of processed messges: 1820

num_used_block per msg: 0.140659

Ack: msg 1609,  size   7066 B                                         

Encode bypass due to:                                                 

    last partial chunk: chunks: 1,  size:    142 B                    

    skipped frame header: messages: 138,  size:  27202 B              

Nacks: total 0                                                        

R-tx: total 0

Encode LZ latency:      0.060 ms per msg

Decode LZ latency:      0.071 ms per msg

Aggregation encode:  Retransmissions: 0                                   <-----Packets lost between peers

    level 0: chunks:        3  hits:        0 miss:        3

    level 1: chunks:        0  hits:        0 miss:        0

    level 2: chunks:        0  hits:        0 miss:        0

    level 3: chunks:        0  hits:        0 miss:        0

Aggregation decode: Collisions: 0

    level 0: chunks:   174093  hits:   128716 miss:        0

    level 1: chunks:        0  hits:        0 miss:        0

    level 2: chunks:        0  hits:        0 miss:        0

    level 3: chunks:        0  hits:        0 miss:        0

Aggregation stack memory usage: Sender:    452 B  Receiver:   9119 B

Noise filter: Chunks: 0, Bytes:      0 B

. . .

If the Retransmissions counter increases, it means that packets are getting lost in the middle, between the two peer WAEs. This situation will result in lower throughput. You should investigate possible causes for packet lose in the network between the two peer WAEs.

You can view the CIFS request statistics by using the show statistics cifs requests command

------------------------------------------------------------------------------------------------------------

6. CIFS AO Logging

The following log files are available for troubleshooting CIFS AO issues:

Transaction log files: /local1/logs/tfo/working.log (and /local1/logs/tfo/tfo_log_*.txt)

CIFS internal log file: /local1/errorlog/cifs/cifs_err.log

Debug log files: /local1/errorlog/cifsao-errorlog.current (and cifsao-errorlog.*)

For easier debugging, you should first set up an ACL to restrict packets to one host.

WAE674(config)# ip access-list extended 150 permit tcp host 10.10.10.10 any

WAE674(config)# ip access-list extended 150 permit tcp any host 10.10.10.10

To enable transaction logging, use the transaction-logs configuration command as follows:

wae(config)# transaction-logs flow enable

wae(config)# transaction-logs flow access-list 150

You can view the end of a transaction log file by using the type-tail command as follows:

wae# type-tail tfo_log_10.10.11.230_20090715_130000.txt

:EXTERNAL CLIENT :00.14.5e.84.24.5f :basic :WAFS :CIFS :F :(DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO)

(DRE,LZ,TFO) : :(CIFS) (CIFS) (CIFS) : :  :180

Wed Jul 15 15:48:45 2009 :1725 :10.10.10.10 :2289 :10.10.100.100 :139 :OT :START :EXTERNAL CLIENT :00.14.5e.84.24.5f :basic :WAFS

:CIFS :F :(DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) : :(CIFS) (CIFS) (CIFS) : : :177

Wed Jul 15 15:48:55 2009 :1725 :10.10.10.10 :2289 :10.10.100.100 :139 :OT :END : EXTERNAL CLIENT :(CIFS) :159 :221

To set up and enable debug logging of the CIFS AO, use the following commands.

NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a production environment.

You can enable detailed logging to the disk as follows:

WAE674(config)# logging disk enable

WAE674(config)# logging disk priority detail

You can enable debug logging for connections in the ACL:

WAE674# debug connection access-list 150

The options for CIFS AO debugging are as follows:

WAE674# debug accelerator cifs ?

  all         enable all CIFS accelerator debugs

  shell       enable CIFS shell debugs

You can enable debug logging for CIFS connections and then display the end of the debug error log as follows:

WAE674# debug accelerator cifs all

WAE674# type-tail errorlog/cifsao-errorlog.current follow

---------------------------------------------------------------------------------------

7.

Send the output of the following:

WAE# show cifs cache entry-count

WAE# show cifs cache disk-use

WAE# show clock

Please check have you set the correct time zone and time on each WAE. Domain integration requires that each domain member, including WAEs, be no more than 5 minutes askew from the relative time on the domain controller. That is, the time difference on the WAE, after accounting for time zone differences, must be within 5 minutes of the relative time on the domain controller.

The simplest way to align time on the WAE device with the domain is to first set the time zone on each WAE and then configure each WAE to use a Network Time Protocol (NTP) server. Although this can be easily accomplished through the device CLI, recommended practice dictates that you create unique device groups, where each device group represents a separate time zone. After you add the appropriate WAE devices into each timezone-based device group, you can apply the NTP server settings to the device group.

Also send me the output of the following command:

WAE# show running-config

................................................................

8. Regarding more infor for your disks

It is also useful to check the Predictive Failure Analysis (PFA) flag for RAID-5 disks by using the show disks tech-support command. You will find the PFA flag at the end of the output. If the PFA flag is set to Yes, it indicates a predicted drive failure and you should replace the disk. A critical alarm is also raised on the WAE.

Also send the output of show disks tech-support command as follows:

wae# show disks tech-support

Controllers found: 1

----------------------------------------------------------------------

Controller information

----------------------------------------------------------------------

   Controller Status : Okay

   Channel description : SAS/SATA

   Controller Model : IBM ServeRAID 8k

   Controller Serial Number : 40453F0

   Physical Slot : 0

   Installed memory : 256 MB

   Copyback : Disabled

   Data scrubbing : Disabled

   Defunct disk drive count : 0

   Logical drives/Offline/Critical : 1/0/0

   ---------------------------------------------------

   Controller Version Information

   ---------------------------------------------------

   BIOS : 5.2-0 (15418)

   Firmware : 5.2-0 (15418)                             <-----Firmware version      

   Driver : 1.1-5 (2449)

   Boot Flash : 5.1-0 (15418)

   ---------------------------------------------------

................................................................

9. Send the output of the

#show license

# show tech

----

Once you send the output of the above commands I can check further.

Thanks

sachin

I think I understand it now, the eviction is occurring quite frequently on our core WAEs, so the time is almost daily. The clock is set to using our NTP servers, standard across the network. How can I view the last eviction time for the DRE cache?  It only shows it for the CFS AO when logging into the GUI for the WAE itself. Is it normal for the eviction to be occurring daily on the CIFS AO? Is there a log to see how many times it occurs? 

You have not send the output of any of the commands I have requested in my prevous post as it can help me in seeing more in details about your device and maybe your sample output can be helpful for others in future when they refer this post. But anyways if you dont want to publish that.

Please send me the output of following if possible:

WAE674# show statistics dre cache

sh statistics dre

Cache:

    Status: Usable, Oldest Data (age): 21m39s

    Total usable disk size: 116735 MB,  Used: 0.01%

      Hash table RAM  size:    417 MB,  Used: 0.00%

Connections:   Total (cumulative): 533   Active: 12

Encode:

   Overall: msg:       3984, in:   2003 KB, out:   1073 KB, ratio:  46.41%

       DRE: msg:       3644, in:   1942 KB, out:   1741 KB, ratio:  10.32%

DRE Bypass: msg:       1036, in:  62427 B

        LZ: msg:       2955, in:   1293 KB, out:    553 KB, ratio:  57.20%

LZ Bypass: msg:       1029, in:    508 KB

    Avg latency:      0.155 ms     Delayed msg:        599

  Encode th-put:   3244 KB/s

  Message size distribution:

    0-1K=87%  1K-5K=12%  5K-15K=0%  15K-25K=0%  25K-40K=0%  >40K=0%

Decode:

   Overall: msg:       4338, in:   1456 KB, out:   2685 KB, ratio:  45.76%

       DRE: msg:       3873, in:   2249 KB, out:   2547 KB, ratio:  11.70%

DRE Bypass: msg:       1495, in:    137 KB

        LZ: msg:       3326, in:    894 KB, out:   1837 KB, ratio:  51.33%

LZ Bypass: msg:       1012, in:    562 KB

    Avg latency:      0.084 ms

  Decode th-put:   7394 KB/s

  Message size distribution:

    0-1K=84%  1K-5K=13%  5K-15K=1%  15K-25K=0%  25K-40K=0%  >40K=0%

Will monitor for a few days.

Using the system message log feature of the WAAS Central Manager GUI, you can view information about events that have occurred in your WAAS network.

From the WAAS Central Manager GUI, choose System > Logs > System Messages. The System Message Log window appears.

From the WAAS Central Manager GUI, choose System > Logs > Audit Trail Logs

The following directories are used by Cisco WAAS for log

files:

–/local1—Root directory for all log files

/local1/logs—Service log files (aka ―admin‖ logs)

/local1/errorlog—Service log files (aka ―debug‖ logs)

–/local1/core_dir—Process core dump files

File system navigation commands:

–cd

–pwd

–dir

–type-tail [| | follow]

–find-pattern

----

Packets can be captured on all WAAS interfacesusing one of the following CLI tools:

–tethereal

–tcpdump

----------------

for Transaction Logs

Every transaction generates log

Multiple transaction attributes recorded

–TCP connection start time

–TCP connection end time

–Optimization done (AO, DRE, LZ, TFO, or PT)

–Flow identification information (L3/L4/L5)

–Bytes

• Origin received/sent

• Optimized received/sent

-----------

Transaction Logs – CM

- Enable transaction logging on branch WAEs

-  /local1/logs/tfo

- Archive log schedule

- The archive filenamesuse this format: tfo_log_IPADDRESS_YYYYMMDD_HHMMSS.txt

-Export log schedule

-----------

to see TFO Transaction Logs

WAE# cd logs/tfo

WAE#

WAE#ls

ftp_export.statustfo_log_22.1.43.10_20090508_190000.txt

tfo_log_22.1.43.10_20090508_200000.txt

tfo_log_22.1.43.10_20090508_210000.txt

working.log

WAE# type-tail working.log

-----------------------

you can use Sawmill Transaction Log Analysis

- Sawmill understands WAAS transaction logs

- Syslog or FTP/SFTP transfer

- Extensive reports

--------------------------

Verify Global DRE Status

WAE674# sh stat dre

Cache:

Status: Usable, Oldest Data (age): 17h

Total usable disk size: 116735 MB, Used: 0.03%

Hash table RAM size: 442 MB, Used:0.00%

Connections: Total (cumulative): 13 Active:0

Encode:

Overall: msg: 11166, in: 67555 KB, out: 5686 KB, ratio:91.58%

DRE: msg: 2264, in: 64697 KB, out: 22962 KB, ratio: 64.51%

DRE Bypass: msg: 12556, in: 2858 KBLZ: msg: 11158, in: 25906 KB, out: 5685 KB, ratio: 78.06%

LZ Bypass: msg: 8, in: 0 B

Avg latency: 0.429 ms Delayed msg: 2706

Encode th-put: 14087 KB/s

Message size distribution:

0-1K=27% 1K-5K=4% 5K-15K=13% 15K-25K=7% 25K-40K=9% >40K=36%

Decode:Overall: msg: 13558, in: 6764 KB, out: 28336 KB, ratio: 76.13%

DRE: msg: 2018, in: 13192 KB, out: 25729 KB, ratio: 48.73%DRE Bypass: msg: 25565, in: 2606 KBLZ: msg: 13514, in: 6288 KB, out: 15419 KB, ratio: 59.22%LZ Bypass: msg: 44, in: 476 KBAvg latency: 0.063 msDecode th-put: 33241 KB/sMessage size distribution:0-1K=17% 1K-5K=45% 5K-15K=10% 15K-25K=5% 25K-40K=8% >40K=12%

------to show statistics connection

WAE674# sh stat conn

and once you see the different connection ID as output of the above commands you can see the details of that connection

WAE674# sh stat conn conn-id 2516

--------------

WAE-SiteA#sh stati

WAE-SiteA#sh statistics dre

Cache:

    Status: Usable, Oldest Data (age): 4m14s

    Total usable disk size:  47527 MB,  Used: 0.00%

      Hash table RAM  size:    189 MB,  Used: 0.00%

Connections:   Total (cumulative): 1   Active: 1

Encode:

   Overall: msg:        426, in:   2120 KB, out:   1558 KB, ratio:  26.47%

       DRE: msg:        426, in:   2120 KB, out:   1843 KB, ratio:  13.04%

DRE Bypass: msg:          0, in:      0 B

        LZ: msg:        374, in:    749 KB, out:    464 KB, ratio:  38.00%

LZ Bypass: msg:         52, in:   1094 KB

    Avg latency:      0.808 ms

  Message size distribution:

    0-1K=69%  1K-5K=8%  5K-15K=7%  15K-25K=6%  25K-40K=5%  >40K=1%

Decode:

   Overall: msg:        397, in: 101158 B, out:   2196 KB, ratio:  95.50%

       DRE: msg:        396, in:    162 KB, out:   2195 KB, ratio:  92.61%

DRE Bypass: msg:          1, in:     89 B

        LZ: msg:        338, in:  67992 B, out:    130 KB, ratio:  48.95%

LZ Bypass: msg:         59, in:  33166 B

    Avg latency:      0.045 ms

  Message size distribution:

    0-1K=65%  1K-5K=9%  5K-15K=8%  15K-25K=4%  25K-40K=11%  >40K=0%

-------------------------------

Following are the details which can be helpful for you to troubleshoot DRE:

For DRE Troubleshooting you can use the following steps:

When application acceleration is expected but not being observed, verify that the appropriate optimizations are being applied to the traffic flow and that the DRE cache is reducing the size of the optimized traffic appropriately.

Policy engine maps for DRE and LZ optimization include the following:

1. DRE + LZ (full): policy-engine application map other optimize full

2.  DRE only: policy-engine application map other optimize DRE yes compression none

3. LZ only: policy-engine application map other optimize DRE no compression LZ

4. TFO pass-through: policy-engine application map other pass-through

Various conditions could cause DRE and/or LZ not to be applied to a connection, even though it is configured:

1. Cache initialization is in progress

2. Disk I/O errors

3. Low memory

4. Data is not compressible or gain is too small

5. Data is encrypted such that it does not contain repeated byte sequences

6. Messages are too small to benefit from compression

Note: In all of the above conditions, the show statistics connection command will report Acceleration of "TDL" for connections where this was the negotiated policy.

Looking at the amount of DRE or LZ bypass traffic will tell you whether DRE or LZ optimizations were actually applied. Use the show statistics connection conn-id command, as described later, and look at the DRE encode numbers to see if the DRE or LZ ratio is near 0% and most of the traffic is bypassed.

The first three conditions will be reported by the "Encode bypass due to" field and the last three conditions result from the traffic data pattern and are accounted for in the reported DRE and LZ ratios.

You can view the statistics for a specific connection to determine what basic optimizations were configured, negotiated with the peer, and applied by using the show statistics connection conn-id command. First you will need to determine the connection ID for a particular connection by using the show statistics connection command, as follows:

WAE#show stat conn

Current Active Optimized Flows:                      1

  Current Active Optimized TCP Plus Flows:           0

  Current Active Optimized TCP Only Flows:           1

  Current Active Optimized TCP Preposition Flows:    0

Current Active Auto-Discovery Flows:                 0

Current Reserved Flows:                              10

Current Active Pass-Through Flows:                   0

Historical Flows:                                    375

D:DRE,L:LZ,T:TCP Optimization RR:Total Reduction Ratio

A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO

ConnID        Source IP:Port          Dest IP:Port            PeerID Accel RR  

   343      10.10.10.10:3300    10.10.100.100:80   00:14:5e:84:24:5f T     00.0%        <------

You will find the connection IDs for each connection listed at the end of the output. To view the statistics for a specific connection, use the show statistics connection conn-id command, as follows:

WAE# sh stat connection conn-id 343

Connection Id:            343

   Peer Id:                  00:14:5e:84:24:5f

   Connection Type:          EXTERNAL CLIENT

   Start Time:               Tue Jul 14 16:00:30 2009

   Source IP Address:        10.10.10.10

   Source Port Number:       3300

   Destination IP Address:   10.10.100.100

   Destination Port Number:  80

   Application Name:         Web                                               <-----Application name

   Classifier Name:          HTTP                                              <-----Classifier name

   Map Name:                 basic

   Directed Mode:            FALSE

   Preposition Flow:         FALSE

   Policy Details:

          Configured:        TCP_OPTIMIZE + DRE + LZ                           <-----Configured policy

             Derived:        TCP_OPTIMIZE + DRE + LZ                                    

                Peer:        TCP_OPTIMIZE + DRE + LZ                                    

          Negotiated:        TCP_OPTIMIZE + DRE + LZ                           <-----Policy negotiated with peer

             Applied:        TCP_OPTIMIZE + DRE + LZ                           <-----Applied policy

. . .

The Application Name and Classifier Name fields tell you the application and classifier applied to this connection.

The optimization policies are listed in the Policy Details section. If the Configured and Applied policies do not match, it means that you configured one policy for this type of connection but a different policy was applied. This could result from the peer being down, misconfigured, or overloaded. Check the peer WAE and its configuration.

The following section of output shows DRE encode/decode-related statistics including the number of messages, how many had DRE applied, LZ applied, or bypassed DRE and LZ:

. . .

DRE: 353

Conn-ID: 353 10.10.10.10:3304 -- 10.10.100.100:139  Peer No:  0 Status: Active

------------------------------------------------------------------------------

Open at 07/14/2009 16:04:30, Still active

Encode:

   Overall: msg:        178, in:  36520 B, out:   8142 B, ratio:  77.71%          <-----Overall compression

       DRE: msg:          1, in:    356 B, out:    379 B, ratio:   0.00%          <-----DRE compression ratio

DRE Bypass: msg:        178, in:  36164 B                                         <-----DRE bypass

        LZ: msg:        178, in:  37869 B, out:   8142 B, ratio:  78.50%          <-----LZ compression ratio

LZ Bypass: msg:          0, in:      0 B                                         <-----LZ bypass

    Avg latency:      0.335 ms     Delayed msg:          0                        <-----Avg latency

  Encode th-put:    598 KB/s                                                      <-----In 4.3.3 and earlier only

  Message size distribution:

    0-1K=0%  1K-5K=0%  5K-15K=0%  15K-25K=0%  25K-40K=0%  >40K=0%                 <-----In 4.3.3 and earlier only

Decode:

   Overall: msg:      14448, in:   5511 KB, out:    420 MB, ratio:  98.72%        <-----Overall compression

       DRE: msg:      14372, in:   5344 KB, out:    419 MB, ratio:  98.76%        <-----DRE compression ratio

DRE Bypass: msg:      14548, in:    882 KB                                        <-----DRE bypass

        LZ: msg:      14369, in:   4891 KB, out:   5691 KB, ratio:  14.07%        <-----LZ compression ratio

LZ Bypass: msg:         79, in:    620 KB                                        <-----LZ bypass

    Avg latency:      4.291 ms                                                    <-----Avg latency

  Decode th-put:   6946 KB/s                                                      <-----In 4.3.3 and earlier only

  Message size distribution:

    0-1K=4%  1K-5K=12%  5K-15K=18%  15K-25K=9%  25K-40K=13%  >40K=40%         <-----Output from here in 4.3.3 and earlier only

. . .

The following statistics are highlighted in the above example for both encoding and decoding:

Overall ratio - the overall compression ratio for the data including both DRE and LZ

DRE ratio - the compression ratio due to DRE alone

DRE Bypass - the number of messages and bytes that bypassed DRE

LZ ratio - the compression ratio due to LZ alone

LZ Bypass - the number of messages and bytes that bypassed LZ

Avg latency - the average latency for the encode or decode operation

If you see a large amount of bypass traffic, the DRE compression ratio will be smaller than expected. It could be due to encrypted traffic, small messages, or otherwise uncompressible data. Consider contacting TAC for further troubleshooting help.

If you see a large amount of LZ Bypass traffic, this could be due to a large amount of encrypted traffic, which is not generally compressible.

The Average latency numbers can be useful for debugging a throughput issue. Both the encode and decode average latency should generally be less than 30 ms. If users experience low throughput and one or both of these numbers are higher, it indicates an issue with encoding or decoding, generally on the side with the higher latency.

It may be useful to look at the DRE statistics data such as the oldest usable data, cache size, percent of cache used, hash table RAM used, and so on by using the show statistics dre detail command, as follows:

WAE# sh stat dre detail

Cache:

    Status: Usable, Oldest Data (age): 10h                        <-----Cache age

    Total usable disk size: 311295 MB,  Used: 0.32%               <-----Percent cache used

      Hash table RAM  size:   1204 MB,  Used: 0.00%               <-----Output from here is in 4.3.3 and earlier only

. . .

If you are not seeing significant DRE compression, it could be because the DRE cache is not populated with enough data. Check if the cache age is short and less than 100 percent of the cache is used, which would indicate this situation. The compression ratio should improve as the cache fills with more data. If 100% of the cache is used and the cache age is short, it indicates that the WAE may be undersized and not able to handle the traffic volume.

If you are not seeing significant DRE compression, look at the Nack/R-tx counters in the following section of command output:

Connection details:

Chunks: encoded 398832,  decoded 269475,  anchor(forced) 43917(9407)       <-----In 4.3.3 and earlier only

Total number of processed messges: 28229                                   <-----In 4.3.3 and earlier only

  num_used_block per msg: 0.053597                                          <-----In 4.3.3 and earlier only

Ack: msg 18088,  size  92509 B                                             <-----In 4.3.3 and earlier only

Encode bypass due to:                                                      <-----Encode bypass reasons

     remote cache initialization: messages: 1,  size:    120 B                   

     last partial chunk: chunks: 482,  size:  97011 B                           

     skipped frame header: messages: 5692,  size:    703 KB

Nacks: total 0                                                             <-----Nacks

R-tx: total 0                                                              <-----Retransmits

Encode LZ latency:      0.133 ms per msg

Decode LZ latency:      0.096 ms per msg

. . .

The Nacks and R-tx counters should generally be low relative to the traffic volume. For example, about 1 per 100 MB of original (unoptimized) traffic. If you see significantly higher counts, it could indicate a DRE cache synchronization problem. Use the clear cache dre command to clear the DRE cache on all devices, or contact TAC.

The encode bypass reasons counters report the number of bytes bypassed due to various reasons. This can help you determine what is causing bypass traffic (other than an unoptimizable data pattern).

It is sometimes helpful to identify the connected and active peer WAEs and look at peer statistics, which you can do with the show statistics peer dre command as follows:

WAE# sh stat peer dre

Current number of connected peers:  1

Current number of active peers:     1

Current number of degrade peers:    0

Maximum number of connected peers:  1

Maximum number of active peers:     1

Maximum number of degraded peers:   0

Active peer details:

Peer-No : 0                    Context: 65027

Peer-ID : 00:14:5e:95:4a:b5

Hostname: wae7.example.com                                            <-----Peer hostname

------------------------------------------------------------------------------

Cache:  Used disk: 544 MB, Age: 14d23h                                <-----Peer cache details in 4.3.3 and earlier only

Cache:  Used disk: 544 MB                                             <-----Peer cache details in 4.4.1 and later only

Peer version:  0.4                                                    <-----

Ack-queue size:   38867 KB                                                 |

Buffer surge control:                                                      |<---In 4.3.3 and earlier only

  Delay:   avg-size          0 B, conn:          0, flush:          0      |

  Agg-ft:  avg-size      20902 B, conn:        388, flush:          0      |

           remote low-buff:  0, received flush:          0            <-----

Connections:   Total (cumulative): 3226861,  Active: 597

Concurrent Connections (Last 2 min): max 593, avg 575

. . .

Other output from this command shows the encode and decode statistics similar to an individual connection.

------------------------------------------------

For TFO Troubleshooting you can use the following troubleshooting steps:

The number of TCP connections, their status, and disposition can give an indication of the health of the WAAS system in a specific location. A healthy system will show a large number of connections, with a significantly large percentage of these closed normally. The show statistics tfo detail command provides an indication of the volume, status, and disposition of connections between a particular WAAS device and other devices in the network.

You can view global TFO statistics by using the show statistics tfo detail command as follows:

WAE# show statistics tfo detail

Total number of connections                          : 2852

No. of active connections                            : 3              <-----Active connections

No. of pending (to be accepted) connections          : 0

No. of bypass connections                            : 711

No. of normal closed conns                           : 2702

No. of reset connections                             : 147

    Socket write failure                              : 0

    Socket read failure                               : 0

    WAN socket close while waiting to write           : 0

    AO socket close while waiting to write            : 2

    WAN socket error close while waiting to read      : 0

    AO socket error close while waiting to read       : 64

    DRE decode failure                                : 0

    DRE encode failure                                : 0

    Connection init failure                           : 0

    WAN socket unexpected close while waiting to read : 32

    Exceeded maximum number of supported connections  : 0

    Buffer allocation or manipulation failed          : 0

    Peer received reset from end host                 : 49

    DRE connection state out of sync                  : 0

    Memory allocation failed for buffer heads         : 0

    Unoptimized packet received on optimized side     : 0

Data buffer usages:

    Used size:          0 B,  B-size:        0 B,  B-num: 0

    Cloned size:        0 B,  B-size:        0 B,  B-num: 0

Buffer Control:

    Encode size:        0 B,  slow:          0,  stop:          0

    Decode size:        0 B,  slow:          0,  stop:          0

Scheduler:

    Queue Size: IO:          0,  Semi-IO:          0, Non-IO:          0

    Total Jobs: IO:    1151608,  Semi-IO:    5511278, Non-IO:    3690931

  Policy Engine Statistics

  -------------------------

  Session timeouts: 0,  Total timeouts: 0

  Last keepalive received 00.5 Secs ago

  Last registration occurred 15:00:17:46.0 Days:Hours:Mins:Secs ago

  Hits:                     7766, Update Released:                   1088

  Active Connections:          3, Completed Connections:             7183

  Drops:                       0

  Rejected Connection Counts Due To: (Total: 0)

     Not Registered      :          0,  Keepalive Timeout   :          0

     No License          :          0,  Load Level          :          0

     Connection Limit    :          0,  Rate Limit          :          0       <-----Connection limit overload

     Minimum TFO         :          0,  Resource Manager    :          0

     Global Config       :          0,  TFO Overload        :          0

     Server-Side         :          0,  DM Deny             :          0

     No DM Accept        :          0

. . .

The No. of active connections field reports the number of connections that are currently being optimized.

In the Policy Engine Statistics section of the output, the Rejected Connection Counts section show various reasons why connections have been rejected. The Connection Limit counter reports the number of times that a connection has been rejected because the maximum number of optimized connections has been exceeded. If you see a high number here, you should look into overload conditions. See the article Troubleshooting Overload Conditions for more information.

Additionally, TFO optimization for connections that are pushed down from other AOs because they cannot optimize the traffic is handled by the generic AO, which is covered in the article Troubleshooting the Generic AO.

You can view TFO connection statistics by using the show statistics connection command.

------------------------------------------

How to Monitor TFO Flows and Overload Conditions

When a WAAS accelerator device is overloaded, you typically see the following Central Manager alarm: Entering overload state due to Max connections (nnn). The number nnn is the number of times the WAE has become overloaded since the last reboot.

The device also logs a syslog error message similar to the following: Sysmon: %WAAS-SYSMON-3-445015: Fault detected: The TFO accelerator is overloaded (connection limit)

You can use various show commands at the CLI to determine the number of allowed and actual connections and gather more information.

Checking the TCP Connection Limit

The first useful command is show tfo detail, which can tell you how many optimized TFO connections that the device can handle, as follows:

wae-7341# show tfo detail

  Policy Engine Config Item            Value

  -------------------------            -----

  State                                Registered

  Default Action                       Use Policy

  Connection Limit                     12000            <-----Maximum number of TFO optimized connections

  Effective Limit                      11988

  Keepalive timeout                    3.0 seconds

The Connection Limit value tells you that this WAAS device can support 12000 TFO optimized connections.

The Effective Limit may be lower than the Connection Limit if the MAPI AO has reserved some connections. The reserved connections are subtracted from the Connection Limit to get the Effective Limit.

For Checking the Optimized TCP Connections use the following steps:


To understand the TCP flows on the device, you can use the show statistics connection command (in version 4.1.1, use the show statistics connection all command). This command displays the currently handled TFO/DRE/LZ flows, pass-through flows, and flows that are being handled by a specific application accelerator. An example of this command follows:

wae# show statistics connection

Current Active Optimized Flows:                      5

   Current Active Optimized TCP Plus Flows:          5

   Current Active Optimized TCP Only Flows:          0

   Current Active Optimized TCP Preposition Flows:   0

Current Active Auto-Discovery Flows:                 0

Current Reserved Flows:                              12             <---------- Added in 4.1.5

Current Active Pass-Through Flows:                   0

Historical Flows:                                    143

D:DRE,L:LZ,T:TCP Optimization,

A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO

ConnID  Source IP:Port        Dest IP:Port          PeerID             Accel

92917   10.86.232.131:41197   70.70.7.11:3268       00:1a:64:69:19:fc  TDL

92918   10.86.232.131:41198   70.70.7.11:3268       00:1a:64:69:19:fc  TDL

92921   10.86.232.131:41216   70.70.7.11:3268       00:1a:64:69:19:fc  TDL

94458   10.86.232.131:45354   70.70.7.11:1026       00:1a:64:69:19:fc  TDL

36883   10.86.232.136:1857    10.86.232.131:1026    00:1a:64:69:19:fc  TDL

From the first line in the output (Current Active Optimized Flows), you can see that the device currently has five active optimized flows. From the second counter (Current Active Optimized TCP Plus Flows), you can see that they are all being handled with TFO/DRE/LZ optimization (TFO Plus means that DRE and/or LZ optimization is being used in addition to TFO). The third counter (Current Active Optimized TCP Only Flows) shows flows that are optimized by TFO only.

Another useful counter is the Current Active Auto-discovery Flows, which displays flows that have not been fully set up to become optimized flows or pass-through flows.

To become fully set up, the connection must see the SYN, SYN ACK, ACK handshake, which is useful to note when dealing with an overload condition.

The Current Active Pass-Through Flows counter shows connections that the device has determined to be pass-through or where the device did not see the SYN, SYN ACK, ACK setup. These flows will not be counted as optimized flows. For pass-through flows, a device should be able to handle up to 10 times the number of optimized flows for which it is rated.

The Current Reserved Flows counter shows the number of connections reserved for the MAPI accelerator. For more details about reserved MAPI connections and their impact on device overload, see the section MAPI Application Accelerator Reserved Connections Impact on Overload.

The sum of the following three counters tells you how close the WAE device is to its connection limit:

  • Current Active Optimized Flows
  • Current Active Auto-Discovery Flows
  • Current Reserved Flows (available only in 4.1.5 and later)
  • If this sum is equal to or greater than the connection limit, the device is in an overload condition.
  • Details about the five optimized flows are displayed in the table below the counters.

Another command that you can use to see the number of TFO flows currently on a device is the show statistics tfo detail command. Two of the most useful counters in the output are "No. of active connections" and under the Policy Engine Statistics the "Active connections", as follows:


wae# show statistics tfo detail

Total number of connections                          : 22915

No. of active connections                            : 3                <-----Current optimized connections

No. of pending (to be accepted) connections          : 0

No. of bypass connections                            : 113

No. of normal closed conns                           : 19124

No. of reset connections                             : 3788

    Socket write failure                              : 2520

    Socket read failure                               : 0

    WAN socket close while waiting to write           : 1

    AO socket close while waiting to write            : 86

    WAN socket error close while waiting to read      : 0

    AO socket error close while waiting to read       : 80

    DRE decode failure                                : 0

    DRE encode failure                                : 0

    Connection init failure                           : 0

    WAN socket unexpected close while waiting to read : 1048

    Exceeded maximum number of supported connections  : 0

    Buffer allocation or manipulation failed          : 0

    Peer received reset from end host                 : 53

    DRE connection state out of sync                  : 0

    Memory allocation failed for buffer heads         : 0

    Unoptimized packet received on optimized side     : 0

Data buffer usages:

    Used size:          0 B,  B-size:        0 B,  B-num: 0

    Cloned size:    54584 B,  B-size:    73472 B,  B-num: 111

Buffer Control:

    Encode size:        0 B,  slow:          0,  stop:          0

    Decode size:        0 B,  slow:          0,  stop:          0

AckQ Control:

    Total:          0, Current:          0

Scheduler:

    Queue Size: IO:          0,  Semi-IO:          0, Non-IO:          0

    Total Jobs: IO:     219110,  Semi-IO:     186629, Non-IO:      49227

  Policy Engine Statistics

  -------------------------

  Session timeouts: 0,  Total timeouts: 0

  Last keepalive received 00.0 Secs ago

  Last registration occurred 8:03:54:38.7 Days:Hours:Mins:Secs ago

  Hits:                    52125, Update Released:                  17945

  Active Connections:          3, Completed Connections:            37257  <-----Active Connections

  Drops:                       0

  Rejected Connection Counts Due To: (Total: 12)

     Not Registered      :         12,  Keepalive Timeout   :          0

     No License          :          0,  Load Level          :          0

     Connection Limit    :          0,  Rate Limit          :          0   <-----Connection Limit

     Minimum TFO         :          0,  Resource Manager    :          0

     Global Config       :          0,  Server-Side         :          0

     DM Deny             :          0,  No DM Accept        :          0

  Auto-Discovery Statistics

  -------------------------

  Total Connections queued for accept:  22907

  Connections queuing failures:         0

  Socket pairs queued for accept:       0

  Socket pairs queuing failures:        0

  AO discovery successful:              0

  AO discovery failure:                 0

In some cases, the two counters will differ and the reason is that the “No. of active connections” displays all the current flows that being optimized by TFO, TFO/DRE, TFO/DRE/LZ, and TFO/DRE/LZ and an application accelerator. The “Active Connections” under the policy engine statistics includes all the flows in the state above plus the connections that are optimized only by TFO and an application accelerator. This situation means that a TCP flow has come in and matched an application accelerator classifier but the SYN, SYN ACK, ACK handshake has not been completed.

In many TFO overload cases, if the problem is still occurring, you can look at these commands and determine if the number of optimized flows is around the rated number of optimized TCP connections for the hardware. If it is, then you can view the flow details and see what is using up all the flows to determine if this traffic is legitimate and overloading the device or is a virus, security scanner, or something else occurring on the network.

The "Connection Limit" counter under the policy engine statistics reports the number of connections rejected and passed through because the WAE has exceeded its rated number of optimized TCP connections. If this counter is high, it means the WAE is frequently getting more connections than it can handle.

If the number of optimized connections is not close to the rated number of optimized TCP connections and you are still getting an overload alarm, then you should look at the Current active auto-discovery flows from the show statistics connection command or the “Active Connections” under Policy

Engine Statistics from the show statistics tfo detail command. In some cases, the number of optimized connections can be very low but the Active Connections under the Policy Engine Statistics are roughly equal to the rated number of optimized flows for the hardware. This situation means that there are many flows that match a classifier but they are not fully established. When a TCP SYN matches a classifier, it will reserve an optimized connection. This connection will not appear in the optimized TCP connections count until the TCP handshake is finished and optimization starts. If the device determines that the flow should not be optimized, it will be removed from the count of active connections under the Policy Engine Statistics.

To further troubleshoot cases where TFO overload is occurring and the Policy Engine Statistics Active Connections seem to be using up all the optimized TCP connections on the device, use the show statistics accelerator detail command. In the output of this command, look at the Active Connections under the Policy Engine Statistics for each application accelerator to determine which application accelerator is receiving these connections that are not fully established.

Next, look at what state these flows might be in by using the show statistics filtering command, which provides you with the number of filtering tuples on the device, as follows:

wae# show statistics filtering

Number of filtering tuples:                             18

Number of filtering tuple collisions:                   0

Packets dropped due to filtering tuple collisions:      0

Number of transparent packets locally delivered:        965106

Number of transparent packets dropped:                  0

Packets dropped due to ttl expiry:                      0

Packets dropped due to bad route:                       10

Syn packets dropped with our own id in the options:     0

Syn-Ack packets dropped with our own id in the options: 0

Internal client syn packets dropped:                    0

Syn packets received and dropped on estab. conn:        0

Syn-Ack packets received and dropped on estab. conn:    0

Syn packets dropped due to peer connection alive:       525

Syn-Ack packets dropped due to peer connection alive:   0

Packets recvd on in progress conn. and not handled:     0

Packets dropped due to peer connection alive:           1614

Packets dropped due to invalid TCP flags:               0

Packets dropped by FB packet input notifier:            0

Packets dropped by FB packet output notifier:           0

Number of errors by FB tuple create notifier:           0

Number of errors by FB tuple delete notifier:           0

Dropped WCCP GRE packets due to invalid WCCP service:   0

Dropped WCCP L2 packets due to invalid WCCP service:    0

Number of deleted tuple refresh events:                 0

Number of times valid tuples found on refresh list:     0

The number of filtering tuples is the number of flows on the device that are optimized, in pass-through, in FIN WAIT state, in setup state, and so on. Each established flow appears as two tuples, one for each side of the flow, so the number that you see in this output may be much larger than the number of flows that you are seeing in the other commands.

To get more information on the flows in the filtering list, you can use the show filtering list command as follows:

wae# show filtering list

E: Established, S: Syn, A: Ack, F: Fin, R: Reset

s: sent, r: received, O: Options, P: Passthrough

B: Bypass, L: Last Ack, W: Time Wait, D: Done

T: Timedout, C: Closed

   Local-IP:Port        Remote-IP:Port          Tuple(Mate)        State

  10.86.232.82:23      10.86.232.134:41784 0xbc1ae980(0x0       )   E

10.86.232.131:58775      70.70.7.11:3268  0x570b2900(0x570b2b80)   EW

    70.70.7.11:3268    10.86.232.131:58775 0x570b2b80(0x570b2900)   EDL

    70.70.7.11:3268    10.86.232.131:57920 0x570b2d80(0x570b2800)   E

10.86.232.131:57920      70.70.7.11:3268  0x570b2800(0x570b2d80)   E

  10.86.232.82:23      161.44.67.102:4752  0xbc1aee00(0x0       )   E

10.86.232.131:58787      70.70.7.11:1026  0x570b2080(0x570b2e80)   EW

    70.70.7.11:1026    10.86.232.131:58787 0x570b2e80(0x570b2080)   EDL

10.86.232.131:48698      70.70.7.11:1026  0x570b2f00(0x570b2880)   PE

10.86.232.131:58774      70.70.7.11:389   0x570b2300(0x570b2180)   EW

    70.70.7.11:389     10.86.232.131:58774 0x570b2180(0x570b2300)   EDL

10.86.232.131:58728      70.70.7.11:1026  0x570b2380(0x570b2a00)   E

10.86.232.131:58784      70.70.7.11:1026  0x570b2e00(0x570b2980)   EW

    70.70.7.11:1026    10.86.232.131:58784 0x570b2980(0x570b2e00)   EDL

    70.70.7.11:1026    10.86.232.131:48698 0x570b2880(0x570b2f00)   PE

10.86.232.131:58790      70.70.7.11:3268  0x570b2100(0x570b2c80)   EW

    70.70.7.11:3268    10.86.232.131:58790 0x570b2c80(0x570b2100)   EDL

If the show statistics accelerator all command shows you which application accelerator is using up all the optimized TFO connections, you can filter on that port or traffic. For example, if you want to filter on port 80 traffic, use the show filtering list | I :80 command.

Look at the legend in the State column. In the case where the flows are in the SYN state, you may see a lot of flows with a state of S. If the WAE has sent back the SYN ACK with options set you may see the state SAsO. This indication may help you determine the state of the flow and from there, you can determine if there is a routing problem, virus, or a problem with the WAE not releasing connections. You may need traces to determine exactly what is happening to the flows but the commands above should give you an idea of what to look for.

MAPI Application Accelerator Reserved Connections Impact on Overload

Often, a TFO overload can be caused by the MAPI application accelerator reserved connections, so it is helpful to understand the process of how the MAPI application accelerator reserves connections.

The MAPI application accelerator reserves TFO connections to ensure that it will have enough connections available to it to accelerate all current and future connections that the clients will make to the Exchange servers. It is normal for a MAPI client to make multiple connections. If a client makes the initial connection through the MAPI application accelerator, but the subsequent connections fail in the MAPI application accelerator, there is a risk that the client's connection might fail.

In order to avoid these potential connection failures, the MAPI application accelerator reserves connection resources as follows:

Before any client connections begin, it reserves 10 connections for itself, as a buffer for anticipated new connections.

For each client connection to the server, it reserves three TFO connections for that client-server pair and one of the three is used as an active connection for this first connection. If the same client makes a second or third connection to the same server, those are handled out of the reserved connection pool. If a client only ever makes a single connection to the server, those two reserved connections will be unused and remain in the reserved pool. If the client makes a connection to a different server, three new connections are again reserved for that client-server pair.

All of these reserved connections are designed to improve performance and to reduce the possibility of a client connection failing because of the inability to make additional connections through the MAPI application accelerator.

Overload occurs when Current Active Optimized Flows + Current Active Auto-Discovery Flows + Current Reserved Flows is greater than the device's fixed Connection Limit.

In general, new connections would then be passed through. But some new MAPI connections may still be optimized. When the device is at the overload point, if a client makes an additional request to a MAPI server it already has connected to, then reserved connections are used. But if there are not enough reserved connections (for example, if a client makes a fourth connection to the same MAPI server and the WAE is already in overload) then an escaped connection condition might occur, and this could lead to erroneous behavior such as a client receiving many duplicate copies of the same single mail message.

If the system did not forward the connection to the MAPI application accelerator, you should see "PT Rjct Resources" or "PT in progress", depending on whether there is activity on the connection. If the connection was forwarded to the MAPI application accelerator and then the reservation failed, the connection will be marked with a "G" for the Accelerator, instead of an "M" (in the show statistics connection optimized mapi command output). For an example of this command, see the article Troubleshooting the MAPI AO.

If you are experiencing frequent overload conditions, it is important to understand how the Outlook clients are making connections (how many connections to how many Exchange servers). With Outlook running on a client, hold the Ctrl key while you right-click on the Outlook icon in the system tray on the task bar. Choose Connection Status to display the list of servers to which the Outlook client has connected. From that you can see how many connections the client is making and to how many different Exchange servers. If the client is making connections to several different servers, it would be helpful to investigate ways to consolidate mail so a user only opens MAPI connections to a single Exchange server, and makes use of multiple connections to that server.

It is also useful to investigate whether there are any other applications that might be making MAPI connections.

Solutions for Overload Conditions

Examine optimized connections to see if they are legitimate. In many cases, a Denial of Service (DoS) attack encountered in the network may be causing the WAE to attempt to optimize connections. If so, employ a DoS protection mechanism in the network to proactively close the connections.

In cases where the connections are legitimate, the WAE deployed in the location is undersized and may need to be upgraded, or an additional WAE can be deployed to increase scalability within that site.

..................................................................

-------------------------------------------------------------

Use the following URL for for monitoring :

http://www.cisco.com/en/US/docs/app_ntwk_services/waas/waas/v401/configuration/guide/monitor.html

-----------------------------------------

Please rate if it help you any.

HTH

Sachin

I don't have a problem with the CIFS acceleration or any other application acceleration service. Can you tell me how to view the eviction for DRE cache if it exists, not just for CIFS AO. I want to compare the oldest data age from the show stat dre cache command to the last eviction. How can I determine how many times a day is the purging happening?????? I need to understand if we need to add additional WAE-7371s to the core environment if it can't keep up with the caching.

show stat dre

Cache:

    Status: Usable, Oldest Data (age): 6d22h

    Total usable disk size: 972799 MB,  Used: 99.75%

      Hash table RAM  size:   3588 MB,  Used: 93.00%

Connections:   Total (cumulative): 887721158   Active: 5115

Encode:

   Overall: msg: 15713080635, in: 30026812 MB, out: 8980472 MB, ratio:  70.09%

       DRE: msg: 14542620806, in: 29719427 MB, out: 13830638 MB, ratio:  53.46%

DRE Bypass: msg: 2153943270, in: 307247 MB

        LZ: msg: 13387089320, in: 8289595 MB, out: 3116934 MB, ratio:  62.40%

LZ Bypass: msg: 2325991315, in: 5848290 MB

    Avg latency:      0.300 ms     Delayed msg: 6522988262

  Encode th-put:   6517 KB/s

  Message size distribution:

    0-1K=79%  1K-5K=12%  5K-15K=4%  15K-25K=2%  25K-40K=1%  >40K=0%

Decode:

   Overall: msg: 15546149095, in: 5384756 MB, out: 15545432 MB, ratio:  65.36%

       DRE: msg: 13905506768, in: 8660775 MB, out: 14644398 MB, ratio:  40.86%

DRE Bypass: msg: 3639425455, in: 901057 MB

        LZ: msg: 13602881560, in: 3116432 MB, out: 7311753 MB, ratio:  57.38%

LZ Bypass: msg: 1943267535, in: 2268323 MB

    Avg latency:      0.188 ms

  Decode th-put:   5445 KB/s

  Message size distribution:

Hi John,

Kindly send the output of the following two commands as well:

WAE# show statistics dre connection

Conn  Peer       Client-ip:port       Server-ip:port   Encode-in/   Status

Id    No                                              Decode-in    (A-Active)

                                                                    (C-Closed)

99674   1    10.10.13.100:1559    10.10.10.100:445     737B/ 1KB    C(6m46s)

99673   1    10.10.13.100:1560    10.10.10.100:139       0B/ 0B     C(6m57s)

99645   1    10.10.13.100:1558    10.10.10.100:80     487MB/ 0B     C(7m49s)

WAE# show statistics dre connection id 99645

Conn-ID: 99645 10.10.13.100:1558 -- 10.10.10.100:80  Peer No:  1 Status: Closed

------------------------------------------------------------------------------

Open at 10/20/2007 17:37:52, Close at 10/20/2007 17:38:32, Duration: 40 secs

Encode:

   Overall: msg:      15427, in:    487 MB, out:   5416 KB, ratio:  98.92%

       DRE: msg:      15427, in:    487 MB, out:   5691 KB, ratio:  98.86%

DRE Bypass: msg:          0, in:      0 B

        LZ: msg:       1562, in:    932 KB, out:    657 KB, ratio:  29.49%

LZ Bypass: msg:      13865, in:   4759 KB

    Avg latency:      1.405 ms

  Message size distribution:

    0-1K=1%  1K-5K=4%  5K-15K=13%  15K-25K=16%  25K-40K=30%  >40K=33%

Decode:

   Overall: msg:          0, in:      0 B, out:      0 B, ratio:   0.00%

       DRE: msg:          0, in:      0 B, out:      0 B, ratio:   0.00%

DRE Bypass: msg:          0, in:      0 B

        LZ: msg:          0, in:      0 B, out:      0 B, ratio:   0.00%

LZ Bypass: msg:          0, in:      0 B

    Avg latency:      0.000 ms

  Message size distribution:

    0-1K=0%  1K-5K=0%  5K-15K=0%  15K-25K=0%  25K-40K=0%  >40K=0%

HTH

sachin

I'm running version 4.2.1. I don't have those options.

ryewaas1#show statistics dre ?        

  detail  Display detailed DRE statistics

  |       Output Modifiers

     

kindly attach the output of the following commands

#show stat dre detail

and

#show tech

for suggesting more.

Sachin

I've logged a case with TAC - SR 618217149. You can view the show tech log from there and the notes. I'm looking for a way to see how frequent the purging is occurring on the DRE cache, without logging into the CLI all the time and do a show stat dre.

Thanks for the help.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: