WAAS and CIFS issues

[corpitsol]

Hello,

I've been trying to nail down issues with remote sites and mapped drives for a week or so now. Posting here to see if anyone can correlate these issues and/or provide me with additional methods of troubleshooting.

Several (Windows 7) users on a WAAS'd network (Wave274->Wave574) all running the latest revision of firmware/images: Cisco Wide Area Application Services Software Release 4.1.7 (build b11 May 14 2010) Version: oe574-4.1.7.11

Issue:

The problem I am having is users are experiencing delays accessing mapped drives. Files opening are delayed sporadically and the actual browsing of the drives as well. I've done the following on the remote server which is running a fully patched SBS 2008 instance:

Server Troubleshooting:

-Turned off AV software and removed firewall drivers from stack.

-Disabled IPV6 interface.

-Turned off Base Filtering Engine and Windows Firewall.

-Disable/Remove any TCP offloading or software acceleration options on the NIC (It's vmware anyways).

The server produces dozens of "SRV" events (ID#2012) which indicate a connection terminating prematurely on the remote side.

WAAS Troubleshooting:

On the WAAS side, I have combed every log file looking for clues as to why this is happening.

-Connections are being optimized by the CIFS AO (TCDL) and not pushed down to some other generic optimization engine. (OK)
-CIFS Digital Signing in Windows 7 is not causing issues or the above problem. (OK)
-Machine I’m dealing with current CIFS optimized session has no retransmissions or collisions (data lost inbetween) (OK)

-The total latency for NT_CREATE_ANDX (per CIFS command) latency via “show statistics cifs requests”.  Also the response time for all commands and finally the average remote command which was >200ms.

Statistics gathering period:  hours: 362 minutes: 36 seconds: 4 ms: 495
Total: 3796803
Remote: 113832
ALL_COMMANDS total: 3796803 remote: 113832 async: 3143835 avg local: 7.36ms avg
remote: 201.39ms

NT_CREATE_ANDX total: 12923 remote: 11737 async: 0 avg local: 1.452ms avg remote
: 32.907ms

Network Troubleshooting:

-Verified network connectivity with the server and general application availability.

-150 other servers running on this cluster most are built from same templates using same AV, several have WAAS and not reported issues.

Anyone have similar issues?? Ideas for me to gather more data pointing me away from WAAS?

Thanks,

Matthew Chambers

[Zach Seils]

Hi Matthew,

Can you please verify that there aren't any duplex mismatches anywhere in the path between the client and server?

How frequently does this problem occur?  If possible, we would like to collect simultaneous packet captures on the WAAS devices on both sides of the link.  Those, along with system reports from both WAAS devices, are what we would want to start going through to check for errors.

Regards,

Zach

[corpitsol]

Zach,

We ruled out the duplex mismatch issue. After trying to reproduce the problem on this network, we found that the problem may be related to the CIFS connections being pushed down after all.

From CIFS_ERR.LOG I can find numerous push downs via AoHandoffTask and other generic optimization only tasks.

I was under the impression that the newer versions of WAAS firmware delt with this.

Do we need to disable SMB signing?

[Zach Seils]

Can you provide a copy of the cifsao-errorlog.0 file from the /local1/errorlog directory?

Thanks,

Zach

[corpitsol]

Attached. All of the traffic from this site is Client (MS XP SP3)->Server (SBS 2008).

Just as a heads up, so there is no Server->Server traffic between these sites. I thought it was worth noting because I've seen these errors with SYSVOL replication between Domain Controllers because of the default security and SMB signing etc..

*Another important note to add: All of the workstations are mapped to the remote server using HTTP:// drives and not \\. The optimization for these connections opens several CIFS sessions, but they all have small sizes (178 Bytes/127 Bytes) and never seem to transmit anything. I'm not sure if this is related to the generic optimization messages, just a heads up. The WAFSBenchmark showed very good results using this method of access.

[Zach Seils]

What is the IP address of the server in question?

Zach

[corpitsol]

172.16.9.15 inside.

[Zach Seils]

The log file indicates that this server requires digital signatures:

10/07/2009 16:59:59.878(Local)(1332) NTCE (878383) Connection to 172.16.9.15 will be handled by generic optimization only, since 172.16.9.15 requires digital signing. [AoShellLog.cpp:22

In order for the CIFS AO to accelerate these connections, digital signatures should be optional.  More information on changing the server configuration is available here:

Overview of Server Message Block Signing

Regards,

Zach

[Patrick Moubarak]

Hi Zach,

does version 4.2(1) fix this? meaning will the CIFS AO apply to SMB signed packets in this version or will the connections still failover to the generic AO?

Thanks,

Patrick Moubarak