WAAS 4.4.3 breaks Sharepoint file downloads

[Tod Larson]

We have a sharepoint 2010 on our internal network at HQ.  HQ and each branch have WAAS 4.4.3.

When a WAN user tries to open a MS office file (e.g. open with excel) off the sharepoint server from a branch it takes 5-10 minutes and sometimes just errors out/failing to open at all.

What a WAN user tries to open an MS office file off the sharepoint server within a web browser it is nearly instant.

Also, many sharepoint pages are very slow to load.

Packet captures reveal lots of 401 unauthorized errors when applications are not performing well.

Also, an Aqualogic portal on our private network as very slow page load times.

--The slowness has been going on for months and seems to be getting worse.  It "might" correlate to when we updated to 4.4.3 but I can't be sure.

Also, a custom ColdFusion line of business web app behaves erratically when accessed from a branch office.  Often Ampersands are left off the http calls. (this started sometime in the past month so I'm not sure if it's the same problem).

However,

If I use a wccp redirect access to deny passing any www (port 80) traffic to WAAS ALL the web applications work great!

It's as if WAAS is breaking sharepoint and these other web apps by screwing up their authentication packets or something.

Any ideas?

[Mohit Chauhan]

Hi Tod,

Do you have SSL configured on the WAE? I guess your sharepoint app would be running on the https instead of http. and if thats the case, my understanding is that you need to get SSL configured and active. I may be wrong, but just a suggestion.

Regards,

Mohit

[Tod Larson]

How do I verify if I have SSL configured? Thank you.

[Mohit Chauhan]

Thru CM GUI, you could do:

Configure > Acceleration > SSL Accelerated Services

and if it is configured, you should see it enabled and also there should be details like server ip, server port number, existing certificate and so on.

Configuring SSL requires some config to be added unlike other services.

Hope that helps - All the best!!

[Tod Larson]

It turns out the web pages are not encrypted, we are hitting are http only. Also, do NOT have any SSL Accelerated Services Applied.

[cfolkerts]

Did you try disabling HTTP acceleration?  What type of device is doing the redirection?  Or are you using inline?

[Tod Larson]

I have 2 ways to workaround the problem.  The problem is that now NONE of our internal http traffic is accelerated thus defeating the purpose of WAAS in the first place, right?

1. Disable the "Web" application for each WAE on Central Manager; Configure -> Acceleration -> Applications -> Web -> Assign Devices.

or

2. Use a redirect list on the route to deny any www traffic

     ip wccp 61 redirect-list waas-wae

     ip wccp 62 redirect-list waas-wae

     extended ip access list waas wae

          deny tcp any any eq www

          permit ip any any

We are using wccp on the WAN router to perform redirection.

We have no inline devices.  We have 674 Appliances, WAE network modules and WAE Service modules.

Thank you for any insight.

[Felix Arrieta]

hi Tod,

if you could reproduce the problem and at the same time start collecting the following logs I hope we can detect any issue. ( these are big log files, so you can save the session and attach it over here if you prefer)

WAE#sh log

WAE#show run | begin HTTP

WAE#show alarms history

WAE#show alarms

WAE# cd errorlog

WAE#type-tail httpao-errorlog.current 300

please also review the HTTP/WAAS troubleshooting page:

http://docwiki.cisco.com/wiki/Cisco_WAAS_Troubleshooting_Guide_for_Release_4.1.3_and_Later_--_Troubleshooting_the_HTTP_AO

see you!

[Tod Larson]

Felix,

The rub is that I my management does not permit me to move data from the standalong network to the internet so I have to re-type everything (a pain I know).  So he's what I have.  In re-typing I have skipped stuff that doesn't seem relevant.

In my testing though I have noticed that the slowness occurs only on the first download when HTTP accelation is enabled.  Downloads work great if accelation is disabled OR if you attempt the download again while the TCP session is still established (this makes me think I should just disable acceleration and move on).  However the web server is >80ms away from the clients so I have to think WAAS will help so I want to make it work.

Also, it's as if the last character gets chopped off from some URLs.  This only happens with winXP, not on Win7

http://webserver/app/comm/main.cfm?ID=84&  <-- this is what is supposed to be presented to the end user

http://webserver/app/comm/main.cfm?ID=84  <-- this is what gets presented to the end user (which doesn't work). The user manually types in the "&" to make it work.

WAE#sh log

syslog to host is enable.

syslog to disk is enabled

priority for disk logging is set to : detail

filename for disk loggin is set to: /local1/syslog.txt

syslog facility is set to *

WAE#show run | begin HTTP

classifier HTTP

match dst port eq 80

match dst port eq 8080

match dst port eq 8000

match dst port eq 8001

match dst port eq 3128

exit

WAE#show alarms history

No alarm history is available.

WAE#show alarms

Critical Alarms

none

Major Alarms

none

Minor Alarms

none

WAE# cd errorlog

WAE#type-tail httpao-errorlog.current 300

*** for this there are hundreds of entires over the 20-30 seconds while I was recreating the issue.  In looking through the logs I don't see anything glaringly obvious.

[Felix Arrieta]

Hi Tod,

From what I've been reading  this kind of behavior is because how the  sharepoint server is configure, for example insisting that local  clients authenticate over Kerberos where remote clients authenticate  over NTLM.

I also found the following link:

http://weblogs.asp.net/felipeferreira/archive/2008/03/11/fixing-sharepoint-401-1-http-errors.aspx

I am still trying to learn on this but at the end captures never lie, so we could take captures at the moment of failure ( with WAAS) and captures when is actually working(without WAAS) gathering this captures simultaniously on both sides of the network is critical, otherwise they're useless.

1- Captures from the Data Center WAE to the share point

2- Capture from the Remote WAE to the  Remote testing PC

3-capture from the DC WAE to the Branch WAE

the main idea is to have a valid packet capture from the Sharepoint to show us whether the packets sent from the DC WAE actually reached the Sharepoint, or if the Sharepoint received it and sent packets (ACK) that  the DC WAE never received...and so it goes the troubleshooting  to the branch office as well.

Maybe the switch/router at the WAE is dropping packets?

Maybe there's a device in between that does not like the packet size??

Maybe it is something else than WAAS?

best of luck!

[cfolkerts]

Is the router performing the redirection an ASR by chance?

[Tod Larson]

No ASR routers on the network.  The wccp routers are a combination 7200, ISR and ISR-G2.

[Balakrishnan Rajvel]

Hi ,

Can you check whether below mentioned HTTP acceleration features are enabled , if so then disable it and check HTTP application.

- Metadata cache for HTTP

- Suppress server compression on HTTP

- DRE hint for HTTP

[Tod Larson]

If I disable HTTP Acceleration then the problem goes away.

If I enable HTTP Acceleration then the problem re-occurs every time.

If I do the following, then the problem re-occurs every time

- enable HTTP Acceleration while doing the following

- Uncheck Enable HTTP metadatacache caching

- Uncheck Enable HTTPS metadatacache caching

- Uncheck Enable local HTTP 301 redirect messages

- Uncheck Enable local HTTP 401 Authentication-required messages

- Uncheck Enable local HTTP 304 Not-Modified messages

- Uncheck Suppress server compression for HTTP and HTTPS

- Uncheck Enable DRE Hints for HTTP and HTTPS