Ask the Expert:Understanding WAAS Platform

[ciscomoderator]

With Michael Schueler

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn about resolving configuration problems on the WAAS platform that are not directly related to the WAAS software.

Peter Van Eynde is a customer support engineer in the Technical Assistance Center in Belgium, where he supports content technologies including Cisco Wide Area Application Services (WAAS), Cisco Application and Content Networking System (ACNS), and Cisco Content Delivery System Internet Streaming (CDS-IS). He holds CCIE certification #23042 in Security.

Michael Schueler has been a Cisco support engineer in the Cisco Technical Support Assistance team in Germany for more than 4 years. He and is an expert on content technologies including Cisco Wide Area Application Services (WAAS), Cisco Application and Content Networking System (ACNS), Cisco Content Delivery System Internet Streaming (CDS-IS), and Cisco Digital Media Suite (DMS). He holds CCIE certification #23835 in Security.

Remember to use the rating system to let Peter and Michael know if you have received an adequate response.  

They might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the discussion forum shortly after the event. This event lasts through November 18, 2011. Visit this forum often to view responses to your questions and the questions of other community members.

There will also be a live webcast on this subject on Tuesday, November 15, at 2 p.m. Pacific Time and 5 p.m. Eastern Time. During the live event, Cisco subject matter expert Bhavin Yadav will be presenting on how to troubleshoot WAAS Application Optimizations (AO) effectively. He will focus more on individual AO troubleshooting and he will explain about the different ways to work around AO issues with minimum impact to the traffic.

To register for this live webcast: visit, http://www.ciscolive.com/ate/index.php?sid=203284_2

[Michael Schueler]

Hi Robert,

I have never heard about ALLOT. Can you give some more details on what this exactly is and what it does? Does it touch traffic at layer 4 or above?

If it only works at layer 3 or below, WAAS should work fine across it.

If it's a firewall or something similar, you might want to consider using Directed Mode, see here:

http://www.cisco.com/en/US/docs/app_ntwk_services/waas/waas/v441/configuration/guide/network.html#wp1059477

Best regards,

Michael

[robertsoto]

Michael,

About ALLOT  can work in layer 4 or layer 7  , the ALLOT is working in priorization of traffic and inspect traffic because It has a Signature for application.

Regards,

Robert soto.

[Michael Schueler]

Hello Robert,

If ALLOT is configured to work in layer 4 or layer 7 you need to consider the following:

1. WAAS uses TCP option 33 (0x21) in all TCP packets with the SYN flag set (to negotiate optimization parameters with the peer WAAS) on the WAN side.
2. If optimization was successfully negotiated for a given connection, WAAS will increment the TCP SEQ and ACK numbers by 2G in all packets on the WAN side, that do not have the SYN flag set.

So you would need to configure ALLOT as follows:

1. Allow TCP option 33 (0x21) to be used.
2. Ignore the increment in TCP SEQ and ACK numbers (this is often called normalization).
3. Not apply any TCP sequence number randomization.

If any of these is not configurable with ALLOT, you would need to use the Directed Mode I mentioned yesterday. When using Directed Mode WAAS encapsulates all traffic on the WAN side in UDP. In this case you would only need to allow UDP traffic on port 4050 between the two WAAS peers to pass through ALLOT.

The only drawback of using Directed Mode is, that on the WAN side all traffic appears as UDP traffic on port 4050 with source and destination IP addresses beeing the two WAAS peers themselves. You do not see the real IP addresses of your clients and servers on the WAN side anymore.

This all only applies, if you have ALLOT on the WAAS's WAN side, as it appears to be in your "First" scenario. If ALLOT is inserted on the WAAS's LAN side, as it appears to be in your "Second" scenario, all should continue to work fine without any configuration changes on ALLOT even after introducing WAAS - on the LAN side WAAS does not apply any changes to the traffic.

Best regards,

Michael

[Stoyan Stoitsev]

Hello,

It woud be good to see a clearly written document on calculation the maximum MSS size for WAAS with GRE and IPSec tunnels. This is a very common scenario for enterprise customers and documentation on this is somehow inconsistent. Additionally some "best pracitce" needs to be defined on whether to configure MSS on the Tunnel interface, on the LAN interface and how exactly to calculate - based on L2 or L3; what should be the MTU and MSS on the router and what on the WAAS box.

It is a very very common issue when WAAS "breaks" SSL aplications due to packet fragmentation which was not accuring before deploying it.

Regards,

Stoyan

[pevaneyn]

Hello Stoyan,

The interaction of WAAS with MTU/MSS is a bit complex. Due to the nature of WCCP redirection WAAS cannot use Path MTU discovery to determine the maximum MTU on a path. (the ICMP messages are not redirected, so the WAAS cannot know the packet it just send was too big) Nor does it work for inline cards, as the card will only send TCP trafic to the WAAS system, again not the ICMP messages.

Not adjusting the MSS properly is bad because it will cause fragmentation, leading to higher CPU load on the router and the WAAS device.

What the WAAS does is to modify the MSS on every SYN packet it sees to a configured value. By default it will set the MSS on both LAN and WAN side to 1432 bytes. See:

http://www.cisco.com/en/US/docs/app_ntwk_services/waas/waas/v441/configuration/guide/policy.html#wp1042887

If you want you can modify these settings with "tfo tcp optimized-mss" or tfo tcp original-mss", see:

http://www.cisco.com/en/US/docs/app_ntwk_services/waas/waas/v441/command/reference/glob_cfg.html#wp1050385

The overhead due to redirection is variable and depends on the redirection, return and egress method (you need to consider all three):

- layer 2, ip forwarding or inline: no overhead only the MAC address is modified

- WCCP GRE:  IP (20) + TCP (20) + GRE with WCCP option (28) = 68 bytes so 1432 remain from a standard 1500 bytes MSS.

- Generic GRE: IP (20) + TCP (20) + GRE (24) = 64 bytes 

So you need to measure the maximum MSS you can use and then substract these numbers from that to get the MSS you need to configure.

There should be no need to configure MSS on the tunnel interface on the router, nor on the ethernet ports of the WAAS device itself.

The breaking of SSL appliactions due to MSS problems should be a thing of the past. Are you seeing this with a recent (4.3.x or 4.4.x) release?

Best regards, Peter

[tasoskypraios]

Hi,

We have install in our network 12 WAAS devices all with WCCP Configuration. Now we are looking to install 3 more but we prefer to install them inline. If the Core WAE has WCCP configuration and the WAE in braches are inline this will not be a problem am i correct?

Thank you

[Michael Schueler]

Hello,

Correct, using WCCP at one site and inline mode at another site is fully supported.

The only restriction is, that you cannot use inline mode and WCCP simultaneously on the same WAE:

---snip---

Inline mode and WCCP redirection are exclusive. You cannot configure inline mode if the WAE is configured for WCCP operation.

----snip---

See here:

http://www.cisco.com/en/US/docs/app_ntwk_services/waas/waas/v441/configuration/guide/traffic.html#wp1062082

Best regards,

Michael

[tasoskypraios]

Thanks for prompt answer. Something i forgot to mention, between Core WAE and Branch WAE there is a FWSM. The only thing i have to do is to enable inspection in FWSM for WAAS, is this also correct?

Thank you

[pevaneyn]

Hello,

Correct you only have to enable the 'waas' inspection for the FWSM to handle the WAAS TCP options and sequence number shift.

However also please remember that any Layer 4 and above inspections will not work on the optimized traffic. So no HTTP or SMTP inspection  for example on the FWSM for optimized traffic.

Best regards, Peter

[Joe Lau]

Hi Peter,

Just read through your reply on the WAAS interacts with MTU/MSS. It's very detail indeed. Anyway, I've checked the WAAS topology on my network and I got a question in one of the scenarios:

- a WAE-574 connects to a FastEthernet port of a Cisco 3845 router

- the router has a primary MPLS circuit connecting to the enterprise MPLS network (MTU 1500)

- the router has another IPSec tunnel connecting back to the regional hub, through an ADSL internet line (tunnel MTU 1400)

- "ip path-mtu-discovery enable" is disabled on the WAE-574

The WAE should be running without any problem on CM registration or optimizing TCP connections with other WAEs on the network. However, would this WAE still be running fine when the site MPLS circuit is down, in which all traffic run on the IPSec tunnel?

Should we

(1) enable the "ip path-mtu-discovery enable" command on the WAE, or

(2) add the mtu 1400 command to the GigabitEthernet 1/0?

Which way should be better?

Thanks & Regards,

Joe

[pevaneyn]

Hi Joe,

The problem with that setup is indeed when the MPLS link fails. The default setting of: 1432 bytes MSS (so ~1472 MTU) would result in the WAAS sending of bypassing a packet which is too big for the IPSec tunnel.

The maxium MSS on the IPSec tunnel is 1400-40 -> 1360.

So you could do two things:

1 On the WAAS configure:

tfo tcp optimized-mss 1360

2 on the router configure on the IPSec tunnel interface:

(config-if)#ip tcp adjust-mss 1360

   which will also modify the WAAS packets in addition to any other packets.

  If you are using a method which will cause the traffic to move from the MPLS to the IPSec network (which is likely) don't forget to also configure this on the MPLS interface.

Option 2 has the advantage of also working for normal traffic.

Best regards, Peter

[Patrick Moubarak]

Hi experts,

I have a question regarding WAAS WCCP load balancing and I'm hoping you could help me figure this out.

I have 1 branch router (Cisco 2900 series IOS 15.2(1)T) and 2 WAVE-274 appliances (version 4.3.3).

The router's LAN interface is let say IP: 10.1.1.1/22

The WAAS are: 10.1.1.7 and 10.1.1.9/22 with the router as default gateway

and the users are DHCP in 10.1.2.1 to 10.1.3.254 /22 with the router as default gateway

so all devices are in the same subnet, same VLAN...

I saw the following when looking into configuring WCCP:

http://www.cisco.com/en/US/prod/collateral/contnetw/ps5680/ps6870/white_paper_c11-608042.html

1- 2900 series supports both L2 and GRE for redirect method

2- 2900 series supports both HASH and MASK for load balancing method

I already posted this but got no replies:

https://supportforums.cisco.com/message/3479690#3479690

Here's my config on the router:

ip cef

ip wccp 61

ip wccp 62

interface GigabitEthernet0/0

description *** LAN Connection ***

ip wccp 61 redirect in

int gi 0/1

description *** WAN ***

ip wccp 62 redirect in

Config on WAAS:

I am currently using L2 redirect since router and WAVE are in the same L2 subnet. WCCP works so far.

Then comes load balancing: I tried HASH first and all traffic went to one WAVE not both although the assignment said 50%; 50% on the router. I left it for days and all traffic still sent to one WAVE.

So I switched to MASK with the new default mask on WAAS:

0xf00 source ip and 0 destination IP (so load balancing is acheived by source IP) and at first it seemed that traffic was now being sent by the router to both WAVE devices.

Then I noticed that one WAVE is up to 2 million packets while the other is still at 1 thousand which he received from day 1. I saw this by doing show ip wccp 61 detail and show ip wccp 62 detail on the router.

WCCP is functional on the router and both WAVEs with no flapping and no errors but load balancing is NOT!!!

I also use the default Egress Method of IP forwarding. Is this the problem with my config? I know that L2 redirect and Generic GRE egress method are not compatible. Redirect method and return method are both WCCP L2.

Then I just recently saw this:

Use WCCP GRE or generic GRE as the egress method if you  want to place WAEs on  the same VLAN or subnet as clients and servers.  This topology is not allowed  when using the IP forwarding egress  method.

Is this the problem? Do I have to switch from WCCP L2 redirect and return to WCCP GRE instead?

If yes, can I do WCCP GRE although the router and WAAS are not seperated by Layer 3 (the are both in the same subnet)? Do I need a GRE tunnel interface on the router?

What is Cisco's best practice on this type of deployment for a remote office?

I was considering inline cluster if all else fails but I honestly don't like that solution too much :

- there is no load balancing unless 1 WAVE is TFO overloaded

- there is downtime if one WAVE reboots (slight disconnect of the inline interfaces during reboot, some pings lost in the process)...

Thanks,

Patrick Moubarak

[pevaneyn]

Hello Patrick,

If I understand correctly your WAAS devices are on the LAN interface. This means that you *need* to use WCCP GRE as redirection/return method.

Otherwise if you use L2 or ip forwarding the router will not be able to tell the returned traffic appart from the original traffic and this could create a loop.

So please:

- shut down WCCP on the router

- change both WAAS devices to use HASH/WCCP GRE

- enable WCCP on the router again

After some time both devices should be accepting and accelerating traffic.

If not please send the output of:

show wccp routers

show wccp wide-area-engine

show wccp gre

show egress

from both WAAS devices and:

show ip wccp interfaces

show ip wccp 61 detail

show ip wccp 62 detail

show ip wccp 61 internal

show ip wccp 62 internal

show ip wccp

from the router.

Regarding your questions:

- WCCP GRE will work when the router is L2 adjecent to the WAE

- there is no need to configure a GRE tunnel on the router (you do need to do this when using Generic GRE)

- WCCP load balancing is not dynamic. If one WAE is overloaded and the other is not, the traffic will not move to the least loaded WAE.

- WCCP will recover from a loss of the WAE, but the connections terminated on the 'lost' WAE will be reset.

Best regards, Peter

[Patrick Moubarak]

Thank you for your reply Peter, greatly appreciated. I will try to apply those changes and see if it resolves the issue!

Patrick

[finn.poulsen]

Hi Peter,

Agree with you that the above setup should be configured with negotiated return in order to encapsulate the response into GRE and instruct the router that it shoudn't redirect the packet from WAAS multiple times.

But wouldn't this "error" have introduced a loop (and thus created much more severe problems) instead of "just" a non-working load-balancing scheme ?

best regards

Finn Poulsen