cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
18256
Views
19
Helpful
2
Comments
Nicolas Fournier
Cisco Employee
Cisco Employee

 

 

 

Introduction

 

As you know, WAAS is changing the layer 4 info of the packets it optimize. Security devices usually don't like so much when a device in the path is adding/changing some fields of the packet so some issues may arise when those changes are detected.

WCCP can also create some issues as it can encapsulate parts of the communication in GRE or when the device the traffic is sent to starts spoofing the reply from the real server.

Here is a summary of all the common problems you can run into and how they can be solved.

 

WAAS optimized traffic going through an ASA/FWSM

 

Problem definition:

 

It is common to have the WAE optimized traffic going through a Firewall in a setup like this one:

 

Firewall.jpg

 

As stated before, the WAE is modifying the TCP header of the packet and two of those changes will be detected by the Firewall:

 

1.) WAE is adding TCP option 0x21 to the packet

 

By default, the firewall will clear all of the non well-known TCP options in a packet so it will remove the WAAS option 0x21. This will prevent auto discovery to happen and will thus also prevent optimization.

 

2.) WAE is bouncing the TCP sequence number of the 3WHS ACK by 2Gb

 

Once the Edge WAE is sending the ACK from the 3WHS of an optimized connection, it increases the sequence number of the TCP connection by 2Gb to be able to differentiate the packets which have been optimized and the one which were not. When this packet is hitting the Firewall, it detects that it is out of the allowed TCP window and drops it. If you issue a show asp drop on the firewall, you'll see the value of "TCP packet SEQ past window" increasing.

 

Solution on ASA 7.2(2) or FWSM 3.1(11) and before:

 

First of all, you'll need to create a policy map that will allow the option 0x21 to go through the Firewall unstripped:

 

Allowing option 0x21 through

access-list All-TCP extended permit tcp any any
!
class-map Match-All-TCP
    match access-list All-TCP
!
tcp-map Allow21
    tcp-options range 33 33 allow
!
policy-map global_policy
    class
Match-All-TCP
        set connection advanced-options Allow21
!
service-policy global_policy global

 

Now to take care of the sequence number jump, you'll have to disable the sequence number randomization of the Firewall so that it doesn't modify it but you'll also need to disable TCP state checking for the traffic which is optimized:

 

Allowing TCP sequence jump

static (inside,outside) X.X.X.X X.X.X.X netmask Z.Z.Z.Z norandseq nailed
static (outside,inside) Y.Y.Y.Y Y.Y.Y.Y netmask Z.Z.Z.Z norandseq nailed
!
failover timeout -1

 

As weird as it seems, the failover timeout command is needed to have the nailed keyword effective on a static nat command

 

As you can see, disabling totally TCP state checking for all the optimized traffic defies the purpose of a Firewall and is a bit drastic so an improvement has been implemented in the newer software releases.

 

Solution on ASA 7.2(3) or FWSM 3.2(1) and after:

 

To make it a bit more user friendly and less impacting, a new feature has been introduced: WAAS inspection.

If you enable it on your firewall, it will automatically let the WAAS optimized traffic go through the Firewall.

 

Enabling WAAS inspection

policy-map global_policy
    class inspection_default
        inspect waas

 

For you reference, here are the release notes of the versions the feature was introduced in : ASA 7.2(3) FWSM 3.2(1)

 

 

WAAS optimized traffic going through a GRE over IPSEC or DMVPN tunnel

 

Problem definition:

 

When the optimized traffic is sent over a WAN, you often want it to be encrypted and you might end up with the following setup:

 

GRE.jpg

 

In such cases and with the default MSS values, the overhead of the WAAS optimization and of the Encryption can cause the IP packet to be fragmented and it can lead to 2 different types of issue:

 

1.) Packets with DF bit set

 

When a packet is set to be fragmented and has the DF bit set, the device supposed to fragment it will drop it instead and will send an ICMP “fragmentation needed and DF bit set” message to the source (Type 3 code 4).

The problem is that this ICMP message won't be redirected by WCCP and thus the Edge WAE will never be aware that the packet was dropped and will simply retransmit it after timeout.

 

2.) Fragmented packets

 

Even when the fragmented packet doesn't have the DF bit set, it will cause issues but on the Core WAE this time. Since WCCP will only redirect packet with TCP headers, it will only redirect the first fragment and the other ones will bypass redirection since they don't have it. This will cause the WAE to miss parts of the communication and will prevent the traffic from being optimized correctly.

 

Solution:

 

There are two commands on the WAE that will have him decrease the MSS of the connections going through it. Based on your network and on the overhead which will be added to your packet, you can adjust those values to prevent fragmentation from occurring. You can fine tune the MSS either on the WAN side:

 

tfo tcp optimized-mss <value>

 

or on the LAN one:

 

tfo tcp original-mss <value>

 

Once those are lowered, it should prevent the fragmentation from occurring.

For you reference, here are the entries for those in the commands guide: "tfo tcp optimized-mss" "tfo tcp original-mss"

 

 

WAAS optimized traffic going through an IPS

 

Problem definition:

 

When you setup your WAAS network, you might need to pass optimized traffic through an IPS running in Inline mode like this:

 

IPS.jpg

 

As for the Firewall, an IPS in Inline mode will run some checks on the layer 4 information. As the firewall does, it will clear the option 0x21 preventing TFO auto discovery and will report the sequence number changes done by the WAE. It can also report false positive attacks triggered by the files services handling of CIFSAO.

 

Solution:

 

1.) Change the Virtual Sensor mode to Asymmetric

 

By default the Virtual Sensor TCP inspection mode is set to Strict to prevent attackers from bypassing the IPS inspection by sending packets out of order. Because of the sequence shift, it can cause issues when inspecting the WAAS optimized traffic. If you set the mode to Asymmetric, the IPS won't normalize the TCP sequence number.

To change the mode on IDM, here is what you need to do:

 

-Go to "Configuration > Policies > IPS Policies"

-Select the virtual sensor the WAE traffic will be inspected by and click on "Edit > Advanced Options"

-Change the normalizer mode to Asymmetric:

 

Asymmetric.jpg

 

-Click on "Ok > Apply"

-Reboot the Sensor

 

If you want to change the mode via the CLI, here is how you need to do:

 

Changing normalizer mode via CLI

ips# conf t
ips(config)# service analysis-engine
ips(config-ana)# virtual-sensor vs0
ips(config-ana-vir)# inline-TCP-evasion-protection-mode asymmetric
ips(config-ana-vir)# exit
ips(config-ana)# exit
Apply Changes?[yes]: yes
Warning: Change of TCP evasion protection mode will not take effect until restart.
ips# reset

 

 

2.) Disable signatures that will fire on WAAS accelerated traffic

 

There are a bunch of signatures that you need to disable if you want to pass WAAS traffic through an IPS. If you don't, you'll get alarms like this one that will keep on firing on your monitoring station:

 

evIdsAlert: eventId=1243171268831920721 severity=low vendor=Cisco
   originator:
     hostId: IPS
     appName: sensorApp
     appInstanceId: 6631
   time: 2009/09/17 09:08:13 2009/09/17 12:08:13 GMT+03:00
   signature: description=TCP Option Other id=1306 created=20050304
type=anomaly version=S272
     subsigId: 0
     sigDetails: TCP Option Other Detected
     marsCategory: Info/Misc
   interfaceGroup: vs0
   vlan: 0
   participants:
     attacker:
       addr: locality=OUT X.X.X.X
       port: 0
     target:
       addr: locality=OUT 0.0.0.0
       port: 0
       os: idSource=unknown relevance=unknown type=unknown
   summary: final=true initialAlert=1243171268831920576
summaryType=Regular 3
   alertDetails: Regular Summary: 3 events this interval ;
   riskRatingValue: targetValueRating=medium 50
   threatRatingValue: 50
   interface: ge1_1
   protocol: tcp

 

This is the signature that will clear the TCP options when going through the IPS but there are others which will fire because of the sequence number change. Here is a list of which one you should disable:

 

Sig ID
Subsig ID
NameDescription

13060TCP Option OtherHere
133012TCP Drop - Segment Out Of OrderHere
133017TCP Drop - Segment out state orderHere
133018TCP Drop - Segment out of windowHere
133019TCP Drop - TCP timestamp option detected when not expectedHere
30300TCP SYN Host SweepHere

 

 

If you are using CIFSAO, you might also want to disable signature 5581/0 SMB Remote Srvsvc Service Access Attempt as it can also get triggered.

 

 

WCCP through an ASA

 

Problem definition:

 

If you are setting up web caching/filtering on that your cache has to be located in a DMZ, you might end up with the following setup:

 

CE.jpg


If this is how your network is connected you might run into two main issues:

 

1.) Reverse path forwarding denying spoofed traffic from your cache

 

If you have reverse path forwarding enabled on your interfaces, the Firewall will compare the Source of the packets he is receiving with his routing table. If he wouldn't route packets with this IP through the interface he got the packet on, he would drop it. If you have your cache engine spoofing the Web sites reply from the DMZ interface, the ASA will drop the reply as he is expecting to receive traffic from such IP to come to the outside interface and not to the DMZ one.

 

2.) GRE encapsulated WCCP redirected traffic without GRE return

 

If the traffic from the router to the cache engine is GRE encapsulated and the return traffic is not, the ASA will drop the SYN/ACK for the session is it didn't see the original matching SYN for this connection.

 

Solution:

 

1.) Disable reverse path forwarding

 

This one is pretty obvious. To prevent uRPF from dropping the traffic, just disable it on the interface your cache is connected to.

Here is how it can be done through the CLI:

 

asa(config)# no ip verify reverse-path interface dmz

 

If you want to do it on ASDM, just go to "Configuration > Firewall > Advanced > Anti-Spoofing" and set the value to False for your interface.

 

2.) Disable TCP state check

 

To prevent the ASA from discarding the SYN/ACK it gets, you'll need to have your Firewall bypass TCP state checking. If your ASA is running a version earlier then 8.2(1) or if your FWSM is older then 3.2(1), you'll need to implement the same solution based on static NAT which is describe in the first point.

If you are running at least those versions, you can use the "tcp-state-bypass" option which has been introduced to the Modular Policy Framework.

Here is how it can be setup:

 

Disabling TCP state check via the MPF

access-list tcp-bypass permit tcp any any
!
class−map tcp-bypass
match access−list tcp-bypass
!
policy−map tcp-bypass_policy
    class bypass-traffic
        set connection advanced−options tcp−state−bypass
!
service−policy tcp_bypass_policy DMZ

 

If you need to do this via ASDM, just go to "Configuration > Firewall > Service Policy Rules" and add a new policy bound to the interface your cache is connected to.

On this policy, go to "Rule Actions > Advanced options" and enable state bypass from there:

 

Bypass.jpg

 

In those situations, you might want to enable WCCP directly on the ASA. Just keep in mind that if you do so, the users and the cache needs to be connected to the ASA through the same interface, not through different ones.

 

Related Information:

 

Troubleshooting Prepositioning on WAAS 4.1.1 and above

GRE Redirection in WCCP Creates new tunnel interfaces

Comments
Nicolas Fournier
Cisco Employee
Cisco Employee

Comments on the document itself or on what could be added to it are more than welcome.

To scale the performance of WAAS / WAE and to provide high reliability, Cisco has a new feature called ITD. Please see ITD (Intelligent Traffic Director) White Paper.

Also, recent blog : ITD: Load Balancing, Traffic Steering & Clustering using Nexus 5k/6k/7k

 

ITD Provides CAPEX and OPEX Savings for Customers

ITD (Intelligent Traffic Director) is a hardware based multi-Tbps Layer 4 load-balancing, traffic steering and clustering solution on Nexus 5K/6K/7K series of switches. It supports IP-stickiness, resiliency, NAT, (EFT), VIP, health monitoring, sophisticated failure handling policies, N+M redundancy, IPv4, IPv6, VRF, weighted load-balancing, bi-directional flow-coherency, and IPSLA probes including DNS.

ITD is much superior than legacy solutions like PBR, WCCP, ECMP, port-channel, layer-4 load-balancer appliances.

 

For example, ITD comparison with WCCP:

Feature/Benefit

N7k WCCP

N7k ITD

Appliance is unaware of the protocol

No

Yes

Protocol support

IPv4

IPv4, IPv6

Number of TCAM entries

(say, 100 SVI, 8 nodes, 20 ACEs)

Very High

16000

Very low

160

Weighted load-balancing

No

Yes

User can specify which bits to use for load-balancing

No

Yes

Number of nodes

32

256

Support for IPSLA probes

No

Yes

Support for Virtual IP

No

Yes

Support for L4-port load-balancing

No

Yes

Capability to choose src or dest IP for load-balancing

No

Yes

Customer support needs to look at switch only, or both the switch and appliance

Both

Switch only

Sup CPU Overhead

High

None

DCNM Support

No

Yes

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: