02-12-2011 12:31 AM - edited 08-28-2017 02:56 AM
As you know, WAAS is changing the layer 4 info of the packets it optimize. Security devices usually don't like so much when a device in the path is adding/changing some fields of the packet so some issues may arise when those changes are detected.
WCCP can also create some issues as it can encapsulate parts of the communication in GRE or when the device the traffic is sent to starts spoofing the reply from the real server.
Here is a summary of all the common problems you can run into and how they can be solved.
Problem definition:
It is common to have the WAE optimized traffic going through a Firewall in a setup like this one:
As stated before, the WAE is modifying the TCP header of the packet and two of those changes will be detected by the Firewall:
By default, the firewall will clear all of the non well-known TCP options in a packet so it will remove the WAAS option 0x21. This will prevent auto discovery to happen and will thus also prevent optimization.
Once the Edge WAE is sending the ACK from the 3WHS of an optimized connection, it increases the sequence number of the TCP connection by 2Gb to be able to differentiate the packets which have been optimized and the one which were not. When this packet is hitting the Firewall, it detects that it is out of the allowed TCP window and drops it. If you issue a show asp drop on the firewall, you'll see the value of "TCP packet SEQ past window" increasing.
First of all, you'll need to create a policy map that will allow the option 0x21 to go through the Firewall unstripped:
Allowing option 0x21 through
access-list All-TCP extended permit tcp any any ! class-map Match-All-TCP match access-list All-TCP ! tcp-map Allow21 tcp-options range 33 33 allow ! policy-map global_policy class Match-All-TCP set connection advanced-options Allow21 ! service-policy global_policy global |
Now to take care of the sequence number jump, you'll have to disable the sequence number randomization of the Firewall so that it doesn't modify it but you'll also need to disable TCP state checking for the traffic which is optimized:
Allowing TCP sequence jump
static (inside,outside) X.X.X.X X.X.X.X netmask Z.Z.Z.Z norandseq nailed static (outside,inside) Y.Y.Y.Y Y.Y.Y.Y netmask Z.Z.Z.Z norandseq nailed ! failover timeout -1 |
As weird as it seems, the failover timeout command is needed to have the nailed keyword effective on a static nat command
As you can see, disabling totally TCP state checking for all the optimized traffic defies the purpose of a Firewall and is a bit drastic so an improvement has been implemented in the newer software releases.
To make it a bit more user friendly and less impacting, a new feature has been introduced: WAAS inspection.
If you enable it on your firewall, it will automatically let the WAAS optimized traffic go through the Firewall.
Enabling WAAS inspection
policy-map global_policy class inspection_default inspect waas |
For you reference, here are the release notes of the versions the feature was introduced in : ASA 7.2(3) FWSM 3.2(1)
When the optimized traffic is sent over a WAN, you often want it to be encrypted and you might end up with the following setup:
In such cases and with the default MSS values, the overhead of the WAAS optimization and of the Encryption can cause the IP packet to be fragmented and it can lead to 2 different types of issue:
When a packet is set to be fragmented and has the DF bit set, the device supposed to fragment it will drop it instead and will send an ICMP “fragmentation needed and DF bit set” message to the source (Type 3 code 4).
The problem is that this ICMP message won't be redirected by WCCP and thus the Edge WAE will never be aware that the packet was dropped and will simply retransmit it after timeout.
Even when the fragmented packet doesn't have the DF bit set, it will cause issues but on the Core WAE this time. Since WCCP will only redirect packet with TCP headers, it will only redirect the first fragment and the other ones will bypass redirection since they don't have it. This will cause the WAE to miss parts of the communication and will prevent the traffic from being optimized correctly.
There are two commands on the WAE that will have him decrease the MSS of the connections going through it. Based on your network and on the overhead which will be added to your packet, you can adjust those values to prevent fragmentation from occurring. You can fine tune the MSS either on the WAN side:
tfo tcp optimized-mss <value> |
or on the LAN one:
tfo tcp original-mss <value> |
Once those are lowered, it should prevent the fragmentation from occurring.
For you reference, here are the entries for those in the commands guide: "tfo tcp optimized-mss" "tfo tcp original-mss"
When you setup your WAAS network, you might need to pass optimized traffic through an IPS running in Inline mode like this:
As for the Firewall, an IPS in Inline mode will run some checks on the layer 4 information. As the firewall does, it will clear the option 0x21 preventing TFO auto discovery and will report the sequence number changes done by the WAE. It can also report false positive attacks triggered by the files services handling of CIFSAO.
By default the Virtual Sensor TCP inspection mode is set to Strict to prevent attackers from bypassing the IPS inspection by sending packets out of order. Because of the sequence shift, it can cause issues when inspecting the WAAS optimized traffic. If you set the mode to Asymmetric, the IPS won't normalize the TCP sequence number.
To change the mode on IDM, here is what you need to do:
-Go to "Configuration > Policies > IPS Policies"
-Select the virtual sensor the WAE traffic will be inspected by and click on "Edit > Advanced Options"
-Change the normalizer mode to Asymmetric:
-Click on "Ok > Apply"
-Reboot the Sensor
If you want to change the mode via the CLI, here is how you need to do:
Changing normalizer mode via CLI
ips# conf t ips(config)# service analysis-engine ips(config-ana)# virtual-sensor vs0 ips(config-ana-vir)# inline-TCP-evasion-protection-mode asymmetric ips(config-ana-vir)# exit ips(config-ana)# exit Apply Changes?[yes]: yes Warning: Change of TCP evasion protection mode will not take effect until restart. ips# reset |
There are a bunch of signatures that you need to disable if you want to pass WAAS traffic through an IPS. If you don't, you'll get alarms like this one that will keep on firing on your monitoring station:
evIdsAlert: eventId=1243171268831920721 severity=low vendor=Cisco originator: hostId: IPS appName: sensorApp appInstanceId: 6631 time: 2009/09/17 09:08:13 2009/09/17 12:08:13 GMT+03:00 signature: description=TCP Option Other id=1306 created=20050304 type=anomaly version=S272 subsigId: 0 sigDetails: TCP Option Other Detected marsCategory: Info/Misc interfaceGroup: vs0 vlan: 0 participants: attacker: addr: locality=OUT X.X.X.X port: 0 target: addr: locality=OUT 0.0.0.0 port: 0 os: idSource=unknown relevance=unknown type=unknown summary: final=true initialAlert=1243171268831920576 summaryType=Regular 3 alertDetails: Regular Summary: 3 events this interval ; riskRatingValue: targetValueRating=medium 50 threatRatingValue: 50 interface: ge1_1 protocol: tcp |
This is the signature that will clear the TCP options when going through the IPS but there are others which will fire because of the sequence number change. Here is a list of which one you should disable:
Sig ID
Subsig ID
NameDescription
1306 | 0 | TCP Option Other | Here |
1330 | 12 | TCP Drop - Segment Out Of Order | Here |
1330 | 17 | TCP Drop - Segment out state order | Here |
1330 | 18 | TCP Drop - Segment out of window | Here |
1330 | 19 | TCP Drop - TCP timestamp option detected when not expected | Here |
3030 | 0 | TCP SYN Host Sweep | Here |
If you are using CIFSAO, you might also want to disable signature 5581/0 SMB Remote Srvsvc Service Access Attempt as it can also get triggered.
If you are setting up web caching/filtering on that your cache has to be located in a DMZ, you might end up with the following setup:
If this is how your network is connected you might run into two main issues:
If you have reverse path forwarding enabled on your interfaces, the Firewall will compare the Source of the packets he is receiving with his routing table. If he wouldn't route packets with this IP through the interface he got the packet on, he would drop it. If you have your cache engine spoofing the Web sites reply from the DMZ interface, the ASA will drop the reply as he is expecting to receive traffic from such IP to come to the outside interface and not to the DMZ one.
If the traffic from the router to the cache engine is GRE encapsulated and the return traffic is not, the ASA will drop the SYN/ACK for the session is it didn't see the original matching SYN for this connection.
This one is pretty obvious. To prevent uRPF from dropping the traffic, just disable it on the interface your cache is connected to.
Here is how it can be done through the CLI:
asa(config)# no ip verify reverse-path interface dmz |
If you want to do it on ASDM, just go to "Configuration > Firewall > Advanced > Anti-Spoofing" and set the value to False for your interface.
To prevent the ASA from discarding the SYN/ACK it gets, you'll need to have your Firewall bypass TCP state checking. If your ASA is running a version earlier then 8.2(1) or if your FWSM is older then 3.2(1), you'll need to implement the same solution based on static NAT which is describe in the first point.
If you are running at least those versions, you can use the "tcp-state-bypass" option which has been introduced to the Modular Policy Framework.
Here is how it can be setup:
Disabling TCP state check via the MPF
access-list tcp-bypass permit tcp any any ! class−map tcp-bypass match access−list tcp-bypass ! policy−map tcp-bypass_policy class bypass-traffic set connection advanced−options tcp−state−bypass ! service−policy tcp_bypass_policy DMZ |
If you need to do this via ASDM, just go to "Configuration > Firewall > Service Policy Rules" and add a new policy bound to the interface your cache is connected to.
On this policy, go to "Rule Actions > Advanced options" and enable state bypass from there:
In those situations, you might want to enable WCCP directly on the ASA. Just keep in mind that if you do so, the users and the cache needs to be connected to the ASA through the same interface, not through different ones.
Related Information:
Comments on the document itself or on what could be added to it are more than welcome.
To scale the performance of WAAS / WAE and to provide high reliability, Cisco has a new feature called ITD. Please see ITD (Intelligent Traffic Director) White Paper.
Also, recent blog : ITD: Load Balancing, Traffic Steering & Clustering using Nexus 5k/6k/7k
ITD Provides CAPEX and OPEX Savings for Customers
ITD (Intelligent Traffic Director) is a hardware based multi-Tbps Layer 4 load-balancing, traffic steering and clustering solution on Nexus 5K/6K/7K series of switches. It supports IP-stickiness, resiliency, NAT, (EFT), VIP, health monitoring, sophisticated failure handling policies, N+M redundancy, IPv4, IPv6, VRF, weighted load-balancing, bi-directional flow-coherency, and IPSLA probes including DNS.
ITD is much superior than legacy solutions like PBR, WCCP, ECMP, port-channel, layer-4 load-balancer appliances.
For example, ITD comparison with WCCP:
Feature/Benefit | N7k WCCP | N7k ITD |
Appliance is unaware of the protocol | No | Yes |
Protocol support | IPv4 | IPv4, IPv6 |
Number of TCAM entries (say, 100 SVI, 8 nodes, 20 ACEs) | Very High 16000 | Very low 160 |
Weighted load-balancing | No | Yes |
User can specify which bits to use for load-balancing | No | Yes |
Number of nodes | 32 | 256 |
Support for IPSLA probes | No | Yes |
Support for Virtual IP | No | Yes |
Support for L4-port load-balancing | No | Yes |
Capability to choose src or dest IP for load-balancing | No | Yes |
Customer support needs to look at switch only, or both the switch and appliance | Both | Switch only |
Sup CPU Overhead | High | None |
DCNM Support | No | Yes |
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: