Showing results for 
Search instead for 
Did you mean: 

Make it stop, just make it stop!

Cisco Employee

Hello dear reader,

This is the first post in hopefully a long line of posts giving short nuggets of tips and tricks for 'content' related products.

I wanted to start the series with the worst possible example: you are using WAAS on you network to accelerate traffic and everything is fine until you get frantic requests to turn it off as it is preventing some important operation to work.

We've all been there: there is little real information, just confusion and people that are urgently asking you how long the problem will still last.

At this moment you (or someone responsible) is going to have to decide on a strategy to follow, the options in general are:

  • start turning things off and reloading equipment until it works
  • investigate the root cause of the problem

This is a clear choice: the moment you take corrective actions that you think will fix the problem it becomes very difficult to determine the root cause. The question you have to ask is: "I can guess what the problem is and do something, which might of might not solve the problem. When I do this we will not know what is the cause of the problem afterwards. Or I can first investigate and then fix the problem. What should I do?".

The preferred option from our point of view obviously is that you call TAC with a 'network down' severity issue. We will ask you the same question and together do what is required to get the network back up.

If this is not possible, then please collect as much information as you can:

  • which clients where impacted
  • what traffic was impacted
  • to what servers
  • when did the problem start
  • was similar traffic working before
  • wccp status from routers/switches if appropriate
  • sysreport or 'show tech' information from the WAAS devices involved

Another point sometimes forgotten is to log all console output on all devices. This might give us the clue to find the root cause...

The requests become more urgent and you decide to try to fix the problem by turning off the interception of traffic by the WAAS. After you have done this WAAS will no longer have an impact on the traffic and whatever problem remains is not due to WAAS (see an exception below). This is better then reloading WAAS as it will preserve some statistics that could give information to find the root cause.

If the impact is limited to one network, server or client you can chose to only ignore traffic from/to this group. With older versions of WAAS you can do this using a bypass static configuration line if you are using WCCP interception. From 4.2 onwards you can use a interception ACL  which is easier to use and works both for WCCP and inline interception.

In case that you are not certain that the impact is limited or you are using inline interception with 4.1 you can turn off all interception with ease. For WCCP interception you simply do:

WAE# conf t

WAE(config)# no wccp version 2

on all WAE's that are doing interception. For inline interception you do:

WAE# conf t

WAE(config)# interface InlineGroup 1/0

WAE((config-if)# shutdown

this will stop all inline interception at the hardware level. If you are using an ip on the inline interface the you cannot shutdown the interface without losing access, in that case do:

WAE# conf t

WAE(config)# interface InlineGroup 1/0

WAE((config-if)# no inline

Naturally you will have to repeat this for the other InlineGroups you have active.

After doing either of these steps WAAS is out of the picture and you should check if this fixed the problem. Except for cabling problems in the inline case, because even if the interface acts as a piece of cable now, the traffic still goes across the cables obviously.

In the next installment I plan to talk about the other option: finding a root cause. Feel free to ask questions or ideas for other posts in the comments below...


This widget could not be displayed.