Re: 2851 IOS upgrade failed with WAE

dchandler_1 · ‎08-26-2010

Hello,

I have a strange situation with WAAS. I am in progress of a project to upgrade my Cisco IP telephony and as a result preferred to update my remote sites routers to 12.4.x T code in order to register DSP resources over SCCP and match closely the CMgr version as many of the sites running mainline code didn't have updated parameters for this. To make a long story short, I upgraded (for other reasons) several non-voice sites routers that have WAE 512s running 4.1.7 with the latest 12.4.xT code and didn't have an issue. All upgraded sites are running 2821's or 2811's except this one site I had an issue with. I needed T code on this particular site router and assumed since others sites reacted well with the IOS upgrade and WAAS that this site wouldn't be different but it was!

Situation: After updating this 2851 router from 12.4.25c mainline to 12.4.24T3, traffic from WAN was directing to WAE but no return. It appeared also that any LAN redirection was also working but no return from WAE. I didn't see any WCCP flow statistics incrementing but traffic was being directed to the WAE. WAE connection statistics were building but all TCP based traffic was getting dropped. I tried 3 different feature sets of same code version wondering if I've encountered a bug but none of the featured images worked. I downgraded to 12.4.24T2 and still nothing. Changed IOS back to mainline code and WAE was back operational as expected. During the testing of each IOS change, I've reset the WAE also hoping it didn't get hung up with some WCCP process from the previous IOS change in case there was some codebug occuring.

Layout:

With respect to this site, we have a 2851 with multilinked serial interfaces to MPLS; WAN interface is redirecting 62 in with an ACL that disallows a couple of hosts. LAN side of router is single Gig port (router on a stick). 5 sub-interfaces configured with DOT1Q encap. including one interface for WAAS. 2 of these user interfaces are redirecting 61 in and of course the interface servicing WAAS is redirect excluded in. The 61 WCCP group is also using ACL to oppositely match WCCP 62 group.

Why would this upgrade cause a problem especially using latest T code? Feature sets would be irrelevant. One more thing that makes this even stranger is I have one other voice site running a 2851 with 12.4.24T3 (already running) and is configured for WCCP the very same with same 512WAE and 4.1.7 code. The only difference is this router is running a "SP" feature set which I didn't try on this site that is failing.

Does this make any sense to anyone? I've seen posted from one of the WAAS experts a document listing compatible IOS version across platforms but these efforts are the latest code. Any help would be greatly appreciated! Thanks everyone.

Bhavin Yadav · ‎08-26-2010

Hi,

From the details of description, it really is not a known issue at this moment.

You may want to capture following CLI commands and verify the output to point us to the issue.

On WAE:
wae#sh wccp gre
wae#show wccp routers
wae#sh egress-methods

On Router:
    1. show version
    2. show ip wccp
    3. show ip wccp interface detail
    4. show ip wccp service
    5. show ip wccp detail
    6. show ip wccp internal (*)
    7. show running-config
    8. show ip wccp <61 / 62> hash
    9. sh wccp mask tcp-promis

1. debug ip wccp events

2. debug ip wccp packets
The debugging output can be fairly verbose, so it is probably best to turn off console logging and log the debugs to a buffer. When done capture the results via “show log” – make sure you capture everything, some terminal emulators can “chop off” text beyond 80 characters.

One quick question though: is this working at any of the other site with same hw+sw version of WAE and 2851?

If you think you can post the cli output from above commands as attachment, please do so or may want to open a TAC case to address this one.

Regards.

dchandler_1 · ‎09-07-2010

Thanks Bhavin.

Sorry, I've been traveling a lot lately for business and I haven't had a chance to respond. And it looks like I'll be heading out again for a few days. Meanwhile, I did want to say that I looked at a lot of these things and didn't see anything apparent. Egress methods show GRE Return. When I ran "show wccp flows tcp" I didnt' see any flow counts. However I could see packets redirecting at the router with "show ip wccp". I debugged events and packets and saw "See You" packets being sent back to the caching appliance. WCCP looked fine from the router's viewpoint. WCCP communications looked good rather. But anything getting redirected to the WAE would get dropped.

I don't have a chance right now to change IOS back to one of the non-working versions to get all of the info. but I will try soon. Unfortunately, the site is where my executives reside and it's always a tough situation to do anything there most of the time.

And to answer your question about having another configuration in place working .... yes I do. I have another 2851 with 12.4(24)T3, same WAE model with WAE-4.1.7 code and everything configured the same including connections as the site that isn't working. The only difference as mentioned before is that this working site router is running SP feature set which I know shouldn't matter but I didn't try a SP feature set however I went through ADV IP, ADV ENTER, and IP VOICE and neither worked with 2 different 12.4(X)Tx versions. I was grasping at straws at this point.

I figured I was going to need to get TAC involved when I could spare some time to run one of the non-working IOS images and could afford to not have WAAS working for the site but was hoping one of you guys might have seen something like this and could offer something to try to fix the issue. It is very odd indeed. I have other 28xx routers and all of the upgrades worked that aren't 2851's.

I'll post more info. when I can.

Thanks again!

Wayne

pevaneyn · ‎08-27-2010

Hi,

The upgrade should have worked.

If I understand correctly WCCP is 'working' and traffic is redirecting, but the replies from the WAE are getting lost?

I guess you checked this on the WAE... how is the traffic returning?

One major difference with the newer version is that it allows Layer2 redirection and you might get L2 redirection instead of WCCP redirection between the WAE and the router. You should check on the WAE with 'show egress' in both cases.

Another reason for problems might be a firewall or HBAR configuration on the router. Are you using something like that?

Peter

dchandler_1 · ‎09-07-2010

Hi.

No firewall in place between sites. You said HBAR but assume you meant NBAR. I do not have NBAR running.

Traffic return is configured for GRE and show egress-methods confirms that. I have more info. in the reply to Peter just above.

When I get a chance to update the IOS again in that site, I am going to use a SP feature set of the T train and see if it works because I do have one other site with a 2851 running T code but with SP feature set. The only thing I can get working right now is mainline code. I can see perhaps version differences within a particular code train causing something like this (wccp compatibility) but not anything related to the IOS feature set or crossing mainline to T.

Any assistance is greatly appreciated. So thanks for your feedback! It is all welcomed!

Regards,

Wayne