02-28-2007 07:01 AM
Hi,
Has anyone ran into this senario before. Before anyone answers with "move your WAE off the user subnet", it already has been.
I have wccp 61 redirect in on the user subnet (gig0/0.83 of a dot1q trunk). The WAE is on gig0/1. Before I apply wccp62 to the serial link, I attempt to telnet from a user pc to the router (same subnet, clients default gateway), and the telnet fails. I get a "looping packet detected" on the router console. It shows the source of the packet as the router (wccp router id actually), and the destination ip of the WAE, but the packet came in gig0/1 (interface connected to wae). Obviously the WAE returned the packet to the router (with the original GRE headers, (router as source)). I thought WCCP would understand this as "don't redirect this traffic to me anymore", but the router, actually tries to route it back down gig0/1 and then sees it as a looping packet. I believe the WAE is returning the encapsulated packet to the router to indicate it doesn't want the flow, and the router is attempting to route the GRE packet, instead of realizing it should remove the GRE header and route the internal packet. Router is IOS 12.4(12) as recommended by my Cisco engineer. 2821 router.
For kicks, I continue the WCCP setup on the datatcenter side. As expected, it doesn't work. When I apply the WCCP to the datacenter router (only redirecting lab subnet), the entire lab subnet is unreachable via TCP (but icmp still works as expected).
The WCCP configuration isn't very complex, I can't believe its something I'm doing. I think its a code issue.
Any advise?
02-28-2007 10:49 AM
Is there an 61 or 62 out anywhere in the mix? If so, then do a redirect exclude command on the g0/1 where the appliance resides. Can you ping to the WAE from user subnet? and does show ip wccp show properly? Can you post router and WAE configs?
02-28-2007 11:08 AM
no "out" anywhere. The LAB router has a WAE list to only allow redirect to the lab WAE. I don't even need the 62 in on the WAN side, just applying 61 in on the LAN side breaks telnet to the router.
LOOPING PACKET DETECTION:
from router console
Feb 27 14:56:32.924: %IP-3-LOOPPAK: Looping packet detected and dropped -
src=132.242.11.18, dst=153.61.83.70, hl=20, tl=76, prot=47, sport=0, dport=0
in=GigabitEthernet0/1, nexthop=153.61.83.70, out=GigabitEthernet0/1
options=none -Process= "IP Input", ipl= 0, pid= 77 -Traceback= 0x410F6978 0x415CC960 0x415CDC60 0x415BBB38 0x415BCF18 0x415BD27C 0x415BD2FC 0x415BD4E8
*********************************************
Router configuration:
ip wccp 61 redirect-list REDIRECT-WAAS-SUBNETS-61 group-list remote-waas-box
interface Loopback0
ip address 132.242.11.18 255.255.255.255
h323-gateway voip bind srcaddr 132.242.11.18
interface GigabitEthernet0/0.83
description << data vlan 83 >>
encapsulation dot1Q 83
ip address 153.61.83.3 255.255.255.192
ip helper-address 192.127.250.22
ip helper-address 149.25.1.182
no ip proxy-arp
ip wccp 61 redirect in
standby 83 ip 153.61.83.1
standby 83 priority 200
standby 83 preempt
standby 83 track Serial0/1/0:0.99 100
interface GigabitEthernet0/1
description << WHQ LAB CE connection
ip address 153.61.83.65 255.255.255.192
load-interval 30
duplex full
speed 100
ip access-list standard remote-waas-box
permit 153.61.83.70
!
ip access-list extended REDIRECT-WAAS-SUBNETS-61
permit ip 153.61.83.0 0.0.0.63 any
*********************************************
WAE configuration:
device mode application-accelerator
!
primary-interface GigabitEthernet 1/0
!
!
!
interface GigabitEthernet 1/0
ip address 153.61.83.70 255.255.255.192
no autosense
bandwidth 100
full-duplex
exit
!
wccp router-list 1 153.61.83.65
wccp tcp-promiscuous router-list-num 1
wccp version 2
wccp slow-start enable
!
02-28-2007 11:27 AM
This is a GRE frame that is looping since the protocol=47. But has nothing to do with WAAS. This is a routing/switching issue. The frame that is being sent out is coming right back into the same interface. The WAE is on a crossover cable? So, this would me the show cdp n detail would show the router/WAE in each device. Please make sure of this. Does show ip arp from each device show the correct MAC address for the other device? IE, in show ip arp, the .65 and .70 are shown. Then do they match when you goto the other device?
02-28-2007 11:33 AM
I agree it has nothing to do with WAAS. I believe the WAE is returning the GRE frame, to signal it doesn't want the flow, but the router is not removing the GRE and forwarding, instead, its trying route the packet, which sends it back down the same interface.
Do you think the crossover is the issue? I use crossovers with my WAE's running ACNS/CDN software without any issues.
The CDP/ARP is as expected.
-------------------------
Device ID: SUSDAY817
Entry address(es):
IP address: 153.61.83.70
Platform: FE511, Capabilities: Host
Interface: GigabitEthernet0/1, Port ID (outgoing port): GigabitEthernet 1/0
Holdtime : 173 sec
Version :
Cisco Wide Area Application Services Software
Copyright (c) 1999-2007 by Cisco Systems, Inc.
Cisco Wide Area Application Services Software Software Release 4.0.5 (build b4
Jan 17 2007)
Version: fe511-4.0.5.4
advertisement version: 1
show arp from router:
Internet 153.61.83.70 0 000d.6084.3056 ARPA GigabitEthernet0/1
Internet 153.61.83.65 - 0019.06db.bbe1 ARPA GigabitEthernet0/1
show arp from WAE:
SUSDAY817#show arp
Protocol Address Flags Hardware Addr Type Interface
153.61.83.65 Adj 00:19:06:DB:BB:E1 ARPA GigabitEthernet1/0
02-28-2007 12:45 PM
Ok, it would appear that this is a software defect known as CSC30875 where wccp blocks access to router via telnet. Going to be resolved in 12.4.13 ( do not know when this will be out).
02-28-2007 12:51 PM
Thanks for your help. Does this just affect telnet access to the router? When I do apply WCCP to both the datacenter and lab sides, I lose complete TCP access to the lab subnet.
Is there a url for the bug? When I look it up, I get "Sorry -- The defect you have requested CSCsc30875 cannot be displayed.
This may be due to one or more of the following:
The defect number does not exist.
The defect does not have a customer-visible description available yet.
The defect has been marked Cisco Confidential."
03-20-2007 06:12 PM
Gary,
I too thought that the WAE returned the packet like a traditional cache engine and compliant with the wccpv2 internet draft using GRE encapsulation. However, according to packet traces and an engineer i spoke with recently the WAE does not understand GRE encapuslation for returning traffic. If you look at a packet trace of proper redirection you will see the incoming packet with a GRE encapsulation then you will see a follow up packet, the unencapsulated GRE packet coming into the WAE with the proper source client ip and destination ip address. Then the WAE will respond with an unencapsulated native IP packet sent to its configured default gateway.
Apparently this is why WAAS has the requirement that the WAE be on a directly attached subnet and not be located on the same user subnet like we used to be able to do with the old cache engines. Since there is not a true tunnel involved there is no alternate path down which to redirect the pack so it (WAE) must be segmented by moving it to a seperate interface and subnet off the redirecting router. This is also why you cannot locate a WAE w/ WAAS multiple hops away from the redirecting router. If you did the WAE would receive the redirected packet with GRE encapsulation but would then use its native default gateway (since there is no further GRE encapsulation and subsequent tunnel to route over on the return response) and the packet would never be returned to the router that redirected the packet in the first place.
Does that make since?
My issue with the looping packet was a wccp redirect routing loop that i solved with redirect ACLs. I actually broke TACACs on one of the WAN routers in my HA scenario before using redirect ACL (next time i will make sure i use them by default). Remember, tcp prom redirects all TCP traffic whether the WAAS appliance passes it through or not. If you redirect to WAAS you are relying on its routing configuration to return the packet back to the redirecting router not the GRE tunnel as for mentioned with cache engines.
HTH
Mike
03-21-2007 06:41 AM
Hi Mike,
Actually, we found the looping packet is a bug in the 12.4.12 code. The router doesn't realize the packet is destined for itself. As far as I can tell, it only impacts telnet requests from directly connected hosts.
As for the WAE being multi-hops away, I talked to 3 cisco engineers, they all assured me you can do it, but none of them had ever actually seen it done. I move the WAE to a directly connected subnet and all is good. Acceleration/cache looks good, and the new auto-server discovery is working perfectly.
You bring up a good point, as far as the GRE return. If you have 2 separate WAN environments, and want the WAE to service both, I believe there will be an issue, as the WAE will return the packet to its default gateway rather than the router that issued the redirect, which may be the wrong router. That router "a" will then pass the packet over the LAN segment to the correct router "b", but now that router "b" will redirect the packet back to the WAE because it came in on a user LAN interface and cause a loop again.
I hope to test this senario in the next couple weeks, but I suspect it will break. Seems like it would make more since for the WAE to always return the packet to the router that send it the packet in the first place!
Gary
03-21-2007 07:18 AM
The default gateway loop is exactly what i experienced. Had to use redirect lists and HSRP interface tracking to solve the issue. It only occured when primary router wan link was down and primary router was still active HSRP gateway.
With the wan link down the next hop for the remote networks normally available via the wan became available via the core interfaces on the gi0/0 and gi0/1 interfaces. Once fowarded to the core the core would forward the packets back to the secondary wan router (standby hsrp normally) that had ip wccp 61 redirect in configured on its internal gi0/0 and gi0/1 interfaces. It would redirect to the WAE and the wae would send the response back out its default gateway, creating the loop.
Are you using the T version of code? Whats the exact IOS level, i want to avoid it.
Here is what i saw
Mar 11 22:24:17: %IP-3-LOOPPAK: Looping packet detected and dropped -
src=10.250.1.2, dst=10.1.33.11, hl=20, tl=76, prot=47, sport=0, dport=0
in=Vlan33, nexthop=10.1.33.11, out=Vlan33
options=none -Process= "IP Input", ipl= 0, pid= 96, -Traceback= 0x615E55A8 0x61
B71BF8 0x61B71E04 0x61B72830 0x61B72C24 0x61B5BBA8 0x61B5D790 0x61B5AEBC 0x61B5B
1B4 0x61B5B270 0x61B5B414
Mar 11 22:26:45: %IP-3-LOOPPAK: Looping packet detected and dropped -
src=10.250.1.2, dst=10.1.33.11, hl=20, tl=76, prot=47, sport=0, dport=0
in=Vlan33, nexthop=10.1.33.11, out=Vlan33
options=none -Process= "IP Input", ipl= 0, pid= 96, -Traceback= 0x615E55A8 0x61
B71BF8 0x61B71E04 0x61B72830 0x61B72C24 0x61B5BBA8 0x61B5D790 0x61B5AEBC 0x61B5B
1B4 0x61B5B270 0x61B5B414
Regarding the GRE packet return from what i can see in the packet captures from the WAE, GRE encapsulation is not being used on the return packet.
03-21-2007 08:00 AM
That is exactly what my syslog message looked like.
I was told by a Cisco engineer w/ CCIE that it should be fixed in 12.4.13, but that wasn't out last I checked.
The bugId is listed in this thread, but when I look it up on CCO it won't give the details.
Current image:
flash:c2800nm-spservicesk9-mz.124-12.bin
Cisco 2821 (revision 53.51) with 249856K/12288K bytes of memory.
Processor board ID FTX1034A00D
2 Gigabit Ethernet interfaces
1 Serial interface
4 Channelized T1/PRI ports
DRAM configuration is 64 bits wide with parity enabled.
239K bytes of non-volatile configuration memory.
125440K bytes of ATA CompactFlash (Read/Write)
03-21-2007 10:54 AM
I am running 12.4(11)T.
c3845-advipservicesk9-mz.124-11.T.bin
ill keep an eye out for the bug as well.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide