06-21-2010 09:16 AM
Greetings,
We are trying to get to the bottom of an issue we are seeing, but unfortunately are not sure where to start. We have (2) 7931's in the Main DC and (1) 7931 in the backup datacenter (BDC), and well over 20 remote sites running NM-WAE, OE574 and OE674. We had an issue over the weekend where traffic from several remote sites was redirected to our BDC due to power outage. When this occurred ldap authentication broke for these sites as well as other CIFS traffic for users that were already authenticated.
We have seen this before and each time we have seen this we have noticed that the access-list on the core routers (7609) used for wccp starts matching (meaning the device is using software instead of hardware). The output below shows what we saw last time a site started experiencing issues such as, could not authenticate, could not open files, etc... We removed the site from the ACL and everything started working, of course we were no longer able to accelerate/optimize traffic going to the BDC once it was removed.
We saw this again this weekend. Several sites reported that they could not authenticate, when we investigated they were going to BDC servers due to a power outage and the ACL's had started incrementing, once again we had to remove them in order for them to be able to authenticate.
At this time we suspect there might have been asymmetric routing occurring during the power outage, but do not have data to back that up at this time. Has anyone see this type of issue before? or can anyone confirm if asymmetric routing could cause this type of behavior.
=================================
Extended IP access list WAAS_WCCP
10 permit ip 192.168.2.0 0.0.0.255 any
20 permit ip any 172.25.2.0 0.0.0.255
---- cut for brevity ------
90 permit ip 10.1.64.0 0.0.0.255 any
100 permit ip any 10.1.64.0 0.0.0.255
110 permit ip 10.1.74.0 0.0.0.255 any
120 permit ip any 10.1.74.0 0.0.0.255
130 permit ip 10.1.130.0 0.0.0.255 any
140 permit ip any 10.1.130.0 0.0.0.255
150 permit ip 10.1.213.0 0.0.0.255 any
160 permit ip any 10.1.213.0 0.0.0.255
170 permit ip 10.1.236.0 0.0.3.255 any
180 permit ip any 10.1.236.0 0.0.3.255
190 permit ip 10.1.24.0 0.0.1.255 any
200 permit ip any 10.1.24.0 0.0.1.255 (1914211 matches)
210 permit ip 10.1.48.0 0.0.0.255 any
220 permit ip any 10.1.48.0 0.0.0.255
===============================================
06-21-2010 10:03 AM
Do you see any indication in the WAAS logs that connections are failing due to a redirection loop? The message in syslog.txt should look something like:
2009 Dec 11 16:08:17 NO-HOSTNAME kernel: %WAAS-SYS-3-900000:1.1.1.1:49114 - 2.2.2.2:22 - opt_syn_rcv: Routing Loop detected -
Packet has our own devid. Packet dropped.
Assuming that WCCP is being handled in software on the the 7609, the counter incrementing in the output you provided would support that traffic isn't being seen symmetrically. That in and of itself shouldn't cause the connections to fail (they should just be handled as pass-through), so I suspect there may be a redirection loop at your BDC site.
Can you provide a topology diagram of your environment?
For the WCCP in software issue on the 7609, can you provide the following output from IOS:
show version
show ip wccp
show ip wccp 61 service
show ip wccp 61 detail
show ip wccp
show running-config
Thanks,
Zach
06-21-2010 12:00 PM
Zach,
Thanks for responding. We do indeed see an error in the syslog.txt file showing a routing loop error:
2010 Jun 20 10:59:26 waas-bdc kernel: %WAAS-SYS-3-900000: 192.168.128.134:18
44 - 192.168.210.217:139 - opt_syn_rcv: Routing Loop detected - Packet has our own
devid. Packet dropped.
Unfortunately I cannot post configs/topology/command output, directly to netpro due to internal security restrictions, however I can send them directly to you if you have time to take a look? I would assume from the above that we need to be lookign at the wccp redirect configuration on the router?
06-22-2010 03:33 AM
06-22-2010 09:53 AM
Thanks Zach, I have sent these along to your email address.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: