cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6027
Views
0
Helpful
17
Replies

Downloads stall with ip inspect enabled

TravisTauzier
Level 1
Level 1

I've been trying to figure this out for some time now.  I have a network setup with a couple of users, as well as a few servers.  I'm using a 3640 as my border router that is connected to a 3750 with L3 routing enabled.  I am using the IOS firewall in the 3640 and am having problems with downloads and connections in general on the LAN.  Downloads will start at fast speeds (~1MB/s) but after a short time, it will begin slowing to a crawl or idle.  I have disabled my ip inspect rules and found that the downloads will function at full speed for the entire download.

I've looked over the router configs several times, and I don't see anything unusual.  Is there anything I should be looking for?  I would like to think that this router can handle the traffic of about 5 computers.  If I'm wrong please let me know.

TravisT

17 Replies 17

wzhang
Cisco Employee
Cisco Employee

Hi,

With application performance issues related to the firewall, one of the first things you want to look for is any packet drops by the firewall feature. You can enable the command "ip inspect log drop-pkt" to monitor such drops and the drop reason. I would start from that. Hope this helps.

Thanks,

Wen

wzhang,

Thanks for the reply.  I've taken a look at that and began a ~900MB download.  I see about 6 log entries of dropped packets from the time the download starts until the time it stalls.  Again, turning the ip inspect rule off results in faster downloads that don't stall.

My access lists include a static entry with a:

permit tcp any any established

and the ip inspect rule that is inspecting the following protocols:

ip inspect log drop-pkt
ip inspect max-incomplete high 20000000
ip inspect max-incomplete low 20000000
ip inspect one-minute high 10000000
ip inspect one-minute low 10000000
ip inspect udp idle-time 1800
ip inspect dns-timeout 7
ip inspect tcp idle-time 14400
ip inspect tcp max-incomplete host 100000 block-time 0
ip inspect name Firewall http
ip inspect name Firewall https
ip inspect name Firewall esmtp
ip inspect name Firewall dns
ip inspect name Firewall tcp
ip inspect name Firewall udp
ip inspect name Firewall isakmp
ip inspect name Firewall ntp
ip inspect name Firewall icmp

I typically get close to 1MB/s downloads with the ip inspect rule turned off.  When I turn it on, my downloads typically start at around 800KB/s and will drop to almost nothing within a minute or so.  If I cancel/resume the download it will begin fast again and then stall.  I can continue to do this until the download completes, but it is frustrating.

I saw another post with someone having the same problem and they found that using CBAC with Windows 7 causes this, but did not say how to fix this or any more info.  Can't find any more info supporting this.  All PC's are Windows 7/Server 2008.  IIRC, my server 2k3 machines download fine but I'd have to verify.

Hi,

So what does the drop log report exactly? It should include a drop reason, and that'd be the first clue on what it is that the firewall doesn't like. Also, the drop messages are rate limited, so there's a good chance you are taking more drops than the 6 that are logged.

Thanks,

Wen

The reason shown is "Out-Of-Order Segment".  Tried googling but didn't find any conclusive info on where to go next.

Travis

Hi,

What version of IOS are you running? In versions earlier than 12.4(11)T, the firewall will drop out of order TCP packets in the inspection engine. You can confirm if the TCP packets are indeed out of order by taking a sniffer trace of the flow in question. We added out of order packet buffering support in 12.4(11)T, and it can be used to mitigate this sort of problems. For details, take a look at:

http://www.cisco.com/en/US/partner/docs/ios/12_4t/12_4t11/ht_ooop.html

Thanks,

Wen

Wen,

I haven't looked at the link you provided yet, but I'm running c3640-ik9o3s-mz.124-23.bin as my image.  If you know off-hand if that feature is enabled in this IOS, I'd be interested in knowing.  Otherwise, I'll dig through the feature navigator and bug reports to see if I can track it down.  If not, what is a good image to run that has the same capabilities as the feature set I'm running now?

Travis

Hi,

No 12.4(23) mainline IOS does not have this feature. Unfortunately I don't beieve the 3640 platforms are supported in the 12.4T IOS train since it has already been End of Life'ed:

http://www.cisco.com/en/US/partner/products/hw/routers/ps274/prod_eol_notice09186a008032d840.html

Thanks,

Wen

Because of these problems along with the 3640s being EOL'd, do you have any recommendations of how to remedy this problem (using this hardware)?  I'm ok with using a solution other than CBAC, but having downloads stall and not being able to utilize the full bandwidth provided by my ISP is not acceptable.  If you have any options worth looking into (ZBF, reflexive ACLs, etc), I'd apprecate any recommendations.

Just noticed something strange.  I tried downloading a large (1GB) file from a website.  They offered three different port options to download the file from (80, 81, 8080).  First download was began from port 80.  Started fast (1MB/s) but ended up averaging somewhere in the range of 10KB/s download rate.  At the same time, I began the same download from port 81.  This download averaged right around 600KB/s.  Finally, port 8080 averaged about 500KB/s.  At this point, I had all three downloads running at the same time.  I think the last (8080) was lower because my bandwidth was maxed.

What would cause this on just port 80?

So this appears to still be acting up even after the changes in configuration.  I'm having a very hard time troubleshooting this.

I have bypassed my router and connected my computer directly to my cable modem.  I have download speeds in the neighborhood of 25MB/s down, and 8MB/s up as shown at speedtest.net.

Just prior to bypassing it, I could not even pull the page up completely at speedtest.net and some pages would pull correctly and some wouldn't.  I checked the CPU utilization during the problem - it was reported at 13% last 5 seconds/6% last 5 mintues.  Not sure but this is unacceptable!

I'll be glad to post my configs up if someone cares to take a look.  Thanks in advance.

TravisT

Hi,

As a first step, I think you really want to better understand why the Firewall is dropping OoO packets. Are the packets really arriving on the FW out of order? If so, the application performance will be severely impacted due to TCP's throttling mechanism if the Firewall has no buffering/re-ording capabilities. In theory, most of the modern TCP stack would handle OoO packets nicely with SACK support, and this could very well explain the big difference in performance with and without the firewall in the middle. To confirm if the packets are indeed arriving out of order, you can take a sniffer trace in the direction of the file transfer and analyze the TCP flow.

Thanks,

Wen

I'll be glad to give that a shot, as well as better understand what is happening there.

The one question I have is that I am not inspecting tcp traffic via the IP inspect rule.  I currently have updated the rule only to inspect dns, esmtp and ntp.  I have an inbound ACL setup permitting tcp established.  Would using the established command have the same problem with OoO packet, or is this just a problem with CBAC (if the established command isn't a form of CBAC)?

Hi,

You are right, the OoO packet inspection is only enforced by TCP stateful inspection (ip inspect name fw tcp). If you are permitting tcp established (allow all TCP packets with the ACK bit set) with an ACL, then you shouldn't have this problem. So I assume you no longer see any drops in the firewall drop-log, even though you are still experiencing slow application performance?

Thanks,

Wen

That is correct, and that is why I'm having a hard time finding this problem.  I don't completely understand the inter-workings of the stateful inspection, but the ACLs that I have setup are what I consider pretty basic.  Not sure why I'm seeing so many intermittent problems.

Another point to bring up is that occasionally I have seen very high cpu utilization (60-80%), but it is sporatic and can't seem to pinpoint the cause.  Seems to be interrupt related though.  Also, while I have experienced this problem during reduced application performance (very occasionally), it is not always a symptom.