Performance Issue Suspected with Zones and Inspect Configuration

[jaesposito]

All,

I suspect that I am experiencing performance issues related to my firewall zone configuration AND/OR the inspection being done on packets.  With that in mind, I have two basic questions based on my attached configuration:

1.)  In looking at my configuration, what purpose do these default firewall zones AND inspect commands have for this router, which I am using on a plain DSL connection in my home?

2.)  Could any part of this configuration be responsible for slowing down some of my home devices such as my AppleTV for streaming Netflix, YouTube?

The router is a 881W and is running 12.4.24.T5.  If you feel that any parts of this configuration are unnecessary and might be contributing to my performance issues, please feel free to chime in.

Thank you for the help!

James E

[Maykol Rojas]

Hi James,

Sorry for the late respond, yes I did and I was able to see the debugs, unfortunately nothing that can be easily be pulled out.

At this point, would be it be so bad for you to open a TAC case? I think you can open it from here (supporforums) and automatically attach all the outputs.

Mike

[jaesposito]

Attached are additional examples of TCP traffic being dropped by INSPECT for legitimate, internal, trusted devices trying to access the Internet.  I am now going to open a Cisco TAC case under my SmartNet contract.

James E

[Maykol Rojas]

Once you have it, let me know, I would like to keep an eye on it.

Mike

[scottaobe]

Hi James,

I was reading this thread with great interest as I have been experiencing the same / very similar issues. Did you ever find out anything? I have had a case open regarding this for a few weeks with no resolution yet. We have a Cisco 1921 running Version 15.0(1)M3. One difference is I am not using the Zone Based Firewall. I did see another post somewhere (sorry but I have looked at so many over the past month I have lost track - but I need to find that one again)  but in essence upgrading to a newer version than 15.0 resolved the issuse for the other party but my tech support agent has said not to just do that without some justification. I'm about ready to give it a try anyway at this point.

Thanks, Scott

[jaesposito]

Scott,

The upgrade to 15.1.3.T1  eliminated the problem impacting Netflix and YouTube on the AppleTV.   However, I'm still left the packets being dropped by ZBFW as a result of the following vague explanations:

due to  Stray Segment with ip ident 0

due to  RST inside current window with ip ident 0

due to SYN inside current window with ip ident 0

due to Retransmitted Segment with Invalid Flags with ip ident 0

All of these dropped packets occurred when surfing the web from my Windows desktop computer.  I'm not experiencing any perceptible performance issues.  I just find it awfully strange that the firewall would be dropping these packets from obvious legitimate use.

I have a Cisco TAC case opened and a CCIE is assigned to it.  He is essentially telling me that somewhere the TCP/HTTP protocols are not being totally followed according to their RFC.  I'm not deep enough of an expert to say whether they are or not.  My contention has been that a simple ACL firewall with NAT wouldnt be dropping these packets since the related sessions are trusted as they were initiated as internal connections.  So, I'm confused as to why the ZBFW would be so aggressive with its INSPECTion and dropping of said packets.  While I "get it" that if they dont follow the RFC that they are suspicious, but that means that a number of widely known websites, such as Yahoo and Google have responses that dont follow the TCP/HTTP RFCs, which I find possible, but unlikely.

I'm sorry I couldnt be of more help, but I'm afraid I'm going to have to live with this as is.

James

[scottaobe]

Hi James,

Thanks for the reply - your description hits the nail on the head and I am basically seeing the same exact drops you describe. I have checked the IP's that are involved this (at first it looked like some kind of Denial Of Service attack) and most of the time it's from large content delivery  providers like Akamai, Limelight, Google. These are all  connections that were originated from our network although I also see this with connections over the VPN to our national office

I have been having issues with RST from a remote server over VPN and it seems to have something to do with the TCP inspection but I haven't been able to pin it down. The application developers say the packets leave the local server but never arrive at the destination and so the remote server sends a RST. Other than that (which is a big problem) there doesn't seem to be perceptible impact - just as you describe.

I'm using the regular firewall instead of the ZBFW so I don't think it's  specific to that. I still have an open case as well and have just  recently separated VPN traffic and Internet traffic to two different  routers and ISP connections.

I don't have the same issue with an 1811 running Version 12.4(15)T14. It's as though the 1921 router gets bogged down inspecting which seems odd that a 6Mb connection could overwhelm the 1921 that the 1811 handles ok - but this was on another connection with a different ISP. I wonder if someones caching or traffic management is getting in the way somewhere.

I have recently added some CPU and bandwidth monitoring as well as Wireshark monitoring to try and get some visibility into whats going on.

Sorry if I've covered the same ground but sometimes it's a little bit of info that solve the problem.

Thanks again for replying, I'll post back if I find a resolution but I'm living with this as well for now.

Scott

[jaesposito]

Very interesting.  I'm glad I'm not the only one experiencing this weirdness.  What makes it even more strange is that you are experiencing it with the classic firewall as opposed to ZBFW.

I was originally running 12.2.24.T2 with no issues.  When I ugpraded to 12.4.24.T5, I started experiencing the issues with dropped data and the inability to stream using YouTube, Netflix etc.  Once I upgraded to the 15.1 code (15.1.3.T1), then the issues with AppleTV, Netflix and YouTube disappeared.  But those strange, dropped packets never stopped although I have no discernible issues with applications.

Some of the websites that I've encountered the aforementioned drops include Yahoo Mail, Google and some of their associated advertisers including Akamai.  I guess its possible that those guys have some systems that are not following the RFP to the exact spec.  But, I find that hard to believe as I'm only using their sites for standard functions, such as Mail, Search, receiving banner ads etc.

So, I'm a bit at a loss.  The CCIE on my case has me headed down the path of testing if there is any performance difference on one of these sites when comparing ZBFW and ACLs.  But, I'm not sure what that proves aside from a possible performance issue.  I just dont see why the dropped packets with debug on dont explain the instances themselves.

Let me know how your TAC case goes and we'll trade notes.  If our cases are truly the same issue, then maybe we can help move this thing forward to resolution one way or another.

James E

[Jonathan Bekkers]

I too have a customer experiencing these issues. He runs an Apple-based business and I too am seeing malformed TCP packets running across the link constantly. I too am not using ZBF but and sticking with old-style ACLS with IP INSPECT. Same stray segment issues you guys have been having.... "stray segments" and "SYN inside current window"

Upgraded Cisco 887 to use 15.1.4M and the customer reports it seems better but I am still seeing errors in the logs same as above.

I will update tomorrow after the customer can do some proper testing.

[jubetz]

Hi Jonathan,

I'm working with James to find out if these drops are legitmate or not.  So far, we haven't been able to find an instance where the drops were because of a perfectly legimate packet that should have been passed.  Alll evendence presented thus far is circumstantial and may be perfectly normal.

A difference between your config and what's being brought up in this thread in that we *ARE* using ZBFW and *NOT* ACLS and IP inspect.  This is different code and we can't say that an issue with one style (ZBFW or CBAC/ip inspect) will also be present in the other.

After you can do some proper testing, you should open a TAC case to get it looked at.  The TAC will need captures taken on both sides of the router along with your drop logs so that we can identify if the packet should have been dropped or not.

Simply observing drops doesnt' necessarily indicate a problem.  A firewall is supposed to drop packets.  We need to identify the packets that are dropped (with captures on both sides of router) and then we need to analyze whether or not the drop was for a legimate reason.

With regards,

-jb

[Jonathan Bekkers]

Forgot I posted here... better late then never. I still remember this issue though. In my case it was resolved through replacing the unit itself. Turned out to be a hardware issue. Thought I'd post this just in case it helps anyone else. Same config, same IOS different device... no issue.