Performance Issue Suspected with Zones and Inspect Configuration - Page 2

jaesposito · ‎06-15-2011

All,

I suspect that I am experiencing performance issues related to my firewall zone configuration AND/OR the inspection being done on packets. With that in mind, I have two basic questions based on my attached configuration:

1.) In looking at my configuration, what purpose do these default firewall zones AND inspect commands have for this router, which I am using on a plain DSL connection in my home?

2.) Could any part of this configuration be responsible for slowing down some of my home devices such as my AppleTV for streaming Netflix, YouTube?

The router is a 881W and is running 12.4.24.T5. If you feel that any parts of this configuration are unnecessary and might be contributing to my performance issues, please feel free to chime in.

Thank you for the help!

James E

Maykol Rojas · ‎06-16-2011

Hi,

Not even 1 error, you say that when you do the downgrade, you dont have any more issues right?

Mike

jaesposito · ‎06-17-2011

That's right. When I downgrade to the older IOS 12.4.20.T3, the "Out-Of-Order Segment" issue disappears and Netflix and YouTube work fine from the AppleTV. What should we do next?

On a side note, when running the older 12.4.20.T3 IOS, I appear to be having an different problem dropping inspected packets when simply surfing the web from my desktop computer (192.168.1.112): "match failure with ip ident 0"

000220: *Jun 16 20:40:54.495 PCTime: %FW-6-DROP_PKT: Dropping udp session 208.46.117.189:3478 192.168.1.112:51636 due to policy match failure with ip ident 0

000221: *Jun 16 20:41:43.863 PCTime: %FW-6-DROP_PKT: Dropping udp session 208.46.117.189:3478 192.168.1.112:51636 due to policy match failure with ip ident 0

000222: *Jun 16 20:42:33.235 PCTime: %FW-6-DROP_PKT: Dropping udp session 208.46.117.189:3478 192.168.1.112:51636 due to policy match failure with ip ident 0

I see no apparent problem from my desktop. But, the above is occurring. What does "match failure with ip ident 0" mean?

James E

Maykol Rojas · ‎06-17-2011

Those seem to be late packets, weird thing is that the behavior changes between versions. They are not related to web browsing, it is a weird udp stream.

If you can go to 15 version, (you may want to read the release notes prior doing it)

http://www.cisco.com/en/US/docs/ios/15_0/release/notes/150MREQS.html

Then you can apply the command for out of order


parameter-map type ooo global
  tcp reassembly memory limit 2048 
  tcp reassembly queue length 85 
  tcp reassembly timeout 54  
  exit

Let me know how it goes.

Mike

jaesposito · ‎06-17-2011

Mike,

Thanks. Before I upgrade to version 15, can you give me a general sense of how different 15 is from 12.4? I've never used 15, but am comfortable with the 12.x IOS. A general idea would be fine.

Also, is there a real significant value to me inspecting packets that originate from the trusted inside of my network? I understand wanting to do this from traffic sourced from an untrusted, outside interface. But, I'm struggling to understand why this is useful for traffic originating from my interior devices. Assuming that my machines do not have malicious software installed, I'm struggling to see the value as it is clearly causing problems where real problems dont exist - our above thread being a perfect example.

Thank you very much for your thoughts!

James E

Maykol Rojas · ‎06-17-2011

Hi James,

What an excellent question. Basically you would like to inspect the traffic that comes from your inside network because, by default, the return traffic would be allowed. If you put just a single access list on the outside interface of your Router, you will need to allow all the responses to every query done from the inside.

With the inspection, sessions that were initiated from the inside network, the return traffic is allowed with no issues at all.

The idea of an stateful firewall is to allow those sessions from trusted sources and deny the rest.

Version 15 is not different from any other IOS version, only new commands and new features were added, but the rest is exactly the same.

If you have any questions, let me know.

Mike

jaesposito · ‎06-17-2011

Ok. I will upgrade to 15.1.3.T1 tonight, input the configuration changes and let you know if this fixes the problem. If I'm still experiencing issues, I'll share the output of the "term mon" in my next post.

Thank you for the help!

James E

Maykol Rojas · ‎06-18-2011

Hey James,

How did it go?

Mike

jaesposito · ‎06-18-2011

Ok. I did the upgrade. I had not yet entered your suggested config changes and it looks like Netflix and YouTube on the AppleTV is working. Do you still recommend that I add these lines? (This is what you said earlier)

parameter-map type ooo global
  tcp reassembly memory limit 2048 
  tcp reassembly queue length 85 
  tcp reassembly timeout 54

Also, I'm still experiencing some weirdness with INSPECT dropping some legitimate data sourced from the inside. Below are the logs. My internal IPs are as follows:

192.168.1.101 - Apple iPhone

192.168.1.102 - AppleTV

192.168.1.103 - Apple iPad

192.168.1.112 - Windows 7 Desktop

000027: *Jun 18 15:43:46.159 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000028: *Jun 18 15:44:18.315 PCTime: %FW-6-DROP_PKT: Dropping tcp session 192.168.1.103:57480 17.172.236.244:5223 on zone-pair ccp-zp-in-out class ccp-insp-traffic due to Invalid Flags with ip ident 0

000029: *Jun 18 15:44:52.183 PCTime: %FW-6-DROP_PKT: Dropping tcp session 192.168.1.103:57480 17.172.236.244:5223 on zone-pair ccp-zp-in-out class ccp-insp-traffic due to Invalid Flags with ip ident 0

000031: *Jun 18 15:46:09.951 PCTime: %FW-6-DROP_PKT: Dropping tcp session 17.155.4.14:443 192.168.1.101:55519 due to Stray Segment with ip ident 0

000033: *Jun 18 15:47:03.691 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000034: *Jun 18 15:47:52.831 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000035: *Jun 18 15:48:42.335 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000036: *Jun 18 15:49:32.379 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000037: *Jun 18 15:50:07.931 PCTime: %FW-6-DROP_PKT: Dropping tcp session 63.218.71.153:80 192.168.1.102:53608 due to Stray Segment with ip ident 0

000038: *Jun 18 15:51:10.531 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000039: *Jun 18 15:51:54.519 PCTime: %FW-6-DROP_PKT: Dropping tcp session 74.125.7.96:80 192.168.1.102:53632 due to Stray Segment with ip ident 0

000040: *Jun 18 15:52:49.275 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000041: *Jun 18 15:53:21.587 PCTime: %FW-6-DROP_PKT: Dropping tcp session 192.168.1.102:53638 74.125.7.96:80 on zone-pair ccp-zp-in-out class ccp-insp-traffic due to Stray Segment with ip ident 0

000042: *Jun 18 15:53:59.787 PCTime: %FW-6-DROP_PKT: Dropping tcp session 68.142.118.254:80 192.168.1.102:53667 due to Stray Segment with ip ident 0

000044: *Jun 18 15:56:06.307 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000046: *Jun 18 15:57:44.583 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000047: *Jun 18 15:58:15.471 PCTime: %FW-6-DROP_PKT: Dropping tcp session 17.155.4.14:443 192.168.1.103:49276 due to Stray Segment with ip ident 0

000048: *Jun 18 15:59:23.107 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000049: *Jun 18 16:00:12.279 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000050: *Jun 18 16:01:01.371 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

000051: *Jun 18 16:01:50.551 PCTime: %FW-6-DROP_PKT: Dropping udp session 69.22.151.206:3478 192.168.1.112:53722 due to policy match failure with ip ident 0

Is there any additional fine tuning that I can do to avoid some of these false positives?

Thanks for the help! I feel like we are almost there!

James E

Rama Darbha · ‎06-20-2011

James,

When you say weirdness, how is it manifesting at an application level? It looks like you're primarily using video streaming applications. Do these packet-drop syslog coincide with jitter/slowing loading/etc?

Also, during this time, is there an active connection for this traffic? You can view the output of:

show policy-map type inspect zone-pair sessions

This shows all the active connection on the ZBFW.

Unfortunately, ZBFW doesn't document all its drop reasons very well. As a result, we should start with the basics to identify the cause of the syslog.

Regards,

Rama

jaesposito · ‎06-20-2011

Rama,

I will take a closer look at what application is responsible for the traffic and advise. I will use your suggested command on comment on my findings.

At this point, my video streaming applications are no longer experiencing any jitter/slowness since the upgrade to IOS 15. So, I suspect that the remaining packets from my desktop computer that are being dropped by the INSPECT command are coming from another application on my computer.

I will investigate and advise. Thanks!

James

jaesposito · ‎06-19-2011

Maykol,

Any thoughts on my response posts?

James E

jaesposito · ‎06-20-2011

Maykol,

Any thoughts on my response posts?

James E

jaesposito · ‎06-18-2011

And more weirdness from my iPhone when watching Netflix:

000131: Jun 18 17:59:51.399 PCTime: %FW-6-DROP_PKT: Dropping tcp session 98.64.169.80:55682 204.236.229.221:443 due to Stray Segment with ip ident 0

000132: Jun 18 18:00:30.211 PCTime: %FW-6-DROP_PKT: Dropping tcp session 208.111.185.181:80 192.168.1.101:55685 on zone-pair ccp-zp-in-out class ccp-insp-traffic due to Stray Segment with ip ident 0

000133: Jun 18 18:01:10.227 PCTime: %FW-6-DROP_PKT: Dropping tcp session 208.111.185.181:80 192.168.1.101:55688 on zone-pair ccp-zp-in-out class ccp-insp-traffic due to Stray Segment with ip ident 0

000134: Jun 18 18:01:40.279 PCTime: %FW-6-DROP_PKT: Dropping tcp session 208.111.185.181:80 192.168.1.101:55691 on zone-pair ccp-zp-in-out class ccp-insp-traffic due to Stray Segment with ip ident 0

000135: Jun 18 18:02:10.287 PCTime: %FW-6-DROP_PKT: Dropping tcp session 208.111.185.181:80 192.168.1.101:55694 on zone-pair ccp-zp-in-out class ccp-insp-traffic due to Stray Segment with ip ident 0

In this case, Netflix is running and causing these logs. But, the router is dropping some TCP sessions while Netflix is running.

Maykol Rojas · ‎06-20-2011

As you can see on the logs, there all seems to be from a udp stream of port 3478, you need to verify what application is using this port, since as you can see is very consistent.

The other errors, as Rama stated are not well explained on Cisco documentation, but as far as Stray segment goes, it is documented as packets that come late (a RST when the connections is already closed etc).

It is very important to know if the application is having slowness at this point.

Mike

jaesposito · ‎06-20-2011

I will take a closer look at what application is responsible for the traffic and advise.

As it relates to "out of order" packets, the upgrade to 15.1.3.T1 eliminated the problem impacting Netflix and YouTube on the AppleTV. Keep in mind that I had no yet implemented the OOO configuration changes that you suggested. Should I do so at this point? This is what you suggested after upgrading to IOS 15:

parameter-map type ooo global

tcp reassembly memory limit 2048

tcp reassembly queue length 85

tcp reassembly timeout 54

Thanks!