07-24-2013 10:01 PM - edited 03-11-2019 07:16 PM
Hi, Cisco 7201, IOS:
Cisco IOS Software, 7200 Software (C7200P-ADVENTERPRISEK9-M), Version 15.0(1)M8, RELEASE SOFTWARE (fc1)
I have implemented zone based firewall with LAN 2 INET policy like:
class-map type inspect match-any CLASS_LAN2INET
match protocol http
match protocol https
match protocol tcp
match protocol udp
match protocol tcp
policy-map type inspect LAN2INET
class type inspect CLASS_LAN2INET
inspect
class class-default
pass
!
many downloads gets stalled, slowed down, sky.fm radio plays with pauses, youtube page get loaded but videos not played.
some (some!) dowloads proceed without problems, like http://ftp.freebsd.org
web pages gets opened mostly without problem
First i blamed ISP, they checked their side and didn't find any problems.
Then is disabled firewall (should have to try it first) and everything started to work like a charm.
Then i narrowed down to removing "match protocol http".
If not use "match protocol http", everything works ok.
Is it a bug, is there a fix? any options to tweak "http"?
07-24-2013 11:36 PM
Hello Utair,
Are you performing any kind of DPI for the HTTP protocol?
If not you can use the following:
ip access-list extended Julio
permit tcp any any eq 80
permit tcp any any eq 443
class-map type inspect Julio
match access-group name Julio
class-map type inspect CLASS_LAN2INET
no match protocol HTTP
no match protocol HTTPS
exit
policy-map type inspect LAN2INET
no class CLASS_LAN2INET
class Julio
inspect
class CLASS_LAN2INET
inspect
That way we are still going to inspect traffic but for the HTTP traffic will be inspected as a layer 4 protocol not layer 7.
Try it and let me know,
For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/
Cheers,
Julio Carvajal Segura
07-25-2013 10:37 PM
jcarvaja wrote:
permit tcp any any eq 443
...
no match protocol HTTPS
I wonder, how Cisco inspects HTTPS on L7? It's encrypted, after all..
ip access-list extended Julio
permit tcp any any eq 80
class-map type inspect Julio
match access-group name Julio
class-map type inspect CLASS_LAN2INET
no match protocol HTTP
Try it and let me know,
I came to this workaround myself, but it's "workaround", not a fix.
Just tried on another 2951 router with 15.1(4)M6 IOS. It has the same issue - youtube does not play videos.
How no one noticed such bug?
PS:
when starting youtube video, it spamming with:
Jul 26 11:13:24: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998688742 1500 bytes is out-of-order; expected seq:1998647862. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT
surgut-inet-2951#
Jul 26 11:13:25: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1993449876 1500 bytes is out-of-order; expected seq:1993385636. Reason: TCP reassembly queue overflow - session 10.10.10.128:55548 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT
surgut-inet-2951#
Jul 26 11:13:26: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1993451336 1500 bytes is out-of-order; expected seq:1993387096. Reason: TCP reassembly queue overflow - session 10.10.10.128:55548 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT
surgut-inet-2951#
Jul 26 11:13:29: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998717942 1500 bytes is out-of-order; expected seq:1998653702. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT
surgut-inet-2951#
Jul 26 11:13:33: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998719402 1500 bytes is out-of-order; expected seq:1998655162. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT
surgut-inet-2951#
Jul 26 11:13:42: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998720862 1500 bytes is out-of-order; expected seq:1998656622. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT
surgut-inet-2951#
Jul 26 11:13:52: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998722322 1500 bytes is out-of-order; expected seq:1998658082. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT
parameter-map has default settings, there is only one user on this router (fresh from the box)
07-26-2013 09:48 AM
Hello,
I am going to provide you a solution again not a workaround:
1-
came to this workaround myself, but it's "workaround", not a fix.
Why do you think this is a work-around. I mean this is the fix!
Why would you inspect a protocol at layer 7 if you are not doing any kind of DPI, it just does not make sense.
That's why you will need to do it at layer 4 , to make the checks faster or live with the fact the DPI engine will look and inspect the packet altough you do not have any policies for that......
2-Check the logs you are getting man, read them and tell me if they are expected on a network:
out-of-order; expected seq:1998655162. Reason: TCP reassembly queue overflow - !!!
I can see a lot of out of order packets (OOO) this is the down side of you network, no need to blame the ZBFW.
Now for this you have 2 solutions:
1- The real solution: Remove the OOO packets of the network
2- Implement the OOO support that means configure the parameter-map type out-of-order so we can hold this OOO packets in a queue (the default is of 5 mileseconds to receive all of the segments in ooo and we can hold up to 16 OOO packets)
Note: Remember to rate all of my posts, I definetly take my time to write this replies
For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/
Cheers,
Julio Carvajal Segura
07-28-2013 08:48 PM
I am going to provide you a solution again not a workaround:
1-
came to this workaround myself, but it's "workaround", not a fix.
Why do you think this is a work-around. I mean this is the fix!
Why would you inspect a protocol at layer 7 if you are not doing any kind of DPI, it just does not make sense.
That's why you will need to do it at layer 4 , to make the checks faster or live with the fact the DPI engine will look and inspect the packet altough you do not have any policies for that......
This is a basic feature, and it should work out of the box. Youtube isn't a rare service. I might want to filter http access by URIs later and how i will guarantee normal operation to users?
1- The real solution: Remove the OOO packets of the network
They are not out of order, there is queue shortage.
2- Implement the OOO support that means configure the parameter-map type out-of-order so we can hold this OOO packets in a queue (the default is of 5 mileseconds to receive all of the segments in ooo and we can hold up to 16 OOO packets)
What reasonable parameters are?
07-28-2013 10:22 PM
Hello Utair,
So you basically say that I am wrong with the information I have provided before,
This is a basic feature, and it should work out of the box. Youtube isn't a rare service. I might want to filter http access by URIs later and how i will guarantee normal operation to users?
They are not out of order, there is queue shortage.
Well.. Then I wish you the best with this deployment, maybe someone else you really trust will answer here,
For Networking Posts check my blog at
http://www.laguiadelnetworking.com/category/english/
Cheers,
Julio Carvajal
07-29-2013 01:11 AM
So you basically say that I am wrong with the information I have provided before
I say, that "match protocol http", should match HTTP protocol and not cause downloads to stall and youtube not working.
Out of the box, without fine-tuning.
Back on topic:
I tuned parameter-map type ooo global, maxed up queue length am set memory limit up to 128M. Everything started to work.
07-29-2013 10:30 AM
Hello Utair,
Back on topic:
I tuned parameter-map type ooo global, maxed up queue length am set memory limit up to 128M. Everything started to work.
So you basically increase the software queue for OOO packets as I said with the parameter-map tuning.(that should not exist) and increase the memory for that traffic. I am glad that it works but keep an eye on the network as basically your solution promotes latency (It´s just a work-around while you fix the real issue the OOO)
For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/
Cheers,
Julio Carvajal Segura
07-29-2013 08:41 PM
Well, after a while i ran into:
Jul 21 09:44:21 10.96.32.37 13665: tmn-pleh-7201: Jul 21 09:44:20: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:608968433 1500 bytes is out-of-order; expected seq:608842873. Reason: Unable to get a buffer to hold the packet - session x.x.x.x:59934 to 46.61.155.211:80 on zone-pair zbf_LAN2INET class zbfc_LAN2INET
Increased memory limit to 512M. Messages stopped. I wonder for how long.
How do i get memory usage statistics for inspection engine?
07-29-2013 10:14 PM
Hello Utair,
Do you understand that those are OOO packets right? Those are not normal nor expected packets?
You are basically wasting resources of the router to support invalid packets.
How do i get memory usage statistics for inspection engine?
There is a limit (you should see it on your platafform if you hit the question mark) I dont think we can increase that as the router need memory for it's other tasks
Any other question? Otherwise mark the question as answered
For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/
Cheers,
Julio Carvajal Segura
07-29-2013 10:21 PM
Do you understand that those are OOO packets right? Those are not normal nor expected packets?
Yes, but who do i complain to? Google? UN?
Increasing memory limit helped for couple hours, then situation repeated.
Removed "match protocol http" completely.
07-29-2013 10:26 PM
Doesn't "match protocol tcp" track for OOO?
07-30-2013 10:10 AM
Hello Utair,
Yes, but who do i complain to? Google? UN?
In this case the OOO packets come from the inside of the network, so the problem it's on the LAN side
Doesn't "match protocol tcp" track for OOO?
It does as well but here is the difference:
It's well known that the HTTP layer 7 inspection of the ZBFW has limitations with the OOO handeling this is why I recommended to use a match protocol TCP as long as possible.
If you then need to use DPI for the HTTP protocol then you should fix the OOO packets.
I wrote a post regarding a ZBFW enviroment on my blog www.laguiadelnetworking.com, You are more than welcome to visit it.
Cheers,
Julio Carvajal Segura
07-30-2013 07:57 PM
Yes, but who do i complain to? Google? UN?
In this case the OOO packets come from the inside of the network, so the problem it's on the LAN side
It is session direction LAN->INET, but how do you know it's packets from LAN, which caused ooo? Packet size was 1500, which is surely packet with data, not just empty ACK, which is usually sent from client to server on download.
The problem appears on long, downloads and multimedia streams, like youtube or internet radio.
Testing environment was very simple, INET-router-Client PC with fresh Win7 setup with all latest service packs and fixes.
07-30-2013 08:13 PM
Because the firewall is lettting you know that......
For Networking Posts check my blog at http://www.laguiadelnetworking.com
Cheers,
Julio Carvajal Segura
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide