Zone based firewall dropping tcp sessions.

[les_davis]

We are currently running 2811 routers for our remote locations.  We have implemented a DMVPN network and we are using a ZBF to allow split tunneling for internet connections.  We have been seeing log messages indicating that packets and sessions are being dropped.

Is there any support documentation for troubleshooting and/or a message hierachy?  Specifically if a log message indicates the session has been dropped due to a stray segment if there are other packets on the wire how will they be handled by the router and how will that action be logged?

[Maykol Rojas]

Hello Les,

The following table is going to explain you the different messages that you are going to see on your FW log

http://www.cisco.com/en/US/docs/ios/security/command/reference/sec_i2.html#wp1048937

Is the traffic that is being dropped from the inside to the DMVPN zone? What traffic is the one that is being dropped?

Let me know.

Cheers

Mike

[les_davis]

The session drops are both ways.  We are trying to figure out how to use the logs to determine if there are any "main" event that causes the other issues.

Sample router log

Dec  7 06:52:03.918 CST: %FW-6-DROP_PKT: Dropping tcp session 10.13.240.199:42905 216.52.207.65:80  due to  Stray Segment with ip ident 0
Dec  7 06:52:34.355 CST: %FW-6-DROP_PKT: Dropping tcp session 173.188.247.146:43004 216.52.207.65:80  due to  Stray Segment with ip ident 0
Dec  7 06:53:04.556 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:42999  due to  policy match failure with ip ident 0
Dec  7 06:53:34.557 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:42999  due to  policy match failure with ip ident 0
Dec  7 06:54:05.690 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:43118  due to  policy match failure with ip ident 0
Dec  7 06:54:36.071 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:43181  due to  policy match failure with ip ident 0
Dec  7 06:55:06.244 CST: %FW-6-DROP_PKT: Dropping tcp session 10.13.240.174:34634 216.52.207.65:80  due to  Stray Segment with ip ident 0
Dec  7 06:56:14.230 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:43208  due to  Stray Segment with ip ident 0
Dec  7 06:57:09.591 CST: %FW-6-DROP_PKT: Dropping tcp session 10.13.240.174:34682 216.52.207.65:80  due to  Stray Segment with ip ident 0
Dec  7 06:58:20.938 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.174:34738  due to  Stray Segment with ip ident 0
Dec  7 06:59:37.604 CST: %FW-6-DROP_PKT: Dropping tcp session 10.13.240.199:43392 216.52.207.65:80  due to  Stray Segment with ip ident 0
Dec  7 07:00:14.321 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.174:34814  due to  policy match failure with ip ident 0
Dec  7 07:00:48.398 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.174:34743  due to  policy match failure with ip ident 0
Dec  7 07:01:36.204 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.174:34815  due to  Stray Segment with ip ident 0
Dec  7 07:05:20.406 CST: %FW-6-DROP_PKT: Dropping tcp session 10.13.240.199:43506 216.52.207.65:80  due to  Stray Segment with ip ident 0
Dec  7 07:06:44.729 CST: %FW-6-DROP_PKT: Dropping tcp session 10.13.240.199:43651 216.52.207.65:80  due to  Stray Segment with ip ident 0
Dec  7 07:08:29.400 CST: %FW-6-DROP_PKT: Dropping tcp session 10.13.240.199:43687 216.52.207.65:80  due to  Stray Segment with ip ident 0
Dec  7 07:09:02.577 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:43778  due to  Stray Segment with ip ident 0
Dec  7 07:13:15.469 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:43883  due to  Stray Segment with ip ident 0
Dec  7 07:14:36.275 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:43923  due to  Stray Segment with ip ident 0
Dec  7 07:15:06.852 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:43963  due to  Stray Segment with ip ident 0
Dec  7 07:16:11.942 CST: %FW-6-DROP_PKT: Dropping tcp session 10.13.240.199:43977 216.52.207.65:80  due to  Stray Segment with ip ident 0
Dec  7 07:17:01.764 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:44006  due to  Stray Segment with ip ident 0
Dec  7 07:18:00.722 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:44016  due to  Stray Segment with ip ident 0
Dec  7 07:18:49.583 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.199:44092  due to  Stray Segment with ip ident 0
Dec  7 07:21:21.496 CST: %FW-6-DROP_PKT: Dropping tcp session 216.52.207.65:80 10.13.240.174:35156  due to  Stray Segment with ip ident 0

[Maykol Rojas]

Hello.

I see, Are you noticing problems on the final PC's? Normally this could be because a rst packet get into the firewall once the session was closed, hence it the firewall should drop it.

Let me know.

Mike

[Jitendriya Athavale · Accepted solution]

this is probably due to the fact that we see  lot of out of order packets coming to the router/firewall

i have seen this error before when there r lot of out of order packets

what code of ios r u running on router... out of order support for ZBF is avalable only from 15.0 code...

i think this is the cause, if you r ok with upgrade you can try to upgrade to 15.0 or higher code, again i am guessing this based on my experience

[les_davis]

We keep running into other issues using 15.x IOS and have not been able to move to that IOS version.  15.1 and .2 has a socket error issue keeping one of the mGRE tunnels from functioning.  15.1 has a problem with router crashing with zone base firewall when fragmentation is on the network.  We are currently regression testing 15.3T but have yet to deploy to the field.  We currently have 1600 loctions converted and don't want to add any other issues without sufficient lab testing.

We are currently running with c2800nm-advsecurityk9-mz.124-24.T1.bin.  The tcp reassembly support is resident with this IOS.  We are currently working to increase the queue depth to 64 packets from the default of 16.

There are several configruation options with the queue depth and I currently have a case requesting more details on that configuration.

The main question is what are the best practices for determining the tcp reassembly queue depth?  I don't think that just raising the value until we stop getting the queue overflow alarms is a good practice.  Besides we can't do that with 1600 remote locations with another 6000 planned.

There must be some formula/process to determine a good starting point based on circuit type and max latency.

The other questions concern the trade-offs between increasing the queue depth and memory usage.  Is there a break even point with the queue and memory usage that you can affect the router performance?  And if you increase the depth to 1024 does that start affecting latency?  How does the queue depth and the max memory used configuration items play together?

Once again what are the best practices and where can I find a configuraiton guide.

[Jitendriya Athavale]

I have been unlucky wih tcp reassembly as far this issue is concerned, but I will certainly want you to try it once.

In any case my overall take is we should always go forward unless we have no option, we u are being affected with bugs in 15.x code u can wait till it is fixed and go to 15.x code

But again I would recommend you open a tac case and have this investigated thoroughly

Regards,

Jitendriya

[iec1128759]

the minimum required version 15 512BM RAM

Greetings

[les_davis]

We do have a case open.  I wasn't getting very good information from tac so I started this thread.  We are seeing success with changing the tcp reassembly.  We have increased the queue to 64 and performance/user experience has improved.  We are also working to "certify" 15.1-3T for our production needs.

Thanks everybody for your input.

[Jitendriya Athavale]

Awesome thts great news you might want to mention the command you put and the code for the benefit of other users

Regards,

Jitendriya

[les_davis]

We are currently using 2811 routers running the c2800nm-advsecurityk9-mz.124-24.T1.bin code.

We applied 2 commands.  1 to log if we were exceeding the tcp reassembly queue buffer and another to increase the queue depth.

Both commands are global commands.

Enabling logging is a must in this type of situation.  The only way to actively find the problem and track if you need to adjust the buffer more is by turning on the logging.

Logging command

ip inspect tcp reassembly alarm on

Sample log output

%FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:4261387804 1400 bytes is out-of-order; expected seq:4261363324. Reason: TCP reassembly queue overflow - session LOCAL IP :49959 to Remote IP:80

Queue depth increase command.

ip inspect tcp reassembly queue length 64

To verify if you are having this problem there are a couple of commands to use to verify if you are receiving out of order packets and how many you may be dropping

sho ip inspect statistics

Interfaces configured for inspection 4294967290
Session creations since subsystem startup or last reset 0
Current session counts (estab/half-open/terminating) [0:0:0]
Maxever session counts (estab/half-open/terminating) [45:23:10]
Last session created 00:00:10
Last statistic reset never
Last session creation rate 15
Maxever session creation rate 241
Last half-open session total 0
TCP reassembly statistics
  received 2846 packets out-of-order; dropped 815
  peak memory usage 94 KB; current usage: 0 KB
  peak queue length 16

sho ip inspect tech-support will also give you the same information.