FWSM 3.2(3) - Intermittent Network Connectivity After Upgrade

[tim.riegert]

Has anyone else upgraded their FWSM to 3.2(3)? It was just released on Monday, and we upgraded this morning because of the FWSM ACL vulnerability (http://www.cisco.com/en/US/products/products_security_advisory09186a00808dda61.shtml). Unfortunately, it caused intermittent network connectivity issues that were amazingly resolved when we back-rev'd to 3.2(2) -- even when the active fwsm was back to the 3.2(2) version, we had the problem, so it appears that the standby fwsm running 3.2(3) was causing issues. We can't really work with TAC to troubleshoot right now because of the downtime that is causes.

Anyone else had any issues? This is a little frustrating when we try to workaround a vulnerability by following a Cisco recommendation (upgrade) and it causes downtime.

[cpoetzel]

I am seeing the same type of behavior. i upgrade d our testbed fwsm running 3.2(2) to 3.2(3) and immediately saw strange connectivity issues to my management interface. I am not passing real traffic through this device at this time. I can no longer ssh into the fwsm after the upgrade. From the fwsm, my tacacs+ servers are unreachable. Though they work with every other fwsm, switch, and router i have, so i don't think the problem lies there.

anyone here anything on this?

[pweichmann]

What is the current situation?

Was there a solution and could it be upgraded with no connection issues?

We want to upgrade to 3.2(4) from 3.1(1.

[tim.riegert]

We attempted the upgrade to 3.2(3) one more time, but we were unable to resolve the problem with TAC. We still saw strange behavior where some of our servers were not responding after the upgrade.

It just so happened that the day that we were troubleshooting, that Cisco was about to release the vulnerablity with 3.2(3) where the FWSM may reload. They advised us to not upgrade to 3.2(3) anymore, and to wait until 3.2(4) arrived.

However, we are going to hold off until the next release (after 3.2.4) because we want the fix for the TCP out-of-order bug as well -- and that fix is not included in the production release of 3.2(4).

We are still running 3.2(2) at this time. Fortunately, we have not run into the bug with 3.2(2) where the FWSM may not recognize ACEs after updating an ACL, so we've been pretty stable with this release -- with exception to the slow throughput of some TCP applications caused by the TCP out-of-order bug. :-P

HTH

[cpoetzel]

we are still running 3.2(2) in production as well with no plans to upgrade to anytime soon. it has been stable sand shows no signs of the problems that i saw in 3.2(3).

chris poetzel

[pweichmann]

Thanks chris.

I just wanted to ask you as well, if you were able to migrate without any special magic from 3.1(x) to 3.2(2) or were there any things that needed meddling. Configuration etc.

What in your opinion were the best features of 3.2?

Regards,

Patrick

[pweichmann]

Thanks Tim.

When you upgraded from 3.1(x) to 3.2(x) did you have to do something out of the ordinary or was it as described in the configuration guide for active/standby?

It's my first upgrade and I don't know yet if a downgrade is possible between 3.2 and 3.1 and if the session synchronization still works when I want to switch back from the standby to the active with the new version.

Kind regards,

Patrick

[tim.riegert]

The upgrade was very smooth when I upgraded (went from 3.1.4 to 3.2.2) -- I just followed the directions step by step. Reverting back to 3.1.x shouldnt be a problem as long as you dont configure any of the new features that exist in the 3.2 train (for instance, if you configure MSRPC application inspection, if you backrev to a 3.1 version it will not be supported).

MSRPC inpsection was the main reason for us upgrading to the 3.2 train (so we could secure access to our domain controllers).

Take care,

-Tim

[tim.riegert]

Oh, right forgot about the failover - you will want to put the new image on both FWSMs, reboot the primary, and then reboot he secondary about 15 seconds later. You cant have two FWs running with diff main version numbers. Consequently, youll need a little 5 min window of downtime for both to come up.

For instance:

you can upgrade 3.1.1 to 3.1.3 without downtime, but you cant upgrade 3.1.1 to 3.2.1 without rebooting both firewalls at the same time. They can both become active bc they wont recognize the failover partner.

HTH

Tim

[pweichmann]

Thanks.

BTW: What is the issue with the tcp out-of-order bug. I have not seen something in the release notes as being open caveats or in a security advisory??

[tim.riegert]

CSCsl10667 Bug Details

FWSM introduces out of order packets into TCP connections

Symptom:

Under rare circumstances the FWSM might re-order packets in a TCP stream. This might cause high speed TCP transfers through the FWSM to report decreased throughput.

Conditions:

The packets must be traversing a FWSM. The faster the packets traverse the firewall (ie: The shorter the inter-packet gap) the more likely the FWSM is to re-order packets in the stream.

Workaround:

In version 3.2 of the FWSM, setting the connection for tcp-state-bypass seems to help the throughput. However, this workaround negates the basic security checks of the FWSM and probably is not appropriate for most situations.

[pweichmann]

Thanks.

I just looked up if I find something about this bug and it says on the bug tracking site that it is fixed. I doesn't say where and when?!

FWSM introduces out of order packets into TCP connections

Symptom:

Under rare circumstances the FWSM might re-order packets in a TCP stream. This might cause high speed TCP transfers through the FWSM to report decreased throughput.

Conditions:

The packets must be traversing a FWSM. The faster the packets traverse the firewall (ie: The shorter the inter-packet gap) the more likely the FWSM is to re-order packets in the stream.

Workaround:

In version 3.2 of the FWSM, setting the connection for tcp-state-bypass seems to help the throughput. However, this workaround negates the basic security checks of the FWSM and probably is not appropriate for most situations.

Status

Fixed

Severity

2

Last Modified

In Last month

Product

Cisco IOS software

Technology

1st Found-In

3.2(2.2)

[tim.riegert]

Yeah - I opened a TAC case when 3.2(4) was released to find out more about it. The fix was not tested pervasively, so it is not included in the production release of 3.2(4). You can enable the beta fix in 3.2(4) with a hidden command (to test it), but it is not recommended in a production environment.

The TAC engineer did say that it is pretty rare for the bug to affect users. If the fix ends up being successful in extended field testing, they will include it in their next release. We did not really want to perform the testing since we have not seen extreme performance degradation. We only noticed the TCP reordering when we were tracking down other performance issues (the other issue turned out to be unrelated to the FWSM - it was due to asic oversubscription on 6148-GE-TX linecards).