I'm experiencing a rather strange issue that has me stumped.
We are running an FWSM on a 6509 with a SUP720. Firmware 3.2(18), in MultiContext Routed Mode, with shared MSFC.
Everything runs fine on this baby most of them time, however occasionally without warning and with no specific pattern the Primary node will fail (as in completely stop responding) and the secondary will takover as active.
Two get the primary up agian, I reset the hw-module and then no failover active on the secondary to return the primary as active. However, after this event, I start to experience strange issues with connectivity. Certain TCP src dst combinations will just not work. Take the following example:
A TCP/1433 connection from Inside IP: 10.3.3.196 to outside IP: 10.252.20.63, logs look like this:
2012-08-07 13:43:13:0868 + 13435 2012-08-07 13:43:09 Local5.Info 192.168.2.7 Aug 07 2012 11:31:19: %FWSM-6-302013: Built outbound TCP connection 145674175523995444 for servers:10.3.3.196/64112 (10.3.3.196/64112) to outside:10.252.20.63/1433 (10.252.20.63/1433)
2012-08-07 13:43:13:0868 + 13436 2012-08-07 13:43:09 Local5.Info 192.168.2.7 Aug 07 2012 11:31:19: %FWSM-6-302014: Teardown TCP connection 145674175523995444 for servers:10.3.3.196/64112 to outside:10.252.20.63/1433 duration 0:00:00 bytes 128 TCP Reset-O
2012-08-07 13:43:13:0868 + 13526 2012-08-07 13:43:09 Local5.Info 192.168.2.7 Aug 07 2012 11:31:19: %FWSM-6-106028: Deny TCP (Connection marked for Deletion) from 10.3.3.196/64112 to 10.252.20.63/1433 flags SYN on interface servers
2012-08-07 13:43:13:0875 + 13670 2012-08-07 13:43:10 Local5.Info 192.168.2.7 Aug 07 2012 11:31:20: %FWSM-6-302013: Built outbound TCP connection 145674175523995445 for servers:10.3.3.196/64112 (10.3.3.196/64112) to outside:10.252.20.63/1433 (10.252.20.63/1433)
2012-08-07 13:43:13:0875 + 13671 2012-08-07 13:43:10 Local5.Info 192.168.2.7 Aug 07 2012 11:31:20: %FWSM-6-302014: Teardown TCP connection 145674175523995445 for servers:10.3.3.196/64112 to outside:10.252.20.63/1433 duration 0:00:00 bytes 124 TCP Reset-O
However I create a specific ACL on the upstream routers interface, to see if I get any matches and the traffic is not even leaving the 6509. I can however ping the remote device without any issues. And I can confirm that the xlate has been built.
This connection was working fine prior to the crash, and the ACL rules are correct and do allow the connection on both the local FWSM and the remote firewall.
Currently my only resolution is to reboot the FWSM on both nodes at the same time so that we have a complete fresh start. This is not ideal!
Anyone know of issues like this? Any suggestions for workarounds or perhaps ways to troubleshoot the reason for the crash?
Thanks!
Craig
