Issues with FWSM and 6509: heartbeat drop?

j-pickering · ‎08-15-2012

Hi,

We have a couple of 6509s with the firewall module (WS-SVC-FWM-1) in them. There is a redundant link between the switches and the firewalls are set for active/passive. They're set up like this with 2 Nexus 5ks behind them:

|\ /|

|/ \|

Last night the network failed behind the 6500s. The only errors we received on the 6500:

Aug 15 19:16:28.002 AWST: %SEC-6-IPACCESSLOGDP: list restrict-UPS-SC denied icmp 172.16.200.1 -> 172.16.40.179 (0/0), 1 packet

Aug 15 19:16:49.510 AWST: %SEC-6-IPACCESSLOGDP: list restrict-UPS-SC denied icmp 172.16.200.22 -> 172.16.40.29 (0/0), 1 packet

Aug 15 19:23:08.540 AWST: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks

Aug 15 19:23:08.652 AWST: %SVCLC-SW2_STBY-5-FWTRUNK: Firewalled VLANs configured on trunks

Aug 15 20:00:58.425 AWST: %SEC-6-IPACCESSLOGDP: list restrict-UPS-SC denied icmp 172.16.205.2 -> 172.16.40.16 (0/0), 1 packet

Aug 15 20:44:01.010 AWST: %SEC-6-IPACCESSLOGDP: list restrict-UPS-SC denied icmp 172.16.200.10 -> 172.16.40.192 (0/0), 1 packet

And on the firewall modules:

/InternalFW/act# sh logg

Syslog logging: enabled

Facility: 22

Timestamp logging: enabled

Name logging: enabled

Standby logging: disabled

Deny Conn when Queue Full: disabled

Console logging: level errors, class auth, 1514 messages logged

Monitor logging: level emergencies, 367 messages logged

Buffer logging: level errors, 1514 messages logged

Trap logging: level informational, facility 22, 73537453 messages logged

Logging to Outside Tftptest errors: 418252 dropped: 72682199

History logging: level warnings, 369290 messages logged

Device ID: disabled

Mail logging: disabled

ASDM logging: level notifications, 370960 messages logged

on interface Shield-B2C

Aug 15 2012 22:26:36: %FWSM-1-105008: (Primary) Testing Interface CIM-Inside

Aug 15 2012 22:26:36: %FWSM-1-105008: (Primary) Testing Interface Shield-B2C

Aug 15 2012 22:26:37: %FWSM-1-105009: (Primary) Testing on interface CIM-Inside Passed

Aug 15 2012 22:26:42: %FWSM-1-105009: (Primary) Testing on interface Shield-B2C Passed

Aug 15 2012 22:26:51: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface Shield-Inside

Aug 15 2012 22:26:51: %FWSM-1-105008: (Primary) Testing Interface Shield-Inside

Aug 15 2012 22:26:51: %FWSM-1-105009: (Primary) Testing on interface Shield-Inside Passed

Aug 15 2012 22:27:51: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface BWS-Inside

Aug 15 2012 22:27:51: %FWSM-1-105008: (Primary) Testing Interface BWS-Inside

Aug 15 2012 22:27:52: %FWSM-1-105009: (Primary) Testing on interface BWS-Inside Passed

Aug 15 2012 22:28:06: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface CIM-Inside

Aug 15 2012 22:28:06: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface Shield-B2C

Aug 15 2012 22:28:06: %FWSM-1-105008: (Primary) Testing Interface CIM-Inside

Aug 15 2012 22:28:06: %FWSM-1-105008: (Primary) Testing Interface Shield-B2C

Aug 15 2012 22:28:07: %FWSM-1-105009: (Primary) Testing on interface Shield-B2C Passed

Aug 15 2012 22:28:09: %FWSM-1-105009: (Primary) Testing on interface CIM-Inside Passed

Aug 15 2012 22:28:36: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface Shield-Inside

Aug 15 2012 22:28:36: %FWSM-1-105008: (Primary) Testing Interface Shield-Inside

Aug 15 2012 22:28:36: %FWSM-1-105009: (Primary) Testing on interface Shield-Inside Passed

Aug 15 2012 22:29:21: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface BWS-Inside

Aug 15 2012 22:29:21: %FWSM-1-105008: (Primary) Testing Interface BWS-Inside

Aug 15 2012 22:29:22: %FWSM-1-105009: (Primary) Testing on interface BWS-Inside Passed

Aug 15 2012 22:29:36: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface CIM-Inside

Aug 15 2012 22:29:36: %FWSM-1-105008: (Primary) Testing Interface CIM-Inside

Aug 15 2012 22:29:40: %FWSM-1-105009: (Primary) Testing on interface CIM-Inside Passed

Aug 15 2012 22:29:51: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface Shield-B2C

Aug 15 2012 22:29:51: %FWSM-1-105008: (Primary) Testing Interface Shield-B2C

Aug 15 2012 22:29:52: %FWSM-1-105009: (Primary) Testing on interface Shield-B2C Passed

Aug 15 2012 22:30:06: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface Shield-Inside

Aug 15 2012 22:30:06: %FWSM-1-105008: (Primary) Testing Interface Shield-Inside

Aug 15 2012 22:30:06: %FWSM-1-105009: (Primary) Testing on interface Shield-Inside Passed

Aug 15 2012 22:31:06: %FWSM-1-105005: (Primary) Lost Failover communications with mate on interface BWS-Inside

Aug 15 2012 22:31:06: %FWSM-1-105008: (Primary) Testing Interface BWS-Inside

No one had logged in to make a change on the switches or FW modules at the time and there is no evidence of the line going down.

Currently the seconardy FW module is powered down. Any ideas what to look for before we power it back up?

The issue is similar to this old thread:

https://supportforums.cisco.com/docs/DOC-4831

Ton V Engelen · ‎08-16-2012

Hi

it seems like this fwsm lost contact with its fail-over fwsm peer.

You say the network behind the 6500 failed. Could it be that the FWSM's lost contact because of this? Some routes disappeared or something?

In that case, if that network is up and running again, they should be able to see each other again.

I woud only worry about what happens when they see each other again.(who will become master) cause i do not know the active/standby configuration of your set up.

Maybe that does not matter in your set up. (it would matter in my setuop, i m running active active over 2 locations...)