FWSM Lost Failover communications with mate

Sung Min Cho · ‎02-17-2010

All,

I experinced critical problem for our customer service pertaing to communication fail for FT between FWSMs.

I can just find the log in FWSM as follows and can't find any physical log as interface down in C6500 at that moment.

Now our FWSM OS version is 3.2(7) and jsut monitor-interface option is applied at outside interface in FWSM.

Main FWSM

1|Feb 13 2010 01:49:19|105005: (Secondary) Lost Failover communications with mate on interface statelink
1|Feb 13 2010 01:49:20|105004: (Secondary) Monitoring on interface statelink normal

Backup FWSM

1|Feb 13 2010 01:49:17|105005: (Primary) Lost Failover communications with mate on interface statelink
1|Feb 13 2010 01:49:22|105004: (Primary) Monitoring on interface statelink normal
1|Feb 13 2010 01:51:24|105005: (Primary) Lost Failover communications with mate on interface outside
1|Feb 13 2010 01:51:24|105008: (Primary) Testing Interface outside
1|Feb 13 2010 01:51:24|105009: (Primary) Testing on interface outside Passed

Our problem is recovered automatically after 2m ~ 3m but that is reoccured after 1 day.

I already opend the TAC case (SR 613646199) but I didn't get any correct cause for problem and they just recommend to change the FWSM through RMA.

Any advice for our problem whould be greatly appreciated.

Thanks,

Sungmin Cho

Jon Marshall · ‎02-17-2010

minmin5063 wrote:
All,
I experinced critical problem for our customer service pertaing to communication fail for FT between FWSMs.
I can just find the log in FWSM as follows and can't find any physical log as interface down in C6500 at that moment.
Now our FWSM OS version is 3.2(7) and jsut monitor-interface option is applied at outside interface in FWSM. 
Main FWSM
1|Feb 13 2010 01:49:19|105005: (Secondary) Lost Failover communications with mate on interface statelink
1|Feb 13 2010 01:49:20|105004: (Secondary) Monitoring on interface statelink normal
Backup FWSM

1|Feb 13 2010 01:49:17|105005: (Primary) Lost Failover communications with mate on interface statelink
1|Feb 13 2010 01:49:22|105004: (Primary) Monitoring on interface statelink normal
1|Feb 13 2010 01:51:24|105005: (Primary) Lost Failover communications with mate on interface outside
1|Feb 13 2010 01:51:24|105008: (Primary) Testing Interface outside
1|Feb 13 2010 01:51:24|105009: (Primary) Testing on interface outside Passed
Our problem is recovered automatically after 2m ~ 3m but that is reoccured after 1 day.
I already opend the TAC case (SR 613646199)  but I didn't get any correct cause for problem and they just recommend to change the FWSM through RMA.  
Any advice for our problem whould be greatly appreciated.
Thanks,
Sungmin Cho

Sungmin

Does the FWSM have a dedicated link between the 6500 switches or are you using the interconnect that all the other traffic uses as well ?

If you are using the same interconnect as other traffic it could be worth trying to setup a dedicated interconnect just for the FWSM.

Jon

Sung Min Cho · ‎02-17-2010

Jon,

We already divided the links between C6500s for FT (Regular and Stateful Failover) and data.

Thanks,

Sungmin

Kureli Sankar · ‎02-17-2010

We have seen similar issues due to defect CSCsl39710.

Make sure you are running a code on the switch side where this is resolved.

Also, make sure the blade is not seeing more traffic than it can handle at any given time. If so, icmp will be given lower priority and interface monitoring may fail as ping test is one of them.

-KS

Sung Min Cho · ‎02-17-2010

KS,

First of all, thank you for your reply.

The followinfg is mac-address-table at that time the problem occured.

* 903 001a.6c3d.9200 dynamic Yes 0 Po273

* 1127 001a.6c3d.9200 dynamic Yes 0 Po273

* 1121 001a.6c3d.9200 dynamic Yes 0 Po273

* 1101 001a.6c3d.9200 dynamic Yes 0 Po273
* 1100 001a.6c3d.9200 dynamic Yes 0 Po273

* 174 001a.6c3d.9200   dynamic Yes          0   Po273
* 175 001a.6c3d.9200   dynamic Yes          0   Po273
* 180 001a.6c3d.9200   dynamic Yes          0   Po273
* 178 001a.6c3d.9200   dynamic Yes          5   Po273
* 177 001a.6c3d.9200   dynamic Yes         10   Po273
* 191 001a.6c3d.9200   dynamic Yes          0   Po273
* 189 001a.6c3d.9200   dynamic Yes          0   Po273
* 187 001a.6c3d.9200   dynamic Yes          0   Po273

.................................................................................

We think that C6500 properly learned the MAC address of the FWSM at that moment.

Thanks,

Sungmin

Rafael Trujilho · ‎12-17-2010

All,

I'm with problems the FWSM in communication for management interface.

Basically the setup is correct, but I have had the record of the logs with a frequency below:

Main FWSM

1|Dec 17 2010 13:01:41|105009: (Secondary_group_2) Testing on interface MANAGEMENT Passed

1|Dec 17 2010 13:01:37|105008: (Secondary) Testing Interface MANAGEMENT

1|Dec 17 2010 13:01:36|105005: (Secondary_group_2) Lost Failover communications with mate on interface MANAGEMENT

FWSM Version 4.1(3)

Catalyst 6509 Version 12.2(33).SXI4a

Could anyone suggest something?

Thanks!

Kureli Sankar · ‎12-17-2010

The only lasts for 5 seconds and then quickly recovers.

It may be a busy interface. You can try to capture all traffic (IP protocol 105) on this interface on both the units when the problem occurs and see why what one unit sends doesn't arrive on the other unit and the interface goes into testing mode.

-KS

Rafael Trujilho · ‎12-20-2010

Hi KS,

Considering that the device is an FWSM, it could mean a hardware problem?

Thanks,

Trujilho

phillchannon · ‎03-29-2011

We've just experienced what seems to be the exact same thing.

Did you get a resolution to this ? How did you go with the TAC case ?

Thanks!

Phill.

Ian Beck · ‎04-12-2011

Hi,

How was this resolved ?

As we are now seeing the same issue.

Many thanks

Ian Beck · ‎04-12-2011

Hi,

Found my issue :

Caused by a static NAT using the Firewall Interface as the NAT Addrees, changed it to a assigned IP Address and the problem went away !!!

Thanks