Problem with Secondary Cisco ASA 5525X SFR and Firesight

[gtsesmelis]

Hi team,

we have two ASA 5525X in failover mode with SFR installed on them and Firesight in a Linux VM which the two SFRs are registered. We use to have the secondary ASA off cause if the primary fails we have to Turn on the secondary and switch manually the wan cable. I turn on the secondary ASA every end of the week in order to take the configuration from the primary for both ASA and SFR and close it again by power button.

Last week i turn on the Secondary ASA and the Firesight couldn't see the secondary SFR and show the below message:

Module Appliance Heartbeat: Appliance <<IP ADDRESS>> is not sending heartbeats.

(I should mention that i can ping the IP ADDRESS)

I tried to investigate the problem with no success.

I also deleted the sensor from Firesight Device Management just in case something is stuck, and I try to re added with no success.

I'm new in Firepower so.........any ideas?

Thanks

[stownsend]

I Updated my System and Health Policies to fix a Time Sync issue and now too I have this issue. Did you have any luck in getting a resolution?

Thanks!

[guillerm]

I also get this same problem with this same configuration (2 ASA 5525X in failover + Firepower module in both ASAs) :

only the Firepower inside the Primary ASA is correctly seen by FireSight;

the other one inside the Secondary ASA results with the referred message :

Appliance 10.160.2.33 is not sending heartbeats

I was wondering if I should not prefer to add a Cluster instead of a Group with 2 ASAs when definings these 2 devices in FireSight  voa Menu : Devices ==> Device Management ==> Add ==> Cluster ...

initially, the 2 ASAs were not in failover mode, during test period, and FireSight was able to se both Firepower properly

I will try that and update the result here

Any feedback ?

thanks

[Tarik Admani]

There is a bug that has been fixed which is related to the heartbeat issues between the ASA and the SFR module. In addition the new code has new commands to monitor the service module.

Also you may want to look and see if you are oversubcribing your service module by looking at the product catalog for ips throughput and running a capture on your disk interface.

Thanks

[guillerm · Accepted solution]

Finally, this problem was solved by re-imaging the Firepower :

see detailled procedure here to perform this re-imaging ;

http://www.cisco.com/c/en/us/support/docs/security/asa-firepower-services/118644-configure-firepower-00.html

Before doing so, it appeared that this Firepower was not very healthy :

after a successful "'configure manager add xxxxx" command, 

the show managers command show nothing ;

it should have shown  this result :

> show managers
Host                      : 193.193.2.75
Registration Key          : AZERTY
Registration              : pending
RPC Status                :

besides, in expert mode, the following command shows several down processes (instead of none in normal state) :

sudo pmtool status | grep -i down

Last point,

after reimaging and reconfiguring everything, this Firepower, installed in the Secondary standby ASA, was seen as OK under Firesight health Monitor,

but after 10mns, it appeared in critical state with the following message :

"Interface 'DataPlaneInterface0' is not receiving any packets"

This is normal and due to the fact that standby ASA receives no flows and same thing for the Firepower inside this ASA ;

by performing a failover from Primary to Secondary ASA, this critical message disappeared for the Firepower inside the Sec ASA and appeared for the Firepower inside the Primary ASA

[walter baziuk]

can i just reboot the asa FirePower module?

this is what my FirePower VM server shows

[guillerm]

Well, after reading http://www.cisco.com/c/en/us/td/docs/security/asa/asa92/configuration/firewall/asa-firewall-cli/modules-sfr.html which says :

Does not support failover directly; when the ASA fails over, any existing ASA FirePOWER flows are transferred to the new ASA. The ASA FirePOWER module in the new ASA begins inspecting the traffic from that point forward; old inspection states are not transferred.

You are responsible for maintaining consistent policies on the ASA FirePOWER modules in the high-availability ASA pair (using FireSIGHT Management Center) to ensure consistent failover behavior.

So, it seems adding a cluster is not the solution  here ..

.

[stownsend]

My Issues turned out to be a Routing Issue of sorts.  My FireSight server has two NICs, with set for Management and Events.  I think it was able to get to one and then I change the routing and it tried to use the 2nd NIC and was failing. I fixed the routing though it was still having the issue even after I told it to run the module again.   I think that there was a Lag in which the SFR module needed to realize that there was some routing issues and it took time to figure out the new path and start sending heartbeats again. 

[guillerm]

Well, you are luckier than me;

after collecting a kind of show tech on the FireSight via CLI command :

sudo sf_troubleshoot.pl

(see http://www.cisco.com/c/en/us/support/docs/security/sourcefire-defense-center/117663-technote-SourceFire-00.html)

(to collect the resulting tgz file, it is possible to connect directly via WinSCP to the FireSight server)

I was able to see the following messages in the dir-archives\var-log/messages FireSight file about the failing Firepower (IP addr = 10.170.2.33)

Aug 20 15:17:15 ZB075 SF-IMS[21182]: [21189] sftunneld:sf_connections [INFO] Start connection to : 10.170.2.33 (wait 14 seconds is up)
Aug 20 15:17:15 ZB075 SF-IMS[21182]: [9942] sftunneld:sf_peers [INFO] Peer 10.170.2.33 needs a single connection
Aug 20 15:17:15 ZB075 SF-IMS[21182]: [9942] sftunneld:sf_ssl [INFO] Connect to 10.170.2.33 on port 8305 - eth0
Aug 20 15:17:15 ZB075 SF-IMS[21182]: [9942] sftunneld:sf_ssl [INFO] Initiate IPv4 connection to 10.170.2.33 (via eth0)
Aug 20 15:17:15 ZB075 SF-IMS[21182]: [9942] sftunneld:sf_ssl [INFO] Initiating IPv4 connection to 10.170.2.33:8305/tcp
Aug 20 15:17:15 ZB075 SF-IMS[21182]: [9942] sftunneld:sf_ssl [ERROR] Unable to connect to port 8305 (IPv4): Connection refused

When issuing the netstat -pan on the Failing FirePower, I see no TCP 8305 in listening state :

admin@IPS01-on-ASA-SEC:~$ sudo netstat -pan | grep 8305
admin@IPS01-on-ASA-SEC:~$

But, this is the same thing on the Firepower(IP addr = 10.170.2.31) installed in the Primary ASA; the only difference is that, on this good FirePower, we can see an active connection from this Firepower towards TCP Port 8305 of the FireSight as shown below :

admin@IPS01-on-ASA-PRI:~$ sudo netstat -pan | grep 8305
Password:
tcp        0      0 10.170.2.31:60247       192.168.2.75:8305       ESTABLISHED 13608/sftunnel
admin@IPS01-on-ASA-PRI:~$

Last point :

I took a TCPDUMP trace as follows in the Failing FirePower in CLI mode :

> expert
admin@IPS01-on-ASA-SEC:/var/common$ sudo tcpdump -nni eth0 host 193.193.2.75 -w /var/common/trace1-on-FP-Sec-FS-Connect.pcap

and I can see, in the resulting trace, that the only packets sent from FireSight are SYN packets sent toward the Failing Firepower on its TCP port 8305,

and all these SYN Packets are rejected by the FirePower with a TCP RST Packet, which seems normal since it is not listening on TCP port 8305 ...

No other idea to investigate;

I also tried to suppress and recreate the Device on FireSight : no change at all

[ste.ant]

Hello guillerm,

I had the exact same problem as you did - what solved it for me was to shorten the authentication key that was used to authenticate the devices and management center to each other during the device registration process. Even though Firepower Management center said that my key was within the maximum character limit it didn't work as I could see the authentication was failing in the syslog output on Firepower Management Center.

I'm running FMC 6.1.0.4 virtual and FP 6.1.0.3 on Cisco ASA 5500X series.

Hope this helps someone who may run into this.

Steve

[gtsesmelis]

After some investigation I also reimaged the secondary ASA sfr module and I was able register the module to firesight.

But as you mentioned after a while the "Interface 'DataPlaneInterface0' is not receiving any packets" appeared.

Anyway i will proceed with the upgrade of sfr modules as it mentioned here that it was bug.