Re: Cannot Register SFR Module to FMC - Page 2

Matthew Martin · ‎12-20-2018

Hello All,

When attempting to Register a new SFR Module to our FMC I receive the message: "Could not establish a connection with device."

I know for sure the reg key is correct and I am able to ping each device from the other without issue.

The FMC device is located in our HQ and the SFR Module is located across WAN in a DR location. Also, I had no problems registering 2 other SFR modules which were located in the same physical location as the FMC.

I tried the "telnet <fmc_ipaddress> 8305" command from the SFR Module to the FMC, and receive the following message:

admin@ASASFR3:~$ telnet 192.168.2.20 8305
Trying 192.168.2.20...
telnet: connect to address 192.168.2.20: Connection refused

Log from /var/log/messages shows:

admin@ASASFR3:~$ tail -f /var/log/messages
Dec 20 22:35:26 ASASFR3 SF-IMS[11384]: [11401] sftunneld:tunnsockets [INFO] Accepted IPv4 connection from 192.168.2.20:55608/tcp
Dec 20 22:35:26 ASASFR3 SF-IMS[11384]: [17278] sftunneld:sf_ssl [INFO] Processing connection from 192.168.2.20:55608/tcp (socket 11)
Dec 20 22:35:33 ASASFR3 SF-IMS[11384]: [11402] sftunneld:sf_peers [INFO] Peer 192.168.2.20 needs a single connection
Dec 20 22:35:33 ASASFR3 SF-IMS[11384]: [11402] sftunneld:sf_connections [INFO] Start connection to : 192.168.2.20 (wait 44 seconds is up)
Dec 20 22:35:33 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_peers [INFO] Peer 192.168.2.20 needs a single connection
Dec 20 22:35:33 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_ssl [INFO] Connect to 192.168.2.20 on port 8305 - eth0
Dec 20 22:35:33 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_ssl [INFO] Initiate IPv4 connection to 192.168.2.20 (via eth0)
Dec 20 22:35:33 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_ssl [INFO] Initiating IPv4 connection to 192.168.2.20:8305/tcp
Dec 20 22:35:33 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_ssl [INFO] Wait to connect to 8305 (IPv6): 192.168.2.20
Dec 20 22:35:33 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_ssl [INFO] Connected to 192.168.2.20:8305 (IPv4)
Dec 20 22:37:10 ASASFR3 SF-IMS[25616]: [25616] CloudAgent:CloudAgent [INFO] IPRep, time to check for updates
Dec 20 22:37:10 ASASFR3 SF-IMS[25616]: [25616] CloudAgent:CloudAgent [INFO] ClamUpd, time to check for updates
Dec 20 22:37:56 ASASFR3 SF-IMS[11384]: [17278] sftunneld:sf_ssl [INFO] Wait SSL_accept_nb: TIMEOUT TO COMPLETE
Dec 20 22:37:56 ASASFR3 SF-IMS[11384]: [17278] sftunneld:sf_ssl [ERROR] Accept:SSL handshake failed
Dec 20 22:37:56 ASASFR3 SF-IMS[11384]: [17278] sftunneld:sf_ssl [WARN] SSL Verification status: ok
Dec 20 22:38:00 ASASFR3 SF-IMS[11384]: [11401] sftunneld:tunnsockets [INFO] Accepted IPv4 connection from 192.168.2.20:58792/tcp
Dec 20 22:38:00 ASASFR3 SF-IMS[11384]: [17403] sftunneld:sf_ssl [INFO] Processing connection from 192.168.2.20:58792/tcp (socket 11)
Dec 20 22:38:03 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_ssl [INFO] Wait SSL_connect_nb: TIMEOUT TO COMPLETE
Dec 20 22:38:03 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_ssl [ERROR] Connect:SSL handshake failed
Dec 20 22:38:03 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_ssl [WARN] SSL Verification status: ok
Dec 20 22:38:03 ASASFR3 SF-IMS[11384]: [17281] sftunneld:sf_ssl [INFO] reconnect to peer '192.168.2.20' in 300 seconds
Dec 20 22:38:07 ASASFR3 SF-IMS[11384]: [11402] sftunneld:sf_peers [INFO] Peer 192.168.2.20 needs a single connection
Dec 20 22:38:07 ASASFR3 SF-IMS[11384]: [11402] sftunneld:sf_connections [INFO] Start connection to : 192.168.2.20 (wait 300 seconds is up)
Dec 20 22:38:07 ASASFR3 SF-IMS[11384]: [17408] sftunneld:sf_peers [INFO] Peer 192.168.2.20 needs a single connection
Dec 20 22:38:07 ASASFR3 SF-IMS[11384]: [17408] sftunneld:sf_ssl [INFO] Connect to 192.168.2.20 on port 8305 - eth0
Dec 20 22:38:07 ASASFR3 SF-IMS[11384]: [17408] sftunneld:sf_ssl [INFO] Initiate IPv4 connection to 192.168.2.20 (via eth0)
Dec 20 22:38:07 ASASFR3 SF-IMS[11384]: [17408] sftunneld:sf_ssl [INFO] Initiating IPv4 connection to 192.168.2.20:8305/tcp
Dec 20 22:38:07 ASASFR3 SF-IMS[11384]: [17408] sftunneld:sf_ssl [INFO] Wait to connect to 8305 (IPv6): 192.168.2.20
Dec 20 22:38:07 ASASFR3 SF-IMS[11384]: [17408] sftunneld:sf_ssl [INFO] Connected to 192.168.2.20:8305 (IPv4)

Any ideas on what the issue could be here? I have rebooted the FMC as well as re-installed SFR module on the ASA and it didn't seem to help.

Any thoughts or suggestions would be greatly appreciated.

Thanks in Advance,

Matt

Matthew Martin · ‎01-16-2019

I did. Basically, he just did a ping from the sfr to firepower with a large byte size and it showed packet loss so he concluded it was a networking issue.

But, I'm not really sure why this would cause an SSL handshake error...

-Matt

Sheraz.Salim · ‎01-16-2019

strange. TAC should pick it up. or could be a junior engineer. just curious if you not explain him the issue?

please do not forget to rate.

Matthew Martin · ‎01-16-2019

Yea, we've been back and forth with email for a little while now and we had a WebEx session where he did the ping tests.

He had also said that in their Database, a similar issue was fixed by adjusting the MTU of the SFR's eth0 to 1300. But, that didn't seem to help any in our case.

We have a 100 Mbps pipe between our HQ where FIrepower VM is located and where the SFR module is located, so there aren't any Bandwidth issues or anything like that (*I checked ISP's reporting of bandwidth usage and that all seems good). We also have a packetshaper between the locations, but that doesn't appear to be limiting the byte transfer or anything along those lines.

So I'm kind of at a loss for what the problem is... He did suggest spinning up a Firepower VM that is local to that SFR module, but we are in our busy season and my manager who handles that stuff is a bit swamped. Plus, I would assume there would be licensing that would get in the way...?

Thanks Again for your replies, much appreciated.

-Matt

Sheraz.Salim · ‎01-16-2019

He did suggest spinning up a Firepower VM that is local to that SFR module, but we are in our busy season and my manager who handles that stuff is a bit swamped. Plus, I would assume there would be licensing that would get in the way...?

okay. in that case yes this is possible. ASA SFR come with traditional lic and you can rehost the problematic SFR to new FMC. but best if to check cisco lic team. I have my ASA SFR with VM FMC running. many time happens doing testing i break up my FMC. so what i do is i re-host my ASA on new FMC (i spin a new vFMC) and rehost my lic on it.

hope this find you any help.

please do not forget to rate.

Matthew Martin · ‎01-22-2019

So I have some new information.

I decided to try and do the pings again using the high byte count that caused the packet loss between the SFR Module and the FMC. But, instead of pinging the FMC address, I pinged some other VMs that are located on the exact same ESXi server (*over the same physical hardware: wire, interface, etc...) and that are on the same Vlan as the FMC. Doing these pings, with the same ping command, I experienced no packet loss at all.

I then hoped onto one of these VMs, and did the ping command with the same high byte count, and pinged the SFR module and experienced no packet loss here either.

So, it appears that the issue lies somewhere with the FMC virtual machine itself. Would you agree?

-Matt

Sheraz.Salim · ‎01-22-2019

I wonder if the FMC is the problem than you have other sfr modules install under this FMC? if so they does not have the issue at all. if this is the case than it point finger to your ASA SFR which is causing an issue and you been looking into this since long time.

if above is not the case than yes seem your point is vaild.

please do not forget to rate.

Matthew Martin · ‎01-22-2019

Yea, I've kind of just been coming back to this in spare time...

Yes, the FMC does have 2 other SFR modules connected to it. Those SFR modules are located on the ASA Failover pair located in the same physical location as FMC, and on same Vlan. Not that should make a difference, but thought I would mention it.

Since I did have that SSD failure on that problem ASA a few weeks ago, is there anyway you could see the SSD being bad and that causing the issue? Even after doing re-image/re-install of SFR module, the same problem persisted...

-Matt

Sheraz.Salim · ‎01-22-2019

Since I did have that SSD failure on that problem ASA a few weeks ago, is there anyway you could see the SSD being bad and that causing the issue? Even after doing re-image/re-install of SFR module, the same problem persisted.

if this is the same old problematic SSD you using in production than might there is a chance. you should replace and check it. if you want to check the health of SSD than i suggest to seek TAC advise if you damage it in terms of software scan checkup might cisco refuse to RMA it.

please do not forget to rate.

Matthew Martin · ‎01-22-2019

Ok, I'll ask TAC about the SSD possibly being the issue.

As per your previous message, I would think the SFR would have issues pinging those other VMs that I mentioned (*that live on the same ESXi server) if the issue was with the SFR...

However, at this point any inkling of an answer would be nice...

Thanks Again,
Matt

Sheraz.Salim · ‎01-22-2019

in your case my input is. if the other SFR are working fine with vFMC and this problematic SFR had history of issue example SSD failure. ping losses etc. to eliminate the problem we have to take a baby steps in order to fix the issue. rest all fine but this sfr had issue. put the new SSD and check how it goes if no sucess than we back again to analyses where the issue is.

just to give you a bit history of my case notes. we have a similar issue where the box restart itself and firepower sfr module config was wipe off competely in my case we RMA the whole box with new SSDs

please do not forget to rate.

rodcespe · ‎01-30-2020

I've had this problem before, the solution is to make sure both the FMC and the Firepower Module has the same date and time, otherwise you will see the following SSL errors:

MSGS: 01-09 02:14:40 firepower SF-IMS[41915]: [41926] sftunneld:tunnsockets [INFO] Accepted IPv4 connection from 10.122.177.57:49837/tcp
MSGS: 01-09 02:14:40 firepower SF-IMS[41915]: [12220] sftunneld:sf_ssl [INFO] Processing connection from 10.122.177.57:49837/tcp (socket 12)
MSGS: 01-09 02:14:40 firepower SF-IMS[41915]: [12220] sftunneld:sf_ssl [ERROR] Accept:SSL handshake failed
MSGS: 01-09 02:14:40 firepower SF-IMS[41915]: [12220] sftunneld:sf_ssl [WARN] SSL Verification status: ok
MSGS: 01-09 02:14:47 firepower SF-IMS[41915]: [41926] sftunneld:tunnsockets [INFO] Accepted IPv4 connection from 10.122.177.57:46989/tcp
MSGS: 01-09 02:14:47 firepower SF-IMS[41915]: [12482] sftunneld:sf_ssl [INFO] Processing connection from 10.122.177.57:46989/tcp (socket 12)
MSGS: 01-09 02:14:47 firepower SF-IMS[41915]: [12482] sftunneld:sf_ssl [ERROR] Accept:SSL handshake failed
MSGS: 01-09 02:14:47 firepower SF-IMS[41915]: [12482] sftunneld:sf_ssl [WARN] SSL Verification status: ok
MSGS: 01-09 02:14:52 firepower SF-IMS[41915]: [41927] sftunneld:sf_peers [INFO] Peer 6267f750-fb53-11de-beb0-92c3fbbeb830DONTRESOLVE needs a single connection
MSGS: 01-09 02:14:54 firepower SF-IMS[41915]: [41926] sftunneld:tunnsockets [INFO] Accepted IPv4 connection from 10.122.177.57:51549/tcp
MSGS: 01-09 02:14:54 firepower SF-IMS[41915]: [12692] sftunneld:sf_ssl [INFO] Processing connection from 10.122.177.57:51549/tcp (socket 12)
MSGS: 01-09 02:14:54 firepower SF-IMS[41915]: [12692] sftunneld:sf_ssl [ERROR] Accept:SSL handshake failed
MSGS: 01-09 02:14:54 firepower SF-IMS[41915]: [12692] sftunneld:sf_ssl [WARN] SSL Verification status: ok
MSGS: 01-09 02:15:01 firepower SF-IMS[41915]: [41926] sftunneld:tunnsockets [INFO] Accepted IPv4 connection from 10.122.177.57:59923/tcp
MSGS: 01-09 02:15:01 firepower SF-IMS[41915]: [12909] sftunneld:sf_ssl [INFO] Processing connection from 10.122.177.57:59923/tcp (socket 12)
MSGS: 01-09 02:15:01 firepower SF-IMS[41915]: [12909] sftunneld:sf_ssl [ERROR] Accept:SSL handshake failed
MSGS: 01-09 02:15:01 firepower SF-IMS[41915]: [12909] sftunneld:sf_ssl [WARN] SSL Verification status: ok
SERR: 01-09 02:15:04 Pruner[12037]: Running Pruning at /ngfw/usr/local/sf/bin/Pruner.pl line 106.
MSGS: 01-09 02:15:08 firepower SF-IMS[41915]: [41926] sftunneld:tunnsockets [INFO] Accepted IPv4 connection from 10.122.177.57:46963/tcp
MSGS: 01-09 02:15:08 firepower SF-IMS[41915]: [13056] sftunneld:sf_ssl [INFO] Processing connection from 10.122.177.57:46963/tcp (socket 12)
MSGS: 01-09 02:15:08 firepower SF-IMS[41915]: [13056] sftunneld:sf_ssl [ERROR] Accept:SSL handshake failed
MSGS: 01-09 02:15:08 firepower SF-IMS[41915]: [13056] sftunneld:sf_ssl [WARN] SSL Verification status: ok
MSGS: 01-09 02:15:14 firepower SF-IMS[41915]: [41927] sftunneld:sf_peers [INFO] Peer 6267f750-fb53-11de-beb0-92c3fbbeb830DONTRESOLVE needs a single connection
MSGS: 01-09 02:15:15 firepower SF-IMS[41915]: [41926] sftunneld:tunnsockets [INFO] Accepted IPv4 connection from 10.122.177.57:38465/tcp
MSGS: 01-09 02:15:15 firepower SF-IMS[41915]: [13214] sftunneld:sf_ssl [INFO] Processing connection from 10.122.177.57:38465/tcp (socket 12)
MSGS: 01-09 02:15:15 firepower SF-IMS[41915]: [13214] sftunneld:sf_ssl [ERROR] Accept:SSL handshake failed
MSGS: 01-09 02:15:15 firepower SF-IMS[41915]: [13214] sftunneld:sf_ssl [WARN] SSL Verification status: ok
MSGS: 01-09 02:15:22 firepower SF-IMS[41915]: [41926] sftunneld:tunnsockets [INFO] Accepted IPv4 connection from 10.122.177.57:37803/tcp
MSGS: 01-09 02:15:22 firepower SF-IMS[41915]: [13493] sftunneld:sf_ssl [INFO] Processing connection from 10.122.177.57:37803/tcp (socket 12)
MSGS: 01-09 02:15:22 firepower SF-IMS[41915]: [13493] sftunneld:sf_ssl [ERROR] Accept:SSL handshake failed
MSGS: 01-09 02:15:22 firepower SF-IMS[41915]: [13493] sftunneld:sf_ssl [WARN] SSL Verification status: ok

The best solution is to make sure the FMC and Firepower sensors are in sync with the NTP server, if not then you can manually change the time by running the following command via linux mode:

date -s "19 APR 2018 11:14:00"

To verify the date and time you can run the following command via Linux mode:

date