Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
3307
Views
2
Helpful
5
Replies

FTD 1120 after upgrade to 7.2.5-208 is not connecting to the FMC

Go to solution
DMaletic
Level 1
Level 1

‎09-13-2023 01:24 AM

Hello

We have two FTDs 1120 in remote offices that have, after being upgraded to 7.2.5 from 7.0.4, lost connectivity to FMC in our main office.
Both devices have been initially configured in the main office and then shipped to the remote office. Management was moved to the data interface prior to shipping them out and everything worked as intended for several months. After the upgrade, which showed no errors in the logs, the sftunnel is not coming up. There appears to be no issues with the traffic flow as the VPN tunnel and BGP are running fine on the devices.

Before doing these upgrades we have successfully performed upgrades on 3 other devices - one 2120 HA in our data center, 1150 HA in our main office and one 1010 in the main office which is pending deployment to the remote office. No issues with these devices whatsoever.

The only difference between the devices that are working (in the main office) and the remote office ones, that are having the sftunnel issue, is that the devices in the main office are using their mgmt interface to connect to the FMC and not the data interface (FMC and the three devices have connectivity over our LAN).

/ngfw/var/log/messages on the affected devices is providing these lines:

Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [INFO] Processing connection from <nat_FMC_public_ip>:32442/tcp (socket 12)
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [INFO] acceptSSLConnection: Device will attempt SSL connection to Peer
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [INFO] establishConnectionUtil: Connecting SSL to peer
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [ERROR] SSL_renegotiate error: 1: error:00000001:lib(0):func(0):reason(1)
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [WARN] establishConnectionUtil: SSL handshake failed
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [WARN] establishConnectionUtil: SSL Verification status: ok
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [WARN] establishConnectionUtil: SSL handshake failed: error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected message
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [INFO] establishConnectionUtil: Failed to connect using SSL to: ''
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [ERROR] acceptSSLConnection: Unable to establish connection error: SSL Error
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [ERROR] acceptSSLConnection: Failed connecting to peer '(null)' using SSL
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58173] sftunneld:sf_ssl [INFO] Failed to Accept SSL connection from: <nat_FMC_public_ip>:32442/tcp
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58171] sftunneld:sf_ssl [ERROR] SSL_renegotiate error: 1: error:00000001:lib(0):func(0):reason(1)
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58171] sftunneld:sf_ssl [WARN] establishConnectionUtil: SSL handshake failed
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58171] sftunneld:sf_ssl [WARN] establishConnectionUtil: SSL Verification status: ok
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58171] sftunneld:sf_ssl [WARN] establishConnectionUtil: SSL handshake failed: error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected message
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58171] sftunneld:sf_ssl [INFO] establishConnectionUtil: Failed to connect using SSL to: ''
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58171] sftunneld:sf_ssl [ERROR] acceptSSLConnection: Unable to establish connection error: SSL Error
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58171] sftunneld:sf_ssl [ERROR] acceptSSLConnection: Failed connecting to peer '(null)' using SSL
Sep 12 08:59:52 <Remote_office-FW01> SF-IMS[7076]: [58171] sftunneld:sf_ssl [INFO] Failed to Accept SSL connection from: <nat_FMC_public_ip>:38239/tcp

 

sftunnel status:

> sftunnel-status

SFTUNNEL Start Time: Wed Sep 13 07:42:01 2023

Both IPv4 and IPv6 connectivity is supported
Broadcast count = 2
Reserved SSL connections: 1
Management Interfaces: 2
management0 (control events) 10.49.40.199,
tap_nlp (control events) 169.254.1.3,fd00:0:0:1::3

***********************

**RUN STATUS****<FMC_FQDN>*************
Connected: No
SSL Verification status: ok
Registration: Completed.
Connection to peer '<FMC_FQDN>' never happened
Connection to peer '<FMC_FQDN>' Attempted at Wed Sep 13 08:10:57 2023 UTC


***********************
peer 4cf58e34-d29e-11ec-857b-a6d565e7010d did not reply at /usr/local/sf/bin/sftunnel_status.pl line 319.
Retry rpc status poll at /usr/local/sf/bin/sftunnel_status.pl line 325.

**RPC STATUS****<FMC_FQDN>*************
RPC status :Failed
Check routes:
No peers to check

>show network

===============[ System Information ]===============
Hostname : <Remote_office_FTD>
DNS Servers : 208.67.222.222
208.67.220.220
2620:119:35::35
DNS from router : enabled
Management port : 8305
IPv4 Default route
Gateway : 10.49.40.10
Netmask : 0.0.0.0


==================[ management0 ]===================
Admin State : enabled
Admin Speed : 1gbps
Operation Speed : indeterminate
Link : link-down
Channels : Management & Events
Mode : Non-Autonegotiation
MDI/MDIX : Auto/MDIX
MTU : 1300
MAC Address : A8:4F:B1:C7:53:80
----------------------[ IPv4 ]----------------------
Configuration : Manual
Address : 10.49.40.199
Netmask : 255.255.255.0
Gateway : 10.49.40.10
----------------------[ IPv6 ]----------------------
Configuration : Disabled

===============[ Proxy Information ]================
State : Disabled
Authentication : Disabled

======[ System Information - Data Interfaces ]======
DNS Servers :
Interfaces : Ethernet1/8

==================[ Ethernet1/8 ]===================
State : Enabled
Link : Up
Name : outside
MTU : 1500
MAC Address : A8:4F:B1:C7:53:AB
----------------------[ IPv4 ]----------------------
Configuration : Manual
Address : x.x.x.x <RO_public_IP>
Netmask : 255.255.255.252
Gateway : x.x.x.x
----------------------[ IPv6 ]----------------------
Configuration : Disabled

only thing that is different from this one and the other FTDs in remote offices (that are running older versions and working) is that the mgmt interface is shown as link down on these two.

We checked the date on the FTD and FMC - is the same. No datetime discrepancies.
MTU was lowered on the FTD to 1300 - no joy.

Currently we have a case with TAC opened.. but they are tapping in the dark for the moment.
If perhaps anyone has any ideas.. would be interested to hear.

Thank you!

0 Helpful
1 Accepted Solution

Accepted Solutions

Go to solution
DMaletic
Level 1
Level 1

‎09-15-2023 12:57 AM - edited ‎09-15-2023 12:59 AM

Update.

After digging around the config settings and a suggestion from our external network support expert, we manually changed the TLS setting in the sftunnel.conf to TLS1.2 and the tunnel went up.

FMC is behind an FTD pair in our DC and on that pair TLS 1.3 inspection is enabled. This appears to have been causing the issue.

DMaletic_0-1694764037810.png

FTD config wise:

FTD version 7.0.4 sftunnel.conf doesnt have TLS version line:

proxyssl {
  proxy_cert   /etc/sf/keys/sftunnel-cert.pem;
  proxy_key    /etc/sf/keys/sftunnel-key.pem;
  proxy_cacert /etc/sf/ca_root/cacert.pem;
  proxy_crl    /etc/sf/ca_root/crl.pem;
  proxy_cipher 1;
};

while the 7.2.5 one has:

proxyssl {
  proxy_tls_version TLSv1.3;
  proxy_cert   /etc/sf/keys/sftunnel-cert.pem;
  proxy_key    /etc/sf/keys/sftunnel-key.pem;
  proxy_cacert /etc/sf/ca_root/cacert.pem;
  proxy_crl    /etc/sf/ca_root/crl.pem;
  proxy_cipher 1;
};

This was changed to 

  proxy_tls_version TLSv1.2;

and after restarting the sftunnel the tunnel went up and TLS 1.2 was accepted.


On the other hand, 7.0.4. seems to be using TLS1.2 by default:

Sep 14 21:42:01 <7.0.4_FTD_REMOTE_OFFICE> SF-IMS[7513]: [29774] sftunneld:sf_ssl [INFO] AcceptSSLConnection accepted Peer TLS version is [TLSv1.2]

It was too late in the evening to even try to get more info in order to try to find the root cause of this.
Hopefully can get TAC on it also.

 

View solution in original post

5 Replies 5

Go to solution
Marius Gunnerud
VIP
VIP

‎09-13-2023 01:59 PM

Have you tried restarting the sftunnel?

pmtool restartbyid sftunnel

You could also cat /var/sf/peers/FMC_UUID/sftunnel.json and verify that the manager information is correct (change FMC_UUID to the FMC UUID listed in that directory.)

--
Please remember to select a correct answer and rate helpful posts
0 Helpful

Go to solution
DMaletic
Level 1
Level 1

‎09-14-2023 01:15 AM

Hello

Yes, the tunnel was restarted a few times. The entire device rebooted.
Last night in call with TAC the certificate used with the sftunnel on the FTD was recreated.
No joy.

 

The information in the peers .json file is:

root@<Remote_office_FTD>:/var/sf/peers/<FMC_UID># cat sftunnel.json
{
"peers_registered" : {
"<FMC_UID>" : {
"upgrade_version" : "",
"reg_key" : "<reg_key>",
"role" : "1",
"regid_preserved" : "<FMC_UID>",
"name" : "<FMC_FQDN>",
"sw_version" : "7.2.5",
"uuid" : "<FMC_UID>",
"ip" : "10.49.0.1",
"domain" : "<FMC_UID>"
}
}
}

From what I can see it is displaying the internal IP of the FMC which was used to connect to when the device was in the main office and initial config rolled out but the rest seems legit and proper data. 

FTD 1010 and another one 1120 running on 7.0.5 dont even have the .json file in the directory?

0 Helpful

Go to solution
DMaletic
Level 1
Level 1

‎09-15-2023 12:57 AM - edited ‎09-15-2023 12:59 AM

Update.

After digging around the config settings and a suggestion from our external network support expert, we manually changed the TLS setting in the sftunnel.conf to TLS1.2 and the tunnel went up.

FMC is behind an FTD pair in our DC and on that pair TLS 1.3 inspection is enabled. This appears to have been causing the issue.

DMaletic_0-1694764037810.png

FTD config wise:

FTD version 7.0.4 sftunnel.conf doesnt have TLS version line:

proxyssl {
  proxy_cert   /etc/sf/keys/sftunnel-cert.pem;
  proxy_key    /etc/sf/keys/sftunnel-key.pem;
  proxy_cacert /etc/sf/ca_root/cacert.pem;
  proxy_crl    /etc/sf/ca_root/crl.pem;
  proxy_cipher 1;
};

while the 7.2.5 one has:

proxyssl {
  proxy_tls_version TLSv1.3;
  proxy_cert   /etc/sf/keys/sftunnel-cert.pem;
  proxy_key    /etc/sf/keys/sftunnel-key.pem;
  proxy_cacert /etc/sf/ca_root/cacert.pem;
  proxy_crl    /etc/sf/ca_root/crl.pem;
  proxy_cipher 1;
};

This was changed to 

  proxy_tls_version TLSv1.2;

and after restarting the sftunnel the tunnel went up and TLS 1.2 was accepted.


On the other hand, 7.0.4. seems to be using TLS1.2 by default:

Sep 14 21:42:01 <7.0.4_FTD_REMOTE_OFFICE> SF-IMS[7513]: [29774] sftunneld:sf_ssl [INFO] AcceptSSLConnection accepted Peer TLS version is [TLSv1.2]

It was too late in the evening to even try to get more info in order to try to find the root cause of this.
Hopefully can get TAC on it also.

 

Go to solution
Marius Gunnerud
VIP
VIP
In response to DMaletic

‎09-15-2023 02:40 AM

Interesting and good to know. Thanks for sharing.

--
Please remember to select a correct answer and rate helpful posts
0 Helpful

Go to solution
DMaletic
Level 1
Level 1

‎09-22-2023 05:11 AM

Well..
Last update from Cisco after providing them with the information was:
"After further internal research, because TLS Server identity was enabled and the minimum TLS version on the 7.2.5 is TLS1.3. enhancement on the version will be considered."

Workaround with 1.2 is still working. 

0 Helpful
Customers Also Viewed These Support Documents
Review Cisco Networking for a $25 gift card