ASA Context Failover - Ifc Failure

johnlloyd_13 — Thu, 25 Mar 2021 16:11:00 GMT

hi,

i'm trying to figure out why anyconnect VPN doesn't work during "failover" to secondary ASA. it only works on the primary.

i already applied the anyconnect apex/premium license on the primary and from my understanding should be "shared" with secondary.

checking 'show failover' it seems there's an interface failure but i don't see any L1 issue on the MGMT0/0 port on both primary and secondary. i can ping each other's MGMT IP and can see the MGMT0/0 MAC address on the ports on separate switches.

the only difference is that primary has FP/SFR module while secondary doesn't. also secondary Internal-Data0/1 is down/down. can someone advise what else needs to be checked? perhaps this is a bug?

should i monitor the 'MGMT' interface in admin context: monitor-interface MGMT?

ciscoasa/pri/act# sh ver

Cisco Adaptive Security Appliance Software Version 9.8(4)20 <system>
Firepower Extensible Operating System Version 2.2(2.124)
Device Manager Version 7.12(2)

Compiled on Thu 02-Apr-20 10:26 PDT by builders
System image file is "disk0:/asa984-20-smp-k8.bin"
Config file at boot was "startup-config"

ciscoasa up 231 days 13 hours
failover cluster up 2 years 303 days

Hardware: ASA5545, 12288 MB RAM, CPU Lynnfield 2659 MHz, 1 CPU (8 cores)
ASA: 6454 MB RAM, 1 CPU (1 core)
Internal ATA Compact Flash, 8192MB
BIOS Flash MX25L6445E @ 0xffbb0000, 8192KB

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)
Boot microcode : CNPx-MC-BOOT-2.00
SSL/IKE microcode : CNPx-MC-SSL-SB-PLUS-0005
IPSec microcode : CNPx-MC-IPSEC-MAIN-0026
Number of accelerators: 1
Baseboard Management Controller (revision 0x1) Firmware Version: 2.4

0: Int: Internal-Data0/0 : address is 6cb2.ae69.6618, irq 11
1: Ext: GigabitEthernet0/0 : address is 6cb2.ae69.661d, irq 5
2: Ext: GigabitEthernet0/1 : address is 6cb2.ae69.6619, irq 5
3: Ext: GigabitEthernet0/2 : address is 6cb2.ae69.661e, irq 10
4: Ext: GigabitEthernet0/3 : address is 6cb2.ae69.661a, irq 10
5: Ext: GigabitEthernet0/4 : address is 6cb2.ae69.661f, irq 5
6: Ext: GigabitEthernet0/5 : address is 6cb2.ae69.661b, irq 5
7: Ext: GigabitEthernet0/6 : address is 6cb2.ae69.6620, irq 10
8: Ext: GigabitEthernet0/7 : address is 6cb2.ae69.661c, irq 10
9: Int: Internal-Data0/1 : address is 0000.0001.0002, irq 0
10: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 0
11: Int: Internal-Data0/2 : address is 0000.0001.0003, irq 0
12: Ext: Management0/0 : address is 6cb2.ae69.6618, irq 0
13: Int: Internal-Data0/3 : address is a2c2.f400.0011, irq 0

Licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 300 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 10 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 2500 perpetual <<<
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 2500 perpetual
Total VPN Peers : 2500 perpetual
AnyConnect for Mobile : Enabled perpetual
AnyConnect for Cisco VPN Phone : Enabled perpetual
Advanced Endpoint Assessment : Enabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 2 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual
Cluster Members : 2 perpetual

This platform has an ASA5545 VPN Premium license.

Failover cluster licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 300 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 20 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 2500 perpetual <<<
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 2500 perpetual
Total VPN Peers : 2500 perpetual
AnyConnect for Mobile : Enabled perpetual
AnyConnect for Cisco VPN Phone : Enabled perpetual
Advanced Endpoint Assessment : Enabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 4 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual

This platform has an ASA5545 VPN Premium license.

ciscoasa/pri/act# sh int ip b
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 unassigned YES unset up up
GigabitEthernet0/0.1605 unassigned YES unset up up
GigabitEthernet0/1 unassigned YES unset up up
GigabitEthernet0/1.50 unassigned YES unset up up
GigabitEthernet0/1.1610 unassigned YES unset up up
GigabitEthernet0/1.1650 unassigned YES unset up up
GigabitEthernet0/2 unassigned YES unset up up
GigabitEthernet0/2.317 unassigned YES unset up up
GigabitEthernet0/2.1653 unassigned YES unset up up
GigabitEthernet0/2.1655 unassigned YES unset up up
GigabitEthernet0/2.1660 unassigned YES unset up up
GigabitEthernet0/2.1666 unassigned YES unset up up
GigabitEthernet0/2.1667 unassigned YES unset up up
GigabitEthernet0/3 unassigned YES unset up up
GigabitEthernet0/3.1137 unassigned YES unset up up
GigabitEthernet0/3.1229 unassigned YES unset up up
GigabitEthernet0/3.1350 unassigned YES unset up up
GigabitEthernet0/4 unassigned YES unset up up
GigabitEthernet0/4.1601 unassigned YES unset up up
GigabitEthernet0/4.1651 unassigned YES unset up up
GigabitEthernet0/4.1652 unassigned YES unset up up
GigabitEthernet0/4.1800 unassigned YES unset up up
GigabitEthernet0/5 unassigned YES unset administratively down down
GigabitEthernet0/6 unassigned YES unset administratively down down
GigabitEthernet0/7 172.31.0.65 YES unset up up
Internal-Control0/0 127.0.1.1 YES unset up up
Internal-Data0/0 unassigned YES unset up up
Internal-Data0/1 unassigned YES unset up up
Internal-Data0/2 unassigned YES unset up up
Internal-Data0/3 unassigned YES unset up up
Management0/0 unassigned YES unset up up

ciscoasa/pri/act# sh failover history
==========================================================================
From State To State Reason
==========================================================================

17:50:55 UTC Mar 1 2021
Active Config Applied Active Other unit wants me Active

17:19:02 UTC Mar 24 2021
Active Failed Interface check

17:19:05 UTC Mar 24 2021
Failed Standby Ready Interface check

17:19:27 UTC Mar 24 2021
Standby Ready Failed Interface check

17:24:07 UTC Mar 24 2021
Failed Standby Ready Interface check

17:24:30 UTC Mar 24 2021
Standby Ready Failed Interface check

17:24:32 UTC Mar 24 2021
Failed Standby Ready Interface check

17:28:33 UTC Mar 24 2021
Standby Ready Just Active Set by the config command

17:28:33 UTC Mar 24 2021
Just Active Active Drain Set by the config command

17:28:33 UTC Mar 24 2021
Active Drain Active Applying Config Set by the config command

17:28:33 UTC Mar 24 2021
Active Applying Config Active Config Applied Set by the config command

17:28:33 UTC Mar 24 2021
Active Config Applied Active Set by the config command

==========================================================================

ciscoasa/pri/act# sh failover state

State Last Failure Reason Date/Time
This host - Primary
Active Ifc Failure 17:24:30 UTC Mar 24 2021
admin MGMT: Failed
Other host - Secondary
Standby Ready Ifc Failure 12:48:17 UTC Feb 25 2021
admin MGMT: Failed

====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set

/pri/act# sh failover
Failover On
Failover unit Primary
Failover LAN Interface: failover GigabitEthernet0/7 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 12 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(4)20, Mate 9.8(4)20
Serial Number: Ours FCH21411111, Mate FCH21412222
Last Failover at: 17:28:33 UTC Mar 24 2021
This host: Primary - Active
Active time: 80373 (sec)

admin Interface MGMT (10.1.1.93): Normal (Monitored)

slot 1: SFR5545 hw/sw rev (N/A/6.2.0-362) status (Up/Up)
ASA FirePOWER, 6.2.0-362, Up, (Not-Monitored)
slot 1: SFR5545 hw/sw rev (N/A/6.2.0-362) status (Up/Up)
ASA FirePOWER, 6.2.0-362, Up, (Not-Monitored)
Other host: Secondary - Standby Ready
Active time: 570 (sec)
slot 0: ASA5545 hw/sw rev (3.1/9.8(4)20) status (Up Sys)

admin Interface MGMT (10.1.1.94): Normal (Monitored)

ciscoasa/pri/act/admin# sh run monitor
monitor-interface outside
no monitor-interface service-module

ciscoasa/pri/act/admin# ping MGMT 10.1.1.94 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 10.1.1.94, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/1/10 ms

ciscoasa/pri/act# sh module

Mod Card Type Model Serial No.
---- -------------------------------------------- ------------------ -----------
0 ASA 5545-X with SW, 8 GE Data, 1 GE Mgmt ASA5545 FCH21411111
ips Unknown N/A FCH21411111
cxsc Unknown N/A FCH21411111
sfr FirePOWER Services Software Module ASA5545 FCH21411111

Mod MAC Address Range Hw Version Fw Version Sw Version
---- --------------------------------- ------------ ------------ ---------------
0 6cb2.ae69.6618 to 6cb2.ae69.6621 3.1 2.1(9)8 9.8(4)20
ips 6cb2.ae69.6616 to 6cb2.ae69.6616 N/A N/A
cxsc 6cb2.ae69.6616 to 6cb2.ae69.6616 N/A N/A
sfr 6cb2.ae69.6616 to 6cb2.ae69.6616 N/A N/A 6.2.0-362

Mod SSM Application Name Status SSM Application Version
---- ------------------------------ ---------------- --------------------------
ips Unknown No Image Present Not Applicable
cxsc Unknown No Image Present Not Applicable
sfr ASA FirePOWER Up 6.2.0-362

Mod Status Data Plane Status Compatibility
---- ------------------ --------------------- -------------
0 Up Sys Not Applicable
ips Unresponsive Not Applicable
cxsc Unresponsive Not Applicable
sfr Up Up

Mod License Name License Status Time Remaining
---- -------------- --------------- ---------------
ips IPS Module Disabled perpetual

-----

ciscoasa/sec/stby# sh ver

Cisco Adaptive Security Appliance Software Version 9.8(4)20 <system>
Firepower Extensible Operating System Version 2.2(2.124)
Device Manager Version 7.12(2)

Compiled on Thu 02-Apr-20 10:26 PDT by builders
System image file is "disk0:/asa984-20-smp-k8.bin"
Config file at boot was "startup-config"

ciscoasa up 231 days 12 hours
failover cluster up 2 years 303 days

Hardware: ASA5545, 12288 MB RAM, CPU Lynnfield 2660 MHz, 1 CPU (8 cores)
ASA: 6454 MB RAM, 1 CPU (1 core)
Internal ATA Compact Flash, 8192MB
BIOS Flash MX25L6445E @ 0xffbb0000, 8192KB

0: Int: Internal-Data0/0 : address is 6cb2.ae69.6528, irq 11
1: Ext: GigabitEthernet0/0 : address is 6cb2.ae69.652d, irq 5
2: Ext: GigabitEthernet0/1 : address is 6cb2.ae69.6529, irq 5
3: Ext: GigabitEthernet0/2 : address is 6cb2.ae69.652e, irq 10
4: Ext: GigabitEthernet0/3 : address is 6cb2.ae69.652a, irq 10
5: Ext: GigabitEthernet0/4 : address is 6cb2.ae69.652f, irq 5
6: Ext: GigabitEthernet0/5 : address is 6cb2.ae69.652b, irq 5
7: Ext: GigabitEthernet0/6 : address is 6cb2.ae69.6530, irq 10
8: Ext: GigabitEthernet0/7 : address is 6cb2.ae69.652c, irq 10
9: Int: Internal-Data0/1 : address is 0000.0001.0002, irq 0
10: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 0
11: Int: Internal-Data0/2 : address is 0000.0001.0003, irq 0
12: Ext: Management0/0 : address is 6cb2.ae69.6528, irq 0
13: Int: Internal-Data0/3 : address is a2c2.f400.0011, irq 0

Licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 300 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 10 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 2 perpetual <<<
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 2500 perpetual
Total VPN Peers : 2500 perpetual
AnyConnect for Mobile : Disabled perpetual
AnyConnect for Cisco VPN Phone : Enabled perpetual
Advanced Endpoint Assessment : Disabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 2 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual
Cluster Members : 2 perpetual

This platform has an ASA5545 VPN Premium license.

Failover cluster licensed features for this platform:
Maximum Physical Interfaces : Unlimited perpetual
Maximum VLANs : 300 perpetual
Inside Hosts : Unlimited perpetual
Failover : Active/Active perpetual
Encryption-DES : Enabled perpetual
Encryption-3DES-AES : Enabled perpetual
Security Contexts : 20 perpetual
Carrier : Disabled perpetual
AnyConnect Premium Peers : 2500 perpetual
AnyConnect Essentials : Disabled perpetual
Other VPN Peers : 2500 perpetual
Total VPN Peers : 2500 perpetual
AnyConnect for Mobile : Enabled perpetual
AnyConnect for Cisco VPN Phone : Enabled perpetual
Advanced Endpoint Assessment : Enabled perpetual
Shared License : Disabled perpetual
Total TLS Proxy Sessions : 4 perpetual
Botnet Traffic Filter : Disabled perpetual
IPS Module : Disabled perpetual
Cluster : Enabled perpetual

This platform has an ASA5545 VPN Premium license.

ciscoasa/sec/stby# sh int ip b
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 unassigned YES unset up up
GigabitEthernet0/0.1605 unassigned YES unset up up
GigabitEthernet0/1 unassigned YES unset up up
GigabitEthernet0/1.50 unassigned YES unset up up
GigabitEthernet0/1.1610 unassigned YES unset up up
GigabitEthernet0/1.1650 unassigned YES unset up up
GigabitEthernet0/2 unassigned YES unset up up
GigabitEthernet0/2.317 unassigned YES unset up up
GigabitEthernet0/2.1653 unassigned YES unset up up
GigabitEthernet0/2.1655 unassigned YES unset up up
GigabitEthernet0/2.1660 unassigned YES unset up up
GigabitEthernet0/2.1666 unassigned YES unset up up
GigabitEthernet0/2.1667 unassigned YES unset up up
GigabitEthernet0/3 unassigned YES unset up up
GigabitEthernet0/3.1137 unassigned YES unset up up
GigabitEthernet0/3.1229 unassigned YES unset up up
GigabitEthernet0/3.1350 unassigned YES unset up up
GigabitEthernet0/4 unassigned YES unset up up
GigabitEthernet0/4.1601 unassigned YES unset up up
GigabitEthernet0/4.1651 unassigned YES unset up up
GigabitEthernet0/4.1652 unassigned YES unset up up
GigabitEthernet0/4.1800 unassigned YES unset up up
GigabitEthernet0/5 unassigned YES unset administratively down down
GigabitEthernet0/6 unassigned YES unset administratively down down
GigabitEthernet0/7 172.31.0.66 YES unset up up
Internal-Control0/0 127.0.1.1 YES unset up up
Internal-Data0/0 unassigned YES unset up up
Internal-Data0/1 unassigned YES unset down down <<<
Internal-Data0/2 unassigned YES unset up up
Internal-Data0/3 unassigned YES unset up up
Management0/0 unassigned YES unset up up

ciscoasa/sec/stby# sh failover history
==========================================================================
From State To State Reason
==========================================================================

17:50:55 UTC Mar 1 2021
Active Standby Ready Set by the config command

17:19:02 UTC Mar 24 2021
Standby Ready Just Active Other unit wants me Active

17:19:02 UTC Mar 24 2021
Just Active Active Drain Other unit wants me Active

17:19:02 UTC Mar 24 2021
Active Drain Active Applying Config Other unit wants me Active

17:19:02 UTC Mar 24 2021
Active Applying Config Active Config Applied Other unit wants me Active

17:19:02 UTC Mar 24 2021
Active Config Applied Active Other unit wants me Active

17:28:33 UTC Mar 24 2021
Active Standby Ready Other unit wants me Standby

==========================================================================

ciscoasa/sec/stby# sh failover state

State Last Failure Reason Date/Time
This host - Secondary
Standby Ready Ifc Failure 12:48:17 UTC Feb 25 2021
admin MGMT: Failed
Other host - Primary
Active Ifc Failure 17:24:30 UTC Mar 24 2021
admin MGMT: Failed

====Configuration State===
Sync Done
Sync Done - STANDBY
====Communication State===
Mac set

ciscoasa/sec/stby# sh module

Mod Card Type Model Serial No.
---- -------------------------------------------- ------------------ -----------
0 ASA 5545-X with SW, 8 GE Data, 1 GE Mgmt ASA5545 FCH21412222
ips Unknown N/A FCH21412222
cxsc Unknown N/A FCH21412222
sfr Unknown N/A FCH21412222

Mod MAC Address Range Hw Version Fw Version Sw Version
---- --------------------------------- ------------ ------------ ---------------
0 6cb2.ae69.6528 to 6cb2.ae69.6531 3.1 2.1(9)8 9.8(4)20
ips 6cb2.ae69.6526 to 6cb2.ae69.6526 N/A N/A
cxsc 6cb2.ae69.6526 to 6cb2.ae69.6526 N/A N/A
sfr 6cb2.ae69.6526 to 6cb2.ae69.6526 N/A N/A

Mod License Name License Status Time Remaining
---- -------------- --------------- ---------------
ips IPS Module Disabled perpetual

Re: ASA Context Failover - Ifc Failure

Sheraz.Salim — Thu, 25 Mar 2021 21:07:40 GMT

No its not bug. its due to InternalData interfaces are used as various communication channels with the Firepower services module

Reference here:

https://www.ciscolive.com/c/dam/r/ciscolive/apjc/docs/2016/pdf/BRKSEC-3055.pdf

the work around here is install the sfr module on the standby firewall to fix the problem. yes you can keep the MGMT port in Admin context.

Re: ASA Context Failover - Ifc Failure

Sheraz.Salim — Fri, 26 Mar 2021 08:32:21 GMT

'sw-module module sfr uninstall' means the software installed on the SSD drive in your ASA will delete this software premantely.

'sw-module module sfr shudown' means it will power off the module so if required you can bring it up when needed.

ciscoasa# sw-module module ips shutdown
ciscoasa# sw-module module ips uninstall

however, just thinking might if you change your mind in future and want you use the SFR moudle what you can do is just mount off the hard disk at the front end of your ASA (de-seat it). but if you do not plan to use it in future than yes just uninstall it as mentioned above on these two command.

please do not forget to rate.

topic Re: ASA Context Failover - Ifc Failure in Network Security

ASA Context Failover - Ifc Failure

Re: ASA Context Failover - Ifc Failure

Re: ASA Context Failover - Ifc Failure

Re: ASA Context Failover - Ifc Failure

Re: ASA Context Failover - Ifc Failure

Re: ASA Context Failover - Ifc Failure

Re: ASA Context Failover - Ifc Failure