Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
3376
Views
5
Helpful
14
Replies

Cisco ASA Failover stateful error! Strange problem tcp udp replica

teatrodelsogno
Level 1
Level 1

‎04-21-2015 07:51 AM - edited ‎03-11-2019 10:48 PM

Hi Everybody,

I have this strange and interesting problem on the failover ASA configuration.

Background information:

ASA5585-SSP-20

Version 9.1(5)

ASA5585-SSP-20, 12288 MB RAM, CPU Xeon 5500 series 2133 MHz, 1 CPU (8 cores)

Free memory:        9636244336 bytes (75%)
Used memory:        3248657552 bytes (25%)

Before I used just one management port for the failover connectivity and it was working well.

Now the only change, is the gbit port and the redundant configuration. (for the failover configuration part)

 

Configuration "extract"

interface red1
member-interface g0/6
member-interface g0/7

failover lan unit secondary
failover lan interface failover red1
failover link failover red1
failover polltime unit 3 holdtime 9
failover interface ip failover 192.168.0.1 255.255.255.252 standby 192.168.0.2
failover

failover lan unit primary
failover lan interface failover red1
failover link failover red1
failover polltime unit 3 holdtime 9
failover interface ip failover 192.168.0.1 255.255.255.252 standby 192.168.0.2
failover

 

Problem

Seems that failover connection are not correctly propagated on the falover units!

If I type sh connection on the stdby unit, I'm able to see just few connections against thousand of connections on the active one.

But the failover system seems working good:

failover exec standby show failover

Failover On

Failover unit Secondary

Failover LAN Interface: failover Redundant1 (up)

Unit Poll frequency 3 seconds, holdtime 9 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 10 of 1025 maximum

Version: Ours 9.1(5), Mate 9.1(5)

Last Failover at: 14:14:05 CEDT Apr 21 2015

This host: Secondary - Standby Ready

 

Failover On

Failover unit Primary

Failover LAN Interface: failover Redundant1 (up)

Unit Poll frequency 3 seconds, holdtime 9 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 10 of 1025 maximum

Version: Ours 9.1(5), Mate 9.1(5)

Last Failover at: 14:14:05 CEDT Apr 21 2015

This host: Primary - Active

 

---------------------

But here the "nightmare":

Stateful Failover Logical Update Statistics

Link : failover Redundant1 (up)

Stateful Obj         xmit       xerr       rcv        rerr    

General                277015     0          500        94981    

sys cmd          486        0          486        0        

up time          0          0          0          0        

RPC services          0          0          0          0        

TCP conn         54968      0          0          23011    

UDP conn         221476     0          0          71835    

ARP tbl          85         0          13         135      

Xlate_Timeout          0          0          0          0        

IPv6 ND tbl          0          0          0          0        

VPN IKEv1 SA         0          0          0          0       

VPN IKEv1 P2         0          0          0          0        

VPN IKEv2 SA         0          0          0          0        

VPN IKEv2 P2         0          0          0          0        

VPN CTCP upd         0          0          0          0        

VPN SDI upd         0          0          0          0        

VPN DHCP upd         0          0          0          0        

SIP Session         0          0          0          0        

Route Session         0          0          0          0        

User-Identity         0          0          1          0        

CTS SGTNAME         0          0          0          0        

CTS PAC         0          0          0          0        

TrustSec-SXP         0          0          0          0        

IPv6 Route         0          0          0          0        

Logical Update Queue Information

                  Cur         Max         Total

Recv Q:         0         6         1109

Xmit Q:         0         1         278547

Similar output and errors to the other units.

 

- Interfaces under monitored are all in normal state without any issue

- Ping between two devices are 1/1/10 ms without packet lost

- Configuration syncing, switch "no fail active" and other things are working well, beside the "renegotiation of all connections", (included BGP and other session for us important for 0 down time)

- show connection on the standby unit is almost empty (beside the thousand of connection on the primary one)

 

I found this link... seems similar to my case, but I'd like to have some more confirm and other way beside the "windows" method :-\

https://supportforums.cisco.com/discussion/10889821/asa-stateful-failover-problem#comment-5433111

for now anyway, i didn't proceed with the reload of the equipments.

if you are sure about this "workaround" I can proceed. But I'd like to have more information about this.

 

If you need some other information, let me know.

In the meantime, many regards in advance

Labels:
0 Helpful
14 Replies 14

teatrodelsogno
Level 1
Level 1

‎04-22-2015 03:37 AM

Hi all,

any feedback for this?

 

regards

0 Helpful

Andre Neethling
Level 4
Level 4
In response to teatrodelsogno

‎04-22-2015 11:02 PM

HI. What seems to be the problem? Are you concerned that the connection count on your Active Unit is high, and the connection count on your Standby unit is low? If so, I think this is normal operation. The Standby unit does not process any connections. When you trigger a failover, are you experiencing issues with BGP or any other traffic?

teatrodelsogno
Level 1
Level 1
In response to Andre Neethling

‎04-23-2015 12:36 AM

Hi Andre,

and thanks in advanced for your answer.

Yes, but in the standby unit, the connection is more than low... there are null! :-)
..thousand of connection on the primary... maximum 3-4 connections on the standby.

BTW, I don't know if is it a random case that you wrote me about BGP... but this is the particularly case that we notice during the switch!! (for other services, we don't know precisely.. a lot of services there are behind this firewall)
But the main thing is that we're losing the BGP membership in the other equipments where the traffic pass through this firewall. YES

But this, it wasn't happening before... with almost the same configuration on the firewall.. (beside failover in redundancy and the migration from gigabit configured interfaces -> to -> portchannelx.x / LACP configuration. )

Then, obliviously, policy map for BGP still in the configuration:

policy-map global_policy
 class BGP-MD5-RANDOMIZE-CLASSMAP
  set connection random-sequence-number disable
  set connection advanced-options BGP-MD5-OPTION-ALLOW
service-policy global_policy global

In other, from "debug fover fail" we have a lot of:

SNP LU Could not locate input interface 4090011000000
SNP LU Could not locate output interface 4090011000000
SNP LU Could not locate input interface 4090077010000
SNP LU Could not locate input interface 409004b010000
SNP LU Could not locate input interface 4090011000000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate input interface 4090077010000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate output interface 4090063000000
SNP LU Could not locate input interface 4090032000000

And I found these cases:

https://tools.cisco.com/bugsearch/bug/CSCtf68934

https://tools.cisco.com/bugsearch/bug/CSCun12419

 

Let me know,

regards

 

0 Helpful

Andre Neethling
Level 4
Level 4
In response to teatrodelsogno

‎04-23-2015 12:54 AM

Did you enable TCP option 19 in the policy map? is your BGP notworking through the firewall at all?

I wouldn't worry too much about the amount of connections on your standby unit.

0 Helpful

teatrodelsogno
Level 1
Level 1
In response to Andre Neethling

‎04-23-2015 01:13 AM

tcp-map BGP-MD5-OPTION-ALLOW
  tcp-options range 19 19 allow

 

BTW, I'd like to underline that connectivity on the active firewall is working well.
NO issue on the connectivity on the active one.

The issue, is happening when I perform the switch on the other unit and only in this moment
I'm losing BGP membership and after few seconds (around 50sec), BGP is turned one once again.

...BTW, why you are concentrated on the BGP matter for this problem?
you are thinking something on the policy map, nat or something like this?
(keeping in mind that connectivity is everything working well if the services is on the firewall in active mode.)



Many regards

 

0 Helpful

Vibhor Amrodia
Cisco Employee
Cisco Employee
In response to teatrodelsogno

‎04-23-2015 11:14 PM

Hi,

To start the investigation on this issue , collect this output from the stanby unit:-

debug fover fail

during the time you see the error counters increment on the ASA device.

Thanks and Regards,

Vibhor Amrodia

 

0 Helpful

teatrodelsogno
Level 1
Level 1
In response to Vibhor Amrodia

‎04-24-2015 12:24 AM

Hi Vibhor,

thanks in advance for your support.

I've already did it, see above in the other answer, or here the summarize:

In other, from "debug fover fail" (only on the standby unit), we have a lot of:

SNP LU Could not locate input interface 4090011000000
SNP LU Could not locate output interface 4090011000000
SNP LU Could not locate input interface 4090077010000
SNP LU Could not locate input interface 409004b010000
SNP LU Could not locate input interface 4090011000000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate input interface 4090077010000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate output interface 4090063000000
SNP LU Could not locate input interface 4090032000000

And I found these cases:

https://tools.cisco.com/bugsearch/bug/CSCtf68934

https://tools.cisco.com/bugsearch/bug/CSCun12419

 

Regards

 

 

0 Helpful

Vibhor Amrodia
Cisco Employee
Cisco Employee
In response to teatrodelsogno

‎04-25-2015 12:53 AM

Hi,

Can you send me this output:-

show failover descriptor from both the ASA units ?

Thanks and Regards,

Vibhor Amrodia

0 Helpful

teatrodelsogno
Level 1
Level 1
In response to Vibhor Amrodia

‎04-28-2015 04:20 AM

ok!

seems for some interfaces different, for other not:

/pri/act# show failover descriptor                                     
outside            send: 00020000ffff0000  receive: 00020000ffff0000
vrf-PreProduzione  send: 000409001e000000  receive: 000409001e000000
Pub_PreProd        send: 0004090029000000  receive: 0004090029000000
Pub_Svil_e_Test    send: 000409002a000000  receive: 000409002a000000
DMZ1_SDM           send: 000409002b000000  receive: 000409002b000000
DMZ2_GDC           send: 000409002c000000  receive: 000409002c000000
vrf-Transito       send: 000409004b010000  receive: 000409004b010000
vrf-Snam           send: 000409004c010000  receive: 000409004c010000
vrf-Freezone       send: 0004090077010000  receive: 0004090077010000
VRF-SVIL_TEST      send: 0004090086010000  receive: 0004090086010000
vrf-Laboratorio    send: 0004090084030000  receive: 0004090084030000
LAB-centro_controllo1 send: 0004090085030000  receive: 0004090085030000
LAB-centro_controllo2 send: 0004090086030000  receive: 0004090086030000
LAB-campo1         send: 0004090087030000  receive: 0004090087030000
LAB-campo2         send: 0004090088030000  receive: 0004090088030000
LAB-CISCO-MNGT     send: 0004090089030000  receive: 0004090089030000
LAB-CISCO-PRODUZIONE send: 000409008a030000  receive: 000409008a030000
vrf-Produzione     send: 0004090014000000  receive: 0004090014000000
avaya              send: 0004090063000000  receive: 0004090063000000
vrf-Management     send: 0004090032000000  receive: 0004090032000000
vrf-Pubblicazione  send: 0004090011000000  receive: 0004090011000000

/sec/stby# show failover descriptor
outside            send: 00020000ffff0000  receive: 00020000ffff0000
vrf-Pubblicazione  send: 0004080011000000  receive: 0004080011000000
vrf-Produzione     send: 0004080014000000  receive: 0004080014000000
vrf-PreProduzione  send: 000408001e000000  receive: 000408001e000000
Pub_PreProd        send: 0004080029000000  receive: 0004080029000000
Pub_Svil_e_Test    send: 000408002a000000  receive: 000408002a000000
DMZ1_SDM           send: 000408002b000000  receive: 000408002b000000
DMZ2_GDC           send: 000408002c000000  receive: 000408002c000000
vrf-Management     send: 0004080032000000  receive: 0004080032000000
avaya              send: 0004080063000000  receive: 0004080063000000
vrf-Transito       send: 000408004b010000  receive: 000408004b010000
vrf-Snam           send: 000408004c010000  receive: 000408004c010000
vrf-Freezone       send: 0004080077010000  receive: 0004080077010000
VRF-SVIL_TEST      send: 0004080086010000  receive: 0004080086010000
vrf-Laboratorio    send: 0004080084030000  receive: 0004080084030000
LAB-centro_controllo1 send: 0004080085030000  receive: 0004080085030000
LAB-centro_controllo2 send: 0004080086030000  receive: 0004080086030000
LAB-campo1         send: 0004080087030000  receive: 0004080087030000
LAB-campo2         send: 0004080088030000  receive: 0004080088030000
LAB-CISCO-MNGT     send: 0004080089030000  receive: 0004080089030000
LAB-CISCO-PRODUZIONE send: 000408008a030000  receive: 000408008a030000

 

Let me know what do you think.

 

regards

0 Helpful

Vibhor Amrodia
Cisco Employee
Cisco Employee
In response to teatrodelsogno

‎04-28-2015 05:44 AM

Hi,

You can use this workaround in order to resolve this issue:-

1) Clear all the configuration from the standby unit and configure it back from the scratch including the failover commands. Rest would be replicated automatically.

2) Re-configure the interface nameif commands.

Thanks and Regards,

Vibhor Amrodia

0 Helpful

teatrodelsogno
Level 1
Level 1
In response to Vibhor Amrodia

‎04-28-2015 05:50 AM

Thanks for your support.

yes, I was thinking to do once again something like you said on 1) (then reboot of the appliance included)

BTW, what do you think for point 2) ?

usually, is needed to perform "no shut" on the physically interface and configure the failover command on the second one. What do you mean about "re-configure" interface?

 

NOTE:

all interfaces beside "outside" are within port-channel LACP sub interface.

This port-channel has been configured on two physically SFP 10gigabit on the ASA 5585.

portchannel1.x

vlan xx

etc....

 

many regards in advance

0 Helpful

fly
Level 2
Level 2
In response to teatrodelsogno

‎08-26-2015 05:41 AM

we found same issue?

Aug 26 2015 15:12:53: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:53: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:54: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:55: %ASA-3-210005: LU allocate connection failed

show failover
Failover On
Failover unit Primary
Failover LAN Interface: g0/1 GigabitEthernet0/1 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 3 seconds, holdtime 15 seconds
Interface Policy 1
Monitored Interfaces 4 of 256 maximum
Version: Ours 8.2(5), Mate 8.2(5)
Last Failover at: 18:38:11 CN Aug 25 2015
        This host: Primary - Standby Ready
                Active time: 29269 (sec)
                slot 0: ASA5550 hw/sw rev (2.0/8.2(5)) status (Up Sys)
                  Interface outside (1.6.55.250): Normal
                  Interface outside-pi-1 (1.6.4.244): Normal
                  Interface outside-cn2 (1.6.3.38): Normal (Not-Monitored)
                  Interface inside (1.6.56.249): Normal
                  Interface management (0.0.0.0): Link Down (Not-Monitored)
                  Interface inside2 (10.6.59.253): Normal
                slot 1: ASA-SSM-4GE-INC hw/sw rev (1.0/1.0(0)10) status (Up)
        Other host: Secondary - Active
                Active time: 74354 (sec)
                slot 0: ASA5550 hw/sw rev (2.0/8.2(5)) status (Up Sys)
                  Interface outside (10.6.55.251): Normal
                  Interface outside-pi-1 (1.6.4.245): Normal
                  Interface outside-cn2 (1.6.3.37): Normal (Not-Monitored)
                  Interface inside (1.6.56.254): Normal
                  Interface management (12.168.1.1): Normal (Not-Monitored)
                  Interface inside2 (1.6.59.254): Normal
                slot 1: ASA-SSM-4GE-INC hw/sw rev (1.0/1.0(0)10) status (Up)

Stateful Failover Logical Update Statistics
        Link : g0/3 GigabitEthernet0/3 (up)
        Stateful Obj    xmit       xerr       rcv        rerr      
        General         15803341   0          25818616   28332     
        sys cmd         13813      0          13813      1         
        up time         0          0          0          0         
        RPC services    0          0          0          0         
        TCP conn        12499770   0          18737420   8810      
        UDP conn        3282047    0          7051875    19521     
        ARP tbl         7711       0          15508      0         
        Xlate_Timeout   0          0          0          0         
        IPv6 ND tbl     0          0          0          0         
        VPN IKE upd     0          0          0          0         
        VPN IPSEC upd   0          0          0          0         
        VPN CTCP upd    0          0          0          0         
        VPN SDI upd     0          0          0          0         
        VPN DHCP upd    0          0          0          0         
        SIP Session     0          0          0          0         

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       17      25996573
        Xmit Q:         0       1469    15834078

 

0 Helpful

teatrodelsogno
Level 1
Level 1
In response to fly

‎08-26-2015 07:52 AM

NO problem Fly.

Cisco updated the related bug, following this issue.

(I don't remember exacltly right now the number of the bug).

 

BTW You need at the end update ASA to new version as new as you can.
I Saw that your firewall is really old version.

Update is needed for this and other possibily bugs.

 

Regards

0 Helpful

fly
Level 2
Level 2

‎08-26-2015 05:39 AM

 

we found same problem. like below from standbym we try to reload standby

Aug 26 2015 15:12:53: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:53: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:54: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:55: %ASA-3-210005: LU allocate connection failed

Stateful Failover Logical Update Statistics
        Link : g0/3 GigabitEthernet0/3 (up)
        Stateful Obj    xmit       xerr       rcv        rerr      
        General         15803341   0          25818616   28332     
        sys cmd         13813      0          13813      1         
        up time         0          0          0          0         
        RPC services    0          0          0          0         
        TCP conn        12499770   0          18737420   8810      
        UDP conn        3282047    0          7051875    19521     
        ARP tbl         7711       0          15508      0         
        Xlate_Timeout   0          0          0          0         
        IPv6 ND tbl     0          0          0          0         
        VPN IKE upd     0          0          0          0         
        VPN IPSEC upd   0          0          0          0         
        VPN CTCP upd    0          0          0          0         
        VPN SDI upd     0          0          0          0        

0 Helpful
Customers Also Viewed These Support Documents
Review Cisco Networking for a $25 gift card