NO problem Fly.Cisco updated

teatrodelsogno · ‎04-21-2015

Hi Everybody,

I have this strange and interesting problem on the failover ASA configuration.

Background information:

ASA5585-SSP-20

Version 9.1(5)

ASA5585-SSP-20, 12288 MB RAM, CPU Xeon 5500 series 2133 MHz, 1 CPU (8 cores)

Free memory: 9636244336 bytes (75%)
Used memory: 3248657552 bytes (25%)

Before I used just one management port for the failover connectivity and it was working well.

Now the only change, is the gbit port and the redundant configuration. (for the failover configuration part)

Configuration "extract"

interface red1
member-interface g0/6
member-interface g0/7

failover lan unit secondary
failover lan interface failover red1
failover link failover red1
failover polltime unit 3 holdtime 9
failover interface ip failover 192.168.0.1 255.255.255.252 standby 192.168.0.2
failover

failover lan unit primary
failover lan interface failover red1
failover link failover red1
failover polltime unit 3 holdtime 9
failover interface ip failover 192.168.0.1 255.255.255.252 standby 192.168.0.2
failover

Problem

Seems that failover connection are not correctly propagated on the falover units!

If I type sh connection on the stdby unit, I'm able to see just few connections against thousand of connections on the active one.

But the failover system seems working good:

failover exec standby show failover

Failover On

Failover unit Secondary

Failover LAN Interface: failover Redundant1 (up)

Unit Poll frequency 3 seconds, holdtime 9 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 10 of 1025 maximum

Version: Ours 9.1(5), Mate 9.1(5)

Last Failover at: 14:14:05 CEDT Apr 21 2015

This host: Secondary - Standby Ready

Failover On

Failover unit Primary

Failover LAN Interface: failover Redundant1 (up)

Unit Poll frequency 3 seconds, holdtime 9 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 10 of 1025 maximum

Version: Ours 9.1(5), Mate 9.1(5)

Last Failover at: 14:14:05 CEDT Apr 21 2015

This host: Primary - Active

---------------------

But here the "nightmare":

Stateful Failover Logical Update Statistics

Link : failover Redundant1 (up)

Stateful Obj xmit xerr rcv rerr

General 277015 0 500 94981

sys cmd 486 0 486 0

up time 0 0 0 0

RPC services 0 0 0 0

TCP conn 54968 0 0 23011

UDP conn 221476 0 0 71835

ARP tbl 85 0 13 135

Xlate_Timeout 0 0 0 0

IPv6 ND tbl 0 0 0 0

VPN IKEv1 SA 0 0 0 0

VPN IKEv1 P2 0 0 0 0

VPN IKEv2 SA 0 0 0 0

VPN IKEv2 P2 0 0 0 0

VPN CTCP upd 0 0 0 0

VPN SDI upd 0 0 0 0

VPN DHCP upd 0 0 0 0

SIP Session 0 0 0 0

Route Session 0 0 0 0

User-Identity 0 0 1 0

CTS SGTNAME 0 0 0 0

CTS PAC 0 0 0 0

TrustSec-SXP 0 0 0 0

IPv6 Route 0 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 6 1109

Xmit Q: 0 1 278547

Similar output and errors to the other units.

- Interfaces under monitored are all in normal state without any issue

- Ping between two devices are 1/1/10 ms without packet lost

- Configuration syncing, switch "no fail active" and other things are working well, beside the "renegotiation of all connections", (included BGP and other session for us important for 0 down time)

- show connection on the standby unit is almost empty (beside the thousand of connection on the primary one)

I found this link... seems similar to my case, but I'd like to have some more confirm and other way beside the "windows" method :-\

https://supportforums.cisco.com/discussion/10889821/asa-stateful-failover-problem#comment-5433111

for now anyway, i didn't proceed with the reload of the equipments.

if you are sure about this "workaround" I can proceed. But I'd like to have more information about this.

If you need some other information, let me know.

In the meantime, many regards in advance

teatrodelsogno · ‎04-22-2015

Hi all,

any feedback for this?

regards

Andre Neethling · ‎04-22-2015

HI. What seems to be the problem? Are you concerned that the connection count on your Active Unit is high, and the connection count on your Standby unit is low? If so, I think this is normal operation. The Standby unit does not process any connections. When you trigger a failover, are you experiencing issues with BGP or any other traffic?

teatrodelsogno · ‎04-23-2015

Hi Andre,

and thanks in advanced for your answer.

Yes, but in the standby unit, the connection is more than low... there are null! :-)
..thousand of connection on the primary... maximum 3-4 connections on the standby.

BTW, I don't know if is it a random case that you wrote me about BGP... but this is the particularly case that we notice during the switch!! (for other services, we don't know precisely.. a lot of services there are behind this firewall)
But the main thing is that we're losing the BGP membership in the other equipments where the traffic pass through this firewall. YES

But this, it wasn't happening before... with almost the same configuration on the firewall.. (beside failover in redundancy and the migration from gigabit configured interfaces -> to -> portchannelx.x / LACP configuration. )

Then, obliviously, policy map for BGP still in the configuration:

policy-map global_policy
class BGP-MD5-RANDOMIZE-CLASSMAP
set connection random-sequence-number disable
set connection advanced-options BGP-MD5-OPTION-ALLOW
service-policy global_policy global

In other, from "debug fover fail" we have a lot of:

SNP LU Could not locate input interface 4090011000000
SNP LU Could not locate output interface 4090011000000
SNP LU Could not locate input interface 4090077010000
SNP LU Could not locate input interface 409004b010000
SNP LU Could not locate input interface 4090011000000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate input interface 4090077010000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate output interface 4090063000000
SNP LU Could not locate input interface 4090032000000

And I found these cases:

https://tools.cisco.com/bugsearch/bug/CSCtf68934

https://tools.cisco.com/bugsearch/bug/CSCun12419

Let me know,

regards

Andre Neethling · ‎04-23-2015

Did you enable TCP option 19 in the policy map? is your BGP notworking through the firewall at all?

I wouldn't worry too much about the amount of connections on your standby unit.

teatrodelsogno · ‎04-23-2015

tcp-map BGP-MD5-OPTION-ALLOW
tcp-options range 19 19 allow

BTW, I'd like to underline that connectivity on the active firewall is working well.
NO issue on the connectivity on the active one.

The issue, is happening when I perform the switch on the other unit and only in this moment
I'm losing BGP membership and after few seconds (around 50sec), BGP is turned one once again.

...BTW, why you are concentrated on the BGP matter for this problem?
you are thinking something on the policy map, nat or something like this?
(keeping in mind that connectivity is everything working well if the services is on the firewall in active mode.)

Many regards

Vibhor Amrodia · ‎04-23-2015

Hi,

To start the investigation on this issue , collect this output from the stanby unit:-

debug fover fail

during the time you see the error counters increment on the ASA device.

Thanks and Regards,

Vibhor Amrodia

teatrodelsogno · ‎04-24-2015

Hi Vibhor,

thanks in advance for your support.

I've already did it, see above in the other answer, or here the summarize:

In other, from "debug fover fail" (only on the standby unit), we have a lot of:

SNP LU Could not locate input interface 4090011000000
SNP LU Could not locate output interface 4090011000000
SNP LU Could not locate input interface 4090077010000
SNP LU Could not locate input interface 409004b010000
SNP LU Could not locate input interface 4090011000000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate input interface 4090077010000
SNP LU Could not locate input interface 4090032000000
SNP LU Could not locate output interface 4090063000000
SNP LU Could not locate input interface 4090032000000

And I found these cases:

https://tools.cisco.com/bugsearch/bug/CSCtf68934

https://tools.cisco.com/bugsearch/bug/CSCun12419

Regards

Vibhor Amrodia · ‎04-25-2015

Hi,

Can you send me this output:-

show failover descriptor from both the ASA units ?

Thanks and Regards,

Vibhor Amrodia

teatrodelsogno · ‎04-28-2015

ok!

seems for some interfaces different, for other not:

/pri/act# show failover descriptor
outside            send: 00020000ffff0000 receive: 00020000ffff0000
vrf-PreProduzione send: 000409001e000000 receive: 000409001e000000
Pub_PreProd        send: 0004090029000000 receive: 0004090029000000
Pub_Svil_e_Test    send: 000409002a000000 receive: 000409002a000000
DMZ1_SDM           send: 000409002b000000 receive: 000409002b000000
DMZ2_GDC           send: 000409002c000000 receive: 000409002c000000
vrf-Transito       send: 000409004b010000 receive: 000409004b010000
vrf-Snam           send: 000409004c010000 receive: 000409004c010000
vrf-Freezone       send: 0004090077010000 receive: 0004090077010000
VRF-SVIL_TEST      send: 0004090086010000 receive: 0004090086010000
vrf-Laboratorio    send: 0004090084030000 receive: 0004090084030000
LAB-centro_controllo1 send: 0004090085030000 receive: 0004090085030000
LAB-centro_controllo2 send: 0004090086030000 receive: 0004090086030000
LAB-campo1         send: 0004090087030000 receive: 0004090087030000
LAB-campo2         send: 0004090088030000 receive: 0004090088030000
LAB-CISCO-MNGT     send: 0004090089030000 receive: 0004090089030000
LAB-CISCO-PRODUZIONE send: 000409008a030000 receive: 000409008a030000
vrf-Produzione     send: 0004090014000000 receive: 0004090014000000
avaya              send: 0004090063000000 receive: 0004090063000000
vrf-Management     send: 0004090032000000 receive: 0004090032000000
vrf-Pubblicazione send: 0004090011000000 receive: 0004090011000000

/sec/stby# show failover descriptor
outside            send: 00020000ffff0000 receive: 00020000ffff0000
vrf-Pubblicazione send: 0004080011000000 receive: 0004080011000000
vrf-Produzione     send: 0004080014000000 receive: 0004080014000000
vrf-PreProduzione send: 000408001e000000 receive: 000408001e000000
Pub_PreProd        send: 0004080029000000 receive: 0004080029000000
Pub_Svil_e_Test    send: 000408002a000000 receive: 000408002a000000
DMZ1_SDM           send: 000408002b000000 receive: 000408002b000000
DMZ2_GDC           send: 000408002c000000 receive: 000408002c000000
vrf-Management     send: 0004080032000000 receive: 0004080032000000
avaya              send: 0004080063000000 receive: 0004080063000000
vrf-Transito       send: 000408004b010000 receive: 000408004b010000
vrf-Snam           send: 000408004c010000 receive: 000408004c010000
vrf-Freezone       send: 0004080077010000 receive: 0004080077010000
VRF-SVIL_TEST      send: 0004080086010000 receive: 0004080086010000
vrf-Laboratorio    send: 0004080084030000 receive: 0004080084030000
LAB-centro_controllo1 send: 0004080085030000 receive: 0004080085030000
LAB-centro_controllo2 send: 0004080086030000 receive: 0004080086030000
LAB-campo1         send: 0004080087030000 receive: 0004080087030000
LAB-campo2         send: 0004080088030000 receive: 0004080088030000
LAB-CISCO-MNGT     send: 0004080089030000 receive: 0004080089030000
LAB-CISCO-PRODUZIONE send: 000408008a030000 receive: 000408008a030000

Let me know what do you think.

regards

Vibhor Amrodia · ‎04-28-2015

Hi,

You can use this workaround in order to resolve this issue:-

1) Clear all the configuration from the standby unit and configure it back from the scratch including the failover commands. Rest would be replicated automatically.

2) Re-configure the interface nameif commands.

Thanks and Regards,

Vibhor Amrodia

teatrodelsogno · ‎04-28-2015

Thanks for your support.

yes, I was thinking to do once again something like you said on 1) (then reboot of the appliance included)

BTW, what do you think for point 2) ?

usually, is needed to perform "no shut" on the physically interface and configure the failover command on the second one. What do you mean about "re-configure" interface?

NOTE:

all interfaces beside "outside" are within port-channel LACP sub interface.

This port-channel has been configured on two physically SFP 10gigabit on the ASA 5585.

portchannel1.x

vlan xx

etc....

many regards in advance

fly · ‎08-26-2015

we found same issue?

Aug 26 2015 15:12:53: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:53: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:54: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:55: %ASA-3-210005: LU allocate connection failed

show failover
Failover On
Failover unit Primary
Failover LAN Interface: g0/1 GigabitEthernet0/1 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 3 seconds, holdtime 15 seconds
Interface Policy 1
Monitored Interfaces 4 of 256 maximum
Version: Ours 8.2(5), Mate 8.2(5)
Last Failover at: 18:38:11 CN Aug 25 2015
        This host: Primary - Standby Ready
                Active time: 29269 (sec)
                slot 0: ASA5550 hw/sw rev (2.0/8.2(5)) status (Up Sys)
                  Interface outside (1.6.55.250): Normal
                  Interface outside-pi-1 (1.6.4.244): Normal
                  Interface outside-cn2 (1.6.3.38): Normal (Not-Monitored)
                  Interface inside (1.6.56.249): Normal
                  Interface management (0.0.0.0): Link Down (Not-Monitored)
                  Interface inside2 (10.6.59.253): Normal
                slot 1: ASA-SSM-4GE-INC hw/sw rev (1.0/1.0(0)10) status (Up)
        Other host: Secondary - Active
                Active time: 74354 (sec)
                slot 0: ASA5550 hw/sw rev (2.0/8.2(5)) status (Up Sys)
                  Interface outside (10.6.55.251): Normal
                  Interface outside-pi-1 (1.6.4.245): Normal
                  Interface outside-cn2 (1.6.3.37): Normal (Not-Monitored)
                  Interface inside (1.6.56.254): Normal
                  Interface management (12.168.1.1): Normal (Not-Monitored)
                  Interface inside2 (1.6.59.254): Normal
                slot 1: ASA-SSM-4GE-INC hw/sw rev (1.0/1.0(0)10) status (Up)

Stateful Failover Logical Update Statistics
        Link : g0/3 GigabitEthernet0/3 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         15803341   0          25818616   28332
        sys cmd         13813      0          13813      1
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        12499770   0          18737420   8810
        UDP conn        3282047    0          7051875    19521
        ARP tbl         7711       0          15508      0
        Xlate_Timeout   0          0          0          0
        IPv6 ND tbl     0          0          0          0
        VPN IKE upd     0          0          0          0
        VPN IPSEC upd   0          0          0          0
        VPN CTCP upd    0          0          0          0
        VPN SDI upd     0          0          0          0
        VPN DHCP upd    0          0          0          0
        SIP Session     0          0          0          0

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       17      25996573
        Xmit Q:         0       1469    15834078

teatrodelsogno · ‎08-26-2015

NO problem Fly.

Cisco updated the related bug, following this issue.

(I don't remember exacltly right now the number of the bug).

BTW You need at the end update ASA to new version as new as you can.
I Saw that your firewall is really old version.

Update is needed for this and other possibily bugs.

Regards

fly · ‎08-26-2015

we found same problem. like below from standbym we try to reload standby

Aug 26 2015 15:12:53: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:53: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:54: %ASA-3-210005: LU allocate connection failed
Aug 26 2015 15:12:55: %ASA-3-210005: LU allocate connection failed

Stateful Failover Logical Update Statistics
        Link : g0/3 GigabitEthernet0/3 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         15803341   0          25818616   28332
        sys cmd         13813      0          13813      1
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        12499770   0          18737420   8810
        UDP conn        3282047    0          7051875    19521
        ARP tbl         7711       0          15508      0
        Xlate_Timeout   0          0          0          0
        IPv6 ND tbl     0          0          0          0
        VPN IKE upd     0          0          0          0
        VPN IPSEC upd   0          0          0          0
        VPN CTCP upd    0          0          0          0
        VPN SDI upd     0          0          0          0

Cisco ASA Failover stateful error! Strange problem tcp udp replica