Anyone who manages ASA or FTD firewall with 98K+ cps, did you ever notice that connection replication can start failing sometimes and the number of successfully replicated connections drops on the standby unit? At the same time conn rate increases on the standby and becomes much higher than on the active unit. E.g., conn rate on the active unit:
And the conn rate on the standby (conn rate of connections being replicated over state link):
This was confirmed from CLI too, as shown below. Notice that the number of conns drops from 9M to 5M on the standby, while it doesn't change on the active unit (9M).
The trigger is not known, although it seems this issue appears when the conn rate exceeds certain threshold. We observed this issue at the end of Dec every day. In Jan conn rate is lower and the issue is no longer observed (so far).
The platform is Firepower 4145 running ASA 9.12.4.47. Almost all connections are HTTPS. There are few which are HTTP and hence not replicated, that is why the number of conns on the standby is usually smaller than on active. The state link has 10G bandwidth and the traffic rate over the state link slightly exceeds 1G.
There are no drops on either active, standby or switches, whatsoever, on either Internal-Data or in ASP path. No indication of high CPU or big CPU hogs, although, when the issue happens, CPU increases a bit on the standby, from 10%-15% to 12%-17% (see below) and hyper-threading appears to be activated, although it should not.
! 10:59 active
Resource Current Peak Limit Denied Context
Syslogs [rate] 0 4651 unlimited 0 CONTEXT
Conns 8993920 10218204 unlimited 0 CONTEXT
Xlates 7249 31730 unlimited 0 CONTEXT
Hosts 1530358 2341219 unlimited 0 CONTEXT
Conns [rate] 98186 249999 250000 28011 CONTEXT
Inspects [rate] 1 1478 unlimited 0 CONTEXT
Routes 16 16 unlimited 0 CONTEXT
! 10:59 standby
Resource Current Peak Limit Denied Context
Syslogs [rate] 0 3964 unlimited 0 CONTEXT
Conns 8941370 9915711 unlimited 0 CONTEXT
Xlates 7229 30622 unlimited 0 CONTEXT
Hosts 1527042 1533170 unlimited 0 CONTEXT
Conns [rate] 94388 153485 250000 0 CONTEXT
Inspects [rate] 0 111 unlimited 0 CONTEXT
Routes 14 14 unlimited 0 CONTEXT
! 11:00 active
Resource Current Peak Limit Denied Context
Syslogs [rate] 0 4651 unlimited 0 CONTEXT
Conns 9016952 10218204 unlimited 0 CONTEXT
Xlates 7231 31730 unlimited 0 CONTEXT
Hosts 1532534 2341219 unlimited 0 CONTEXT
Conns [rate] 98329 249999 250000 28011 CONTEXT
Inspects [rate] 0 1478 unlimited 0 CONTEXT
Routes 16 16 unlimited 0 CONTEXT
! 11:00 standby
Resource Current Peak Limit Denied Context
Syslogs [rate] 0 3964 unlimited 0 CONTEXT
Conns 8937989 9915711 unlimited 0 CONTEXT
Xlates 7266 30622 unlimited 0 CONTEXT
Hosts 1527402 1533170 unlimited 0 CONTEXT
Conns [rate] 118479 153485 250000 0 CONTEXT
Inspects [rate] 0 111 unlimited 0 CONTEXT
Routes 14 14 unlimited 0 CONTEXT
! 11:01 active
Resource Current Peak Limit Denied Context
Syslogs [rate] 0 4651 unlimited 0 CONTEXT
Conns 9040104 10218204 unlimited 0 CONTEXT
Xlates 6873 31730 unlimited 0 CONTEXT
Hosts 1534653 2341219 unlimited 0 CONTEXT
Conns [rate] 99311 249999 250000 28011 CONTEXT
Inspects [rate] 0 1478 unlimited 0 CONTEXT
Routes 16 16 unlimited 0 CONTEXT
! 11:01 standby
Resource Current Peak Limit Denied Context
Syslogs [rate] 4 3964 unlimited 0 CONTEXT
Conns 8761384 9915711 unlimited 0 CONTEXT
Xlates 6857 30622 unlimited 0 CONTEXT
Hosts 1523207 1533170 unlimited 0 CONTEXT
Conns [rate] 132348 153485 250000 0 CONTEXT
Inspects [rate] 0 111 unlimited 0 CONTEXT
Routes 14 14 unlimited 0 CONTEXT
! 11:02 active
Resource Current Peak Limit Denied Context
Syslogs [rate] 0 4651 unlimited 0 CONTEXT
Conns 9047853 10218204 unlimited 0 CONTEXT
Xlates 7481 31730 unlimited 0 CONTEXT
Hosts 1534761 2341219 unlimited 0 CONTEXT
Conns [rate] 98881 249999 250000 28011 CONTEXT
Inspects [rate] 1 1478 unlimited 0 CONTEXT
Routes 16 16 unlimited 0 CONTEXT
! 11:02 standby
Resource Current Peak Limit Denied Context
Syslogs [rate] 0 3964 unlimited 0 CONTEXT
Conns 8455533 9915711 unlimited 0 CONTEXT
Xlates 7449 30622 unlimited 0 CONTEXT
Hosts 1511153 1533170 unlimited 0 CONTEXT
Conns [rate] 134675 153485 250000 0 CONTEXT
Inspects [rate] 0 111 unlimited 0 CONTEXT
Routes 14 14 unlimited 0 CONTEXT
...
! 11:20 active
Resource Current Peak Limit Denied Context
Syslogs [rate] 33 4651 unlimited 0 CONTEXT
Conns 9134536 10218204 unlimited 0 CONTEXT
Xlates 7449 31730 unlimited 0 CONTEXT
Hosts 1543073 2341219 unlimited 0 CONTEXT
Conns [rate] 98303 249999 250000 28011 CONTEXT
Inspects [rate] 0 1478 unlimited 0 CONTEXT
Routes 16 16 unlimited 0 CONTEXT
! 11:20 standby
Resource Current Peak Limit Denied Context
Syslogs [rate] 0 3964 unlimited 0 CONTEXT
Conns 5249317 9915711 unlimited 0 CONTEXT
Xlates 7429 30622 unlimited 0 CONTEXT
Hosts 1140899 1533170 unlimited 0 CONTEXT
Conns [rate] 147060 153485 250000 0 CONTEXT
Inspects [rate] 0 111 unlimited 0 CONTEXT
Routes 14 14 unlimited 0 CONTEXT
! standby CPU
Break down of per-core data path versus control point cpu usage:
Core 5 sec 1 min 5 min
Core 0 7.2 ( 7.0 + 0.2) 7.6 ( 6.9 + 0.6) 7.6 ( 6.8 + 0.7)
Core 1 13.0 (13.0 + 0.0) 12.8 (12.8 + 0.0) 12.7 (12.7 + 0.0)
Core 2 12.0 (12.0 + 0.0) 11.9 (11.9 + 0.0) 11.8 (11.8 + 0.0)
Core 3 12.0 (12.0 + 0.0) 11.9 (11.9 + 0.0) 11.8 (11.8 + 0.0)
Core 4 11.8 (11.8 + 0.0) 12.0 (12.0 + 0.0) 12.0 (12.0 + 0.0)
Core 5 11.8 (11.8 + 0.0) 11.9 (11.9 + 0.0) 11.8 (11.8 + 0.0)
Core 6 12.0 (12.0 + 0.0) 12.0 (12.0 + 0.0) 12.0 (12.0 + 0.0)
Core 7 11.8 (11.8 + 0.0) 11.8 (11.8 + 0.0) 11.8 (11.8 + 0.0)
Core 8 11.8 (11.8 + 0.0) 11.7 (11.7 + 0.0) 11.6 (11.6 + 0.0)
Core 9 11.8 (11.8 + 0.0) 11.8 (11.8 + 0.0) 11.8 (11.8 + 0.0)
Core 10 12.0 (12.0 + 0.0) 12.0 (12.0 + 0.0) 11.9 (11.9 + 0.0)
Core 11 11.6 (11.6 + 0.0) 11.7 (11.7 + 0.0) 11.6 (11.6 + 0.0)
Core 12 12.0 (12.0 + 0.0) 12.1 (12.1 + 0.0) 12.0 (12.0 + 0.0)
Core 13 12.0 (12.0 + 0.0) 12.1 (12.1 + 0.0) 12.0 (12.0 + 0.0)
Core 14 11.8 (11.8 + 0.0) 11.8 (11.8 + 0.0) 11.8 (11.8 + 0.0)
Core 15 12.0 (12.0 + 0.0) 12.0 (12.0 + 0.0) 12.0 (12.0 + 0.0)
Core 16 11.6 (11.6 + 0.0) 11.6 (11.6 + 0.0) 11.6 (11.6 + 0.0)
Core 17 11.8 (11.8 + 0.0) 12.0 (12.0 + 0.0) 12.0 (12.0 + 0.0)
Core 18 12.2 (12.2 + 0.0) 12.1 (12.1 + 0.0) 12.0 (12.0 + 0.0)
Core 19 11.8 (11.8 + 0.0) 12.0 (12.0 + 0.0) 11.9 (11.9 + 0.0)
Core 20 11.8 (11.8 + 0.0) 11.9 (11.9 + 0.0) 11.8 (11.8 + 0.0)
Core 21 12.6 (12.2 + 0.4) 12.9 (12.4 + 0.5) 12.9 (12.3 + 0.5)
Core 22 16.6 (16.6 + 0.0) 17.0 (17.0 + 0.0) 16.9 (16.9 + 0.0)
Core 23 17.2 (17.2 + 0.0) 17.5 (17.5 + 0.0) 17.4 (17.4 + 0.0)
Core 24 16.8 (16.8 + 0.0) 17.0 (17.0 + 0.0) 16.9 (16.9 + 0.0)
Core 25 16.8 (16.8 + 0.0) 17.1 (17.1 + 0.0) 17.0 (17.0 + 0.0)
Core 26 16.8 (16.8 + 0.0) 17.0 (17.0 + 0.0) 16.9 (16.9 + 0.0)
Core 27 17.8 (17.8 + 0.0) 17.8 (17.8 + 0.0) 17.6 (17.6 + 0.0)
Core 28 17.0 (17.0 + 0.0) 17.1 (17.1 + 0.0) 17.0 (17.0 + 0.0)
Core 29 17.0 (17.0 + 0.0) 17.2 (17.2 + 0.0) 17.1 (17.1 + 0.0)
Core 30 16.8 (16.8 + 0.0) 17.2 (17.2 + 0.0) 17.0 (17.0 + 0.0)
Core 31 17.0 (17.0 + 0.0) 17.1 (17.1 + 0.0) 17.0 (17.0 + 0.0)
Core 32 17.0 (17.0 + 0.0) 17.0 (17.0 + 0.0) 16.9 (16.9 + 0.0)
Core 33 17.0 (17.0 + 0.0) 17.0 (17.0 + 0.0) 17.0 (17.0 + 0.0)
Core 34 17.0 (17.0 + 0.0) 17.2 (17.2 + 0.0) 17.0 (17.0 + 0.0)
Core 35 17.0 (17.0 + 0.0) 17.1 (17.1 + 0.0) 17.0 (17.0 + 0.0)
Core 36 17.0 (17.0 + 0.0) 17.2 (17.2 + 0.0) 17.1 (17.1 + 0.0)
Core 37 16.8 (16.8 + 0.0) 17.0 (17.0 + 0.0) 16.9 (16.9 + 0.0)
Core 38 16.6 (16.6 + 0.0) 16.9 (16.9 + 0.0) 16.8 (16.8 + 0.0)
Core 39 16.8 (16.8 + 0.0) 17.3 (17.3 + 0.0) 17.2 (17.2 + 0.0)
Core 40 17.2 (17.2 + 0.0) 17.3 (17.3 + 0.0) 17.2 (17.2 + 0.0)
Core 41 16.8 (16.8 + 0.0) 17.1 (17.1 + 0.0) 17.0 (17.0 + 0.0)
Core 42 16.8 (16.8 + 0.0) 17.1 (17.1 + 0.0) 16.9 (16.9 + 0.0)
Core 43 0.0 ( 0.0 + 0.0) 0.3 ( 0.1 + 0.2) 0.0 ( 0.0 + 0.0)
Core 44 0.0 ( 0.0 + 0.0) 0.2 ( 0.2 + 0.0) 0.0 ( 0.0 + 0.0)
Core 45 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 46 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 47 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 48 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 49 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 50 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 51 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 52 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 53 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 54 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 55 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 56 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 57 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 58 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 59 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 60 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 61 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 62 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 63 0.0 ( 0.0 + 0.0) 0.1 ( 0.1 + 0.0) 0.0 ( 0.0 + 0.0)
Core 64 0.0 ( 0.0 + 0.0) 0.5 ( 0.4 + 0.1) 0.3 ( 0.0 + 0.0)
Core 65 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 66 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 67 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 68 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 69 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 70 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 71 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 72 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 73 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 74 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 75 0.0 ( 0.0 + 0.0) 0.3 ( 0.3 + 0.0) 0.0 ( 0.0 + 0.0)
Core 76 0.0 ( 0.0 + 0.0) 0.3 ( 0.3 + 0.0) 0.0 ( 0.0 + 0.0)
Core 77 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 78 0.0 ( 0.0 + 0.0) 0.3 ( 0.3 + 0.0) 0.0 ( 0.0 + 0.0)
Core 79 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 80 0.0 ( 0.0 + 0.0) 0.3 ( 0.3 + 0.0) 0.0 ( 0.0 + 0.0)
Core 81 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 82 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 83 0.0 ( 0.0 + 0.0) 0.3 ( 0.3 + 0.0) 0.0 ( 0.0 + 0.0)
Core 84 0.0 ( 0.0 + 0.0) 0.4 ( 0.4 + 0.0) 0.0 ( 0.0 + 0.0)
Core 85 0.0 ( 0.0 + 0.0) 0.3 ( 0.3 + 0.0) 0.0 ( 0.0 + 0.0)
Current control plane usage versus the control plane cores elapsed for:
5 seconds = 0.6%; 1 minute: 1.5%; 5 minutes: 1.7%
CPU utilization of external processes for:
5 seconds = 0.0%; 1 minute: 0.0%; 5 minutes: 0.0%
Total CPU utilization for:
5 seconds = 7.2%; 1 minute: 7.4%; 5 minutes: 7.2%
