CSCvd78303 - ARP functions fail after 213 days of uptime, drop with error and39;punt-rate-limit-exceeded'

[Evgeniy]

Is there such a bug in firmware 9.4 (3) 12? We need expert opinion.

[Rob Huffman]

Hi Evgeniy,

Yes, that version would be affected;

Conditions:
This is seen when the ASA's uptime reaches 213 days.

This problem affects ASA and FTD versions:
ASA version 9.1 releases 9.1(7)8 and higher
ASA version 9.2 releases 9.2(4)15 and higher
ASA version 9.4 releases 9.4(3)5 and higher including 9.4(4)
ASA version 9.5 releases 9.5(3) and higher
ASA version 9.6 releases 9.6(2)1 and higher including 9.6(3)
ASA version 9.7 releases 9.7(1) and higher
FTD version 6.1 releases 6.1.0.1 and higher
FTD version 6.2 releases 6.2.0 and higher

And from the recently updated release notes for 9.4.x

Important Notes

• Potential Traffic Outage (9.4(3.11) through 9.4(4))—Due to bug CSCvd78303, the ASA may stop passing traffic after 213 days of uptime. The effect on each network will be different, but it could range from an issue of limited connectivity to something more extensive like an outage. You must upgrade to a new version without this bug, when available. In the meantime, you can reboot the ASA to gain another 213 days of uptime. Other workarounds may be available. See Field Notice FN-64291 for affected versions and more information.

From;

http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/release/notes/asarn94.html

[efrazee]

We are running effected version 9.4(3)12 and our up time was 209 days. We believe we hit this bug. Has anyone heard of this bug being triggered before 213 days?

[mbluemel]

I manage a number of customer ASA's and I have seen this at least 6 times now and it has always been 213 days and various hours. The 213 days seems to be a constant.

[shawnwelsh]

We appear to have been impacted at 221 days so I would say that 213 is not exact science. I've ruled out anything else that could have been wrong. 

[nb-taos]

What would this bug's effect on an HA failover pair be?

I'm investigating an ASA HA pair outage, on 5585s that were running 9.4(3)12, configured for Active/Standby HA and stateful failover.

The Primary had all the earmarks of this bug (no ARP table, console still responsive). The engineers report that the Secondary was still responsive, except the failover link.  (Not sure the ARP table was checked on the Secondary during the crisis.)

This bug would have caused the failover link to miss hello keepalives, which would then cause the Secondary to start testing the monitored data interfaces.  Link up would have passed (all interfaces attached to switches), but hello's and ping tests would have failed.   So the Secondary should have moved to Active -- but it didn't until the Primary was uncabled and completely powered off. 

Would this bug trigger an HA failover from Primary to Secondary?

Does this bug affect the Primary/Secondary equally, or only one?