CSCvv87495 - FMC unresponsive no SSH, no GUI - Error 500, Console init: cannot fork, retry.. add 6.6.1 to affected versions
I encountered this bug on Friday 5th March 2021, after upgrading a HA pair of FMC1600's from 18.104.22.168 to 6.6.1-91, approx 15hrs after the upgrade was complete & everything was working fine, the HA sync failed due to loss of SSL between units caused by this bug.
Having discovered the loss of GUI & SSH access, we sent an engineer to site as Cisco TAC wanted to diagnose the fault, as did I. Console access when we were finally able to get an engineer to the lights out DC showed the following error
init: cannot fork, retry..
This was repeated on the screen pretty much continuously stopping us logging in. When I searched for this error, the bug CSCvv87495 was the first entry in the search, which matched our symptoms exactly & the only thing that changed during the upgrade was the new login page & GUI look, also offered at login to activate "Cisco Support Diagnostics" which I thought would be a good idea, turned out I was wrong
Cisco need to update the 6.6.1 release notes ! Located here
In Version 6.5.0+, Cisco Support Diagnostics (sometimes called Cisco Proactive Support) sends configuration and operational health data to Cisco, and processes that data through our automated problem detection system, allowing us to proactively notify you of issues. This feature also allows Cisco TAC to collect essential information from your devices during the course of a TAC case.
During initial setup and upgrades, you may be asked to accept or decline participation. You can also opt in or out at any time."
I asked TAC to check this twice, but did NOT get a response. During the whole issue (approx 24hrs) the Primary FMC was still communicating with Cisco, was just not accessible locally. The fix was to reboot the Primary FMC, await its long return (circa 25minutes) to an operational state, plus 16minutes for a DB sync with the Secondary FMC, make it active. Then disable the Cisco Support Diagnostics under SYSTEM>Licences>Smart Licenses.
Guided workflow allows for automated troubleshooting cases (non-RMA) and RMA creation process for Service Requests (SR) opened for Cisco products.
Less troubleshooting - the IRE (Intelligent RMA Experience) Prediction Engine does ...
1. Asset Recovery EMEAR - Scope2. Cisco Returns Portal – Quick Overview3. Update your RMA – Options and Walkthrough4. POWR Tool and How to Return - Quick guide and Scenarios5. Returned but not closed - Frequent Scenarios6. I cannot return my faulty part. ...
Full analysis if the problem
The root cause behind this problem is the delay in sync between CUCM and CCX.
CCX and CUCM has an automated sync mechanism which might vary from 5 - 10 minutes based on various parameters.
Also there is an option to sync...