FMC GUI is stuck in "System processes are starting"

[Ravi Thakur]

Unable to login to FMC GUI, following processes are showing down on FMC CLI:

root@FMC01:/Volume/home/ravi thakur# pmtool status | grep -i gui
mysqld (system,gui,mysql) - Running 29318
monetdb (system,gui) - Started
httpsd (system,gui) - Running 29339
sybase_arbiter (system,gui) - Running 29341
vmsDbEngine (system,gui) - Down
ESS (system,gui) - Running 29990
DCCSM (system,gui) - Down
Tomcat (system,gui) - Down
VmsBackendServer (system,gui) - Down
mojo_server (system,gui) - Running 30168

As per some of the previous posts, restarted the process showing as Down by using RestartByID but dint work.

Have tried command "/etc/rc.d/init.d/console restart"

root@FMC01:/Volume/home/ravi thakur# /etc/rc.d/init.d/console restart
Stopping Cisco Firepower Management Center 4600......ok
Starting Cisco Firepower Management Center 4600, please wait......started.

still the above process are Down.

[UdupiKrishna]

I would start looking at latest /var/log/messages to understand if there are any potential problems. You can run "tail -n <number> /var/log/messages. Feel free to use different numbers to print number of lines needed. Try restarting the console services again and tail the latest logs.

Additionally, paste the output of df -h. There are known problems when there's high disk usage and RabbitMQ doesn't come up and other processes like tomcat, DCCSM continue to be down

[Ravi Thakur · Accepted solution]

Thanks Krishna for your response...

We found two below bugs related to this issue & had to involve TAC engineer as the workaround mentioned is to connect with TAC.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvz62172

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvn25053

TAC engineer applied some internal workaround post which all the process came up & we are now able to login GUI.

[cisco]

Hi, 

by any chance, you have saved the commands that the TAC issued on your FMC to bring it up again? 
I just had a 2-weeks Tac-Case open, in which i was asked to send them the t-shoot files and other not related outputs, and after two weeks of nothing i got the notification that the server is lost, as there is no workaround for this bug. 

Would love to see the steps, the TAC took in your case. 

Many thanks!

[Ravi Thakur]

Hi 

Following are few of the history commands which were logged during the session:

-FM01:/Volume/home/rootuser# pmtool RestartByID vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool RestartByID DCCSM
-FM01:/Volume/home/rootuser# pmtool RestartByID Tomcat
-FM01:/Volume/home/rootuser# pmtool RestartByID VmsBackendServer
-FM01:/Volume/home/rootuser# pmtool status | grep -i gui
-FM01:/Volume/home/rootuser# pmtool status | grep sftunnel
-FM01:/Volume/home/rootuser# pmtool status | grep -i gui
-FM01:/Volume/home/rootuser# pmtool DisableByID vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool status | grep vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool EnableByID vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool status | grep vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool DisableByID vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool RestartByID vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool status | grep vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool EnableByID vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool RestartByID vmsDbEngine
-FM01:/Volume/home/rootuser# pmtool status | grep vmsDbEngine
-FM01:/Volume/home/rootuser# tail -f /var/log/messages
-FM01:/Volume/home/rootuser# /etc/rc.d/init.d/pm restart
-FM01:/Volume/home/rootuser# pmtool status | grep -i down
-FM01:/Volume/home/rootuser# cd /var/opt/CSCOpx/databases/vms
-FM01:/var/opt/CSCOpx/databases/vms# ls -ltrh
-FM01:/var/opt/CSCOpx/databases/vms# chmod 600 vms.db
-FM01:/var/opt/CSCOpx/databases/vms# chmod 644 vms.log
-FM01:/var/opt/CSCOpx/databases/vms# chown casuser:casusers vms.dbchown casuser:casusers vms.db
-FM01:/var/opt/CSCOpx/databases/vms# chown casuser:casusers vms.db
-FM01:/var/opt/CSCOpx/databases/vms# chown casuser:casusers vms.log
-FM01:/var/opt/CSCOpx/databases/vms# ls -ltrh
-FM01:/var/opt/CSCOpx/databases/vms# /etc/rc.d/init.d/pm restart
-FM01:/var/opt/CSCOpx/databases/vms# DBCheck.pl
-FM01:/var/opt/CSCOpx/databases/vms# pmtool status | grep -i down
-FM01:/var/opt/CSCOpx/databases/vms# vi /etc/sf/gch/call_home_ca
-FM01:/var/opt/CSCOpx/databases/vms# ping <ip address of secondary FMC>
-FM01:/var/opt/CSCOpx/databases/vms# cd
-FM01:~# tail -f /var/log/messages