FMC Stuck in System Processes are starting, please wait.

[laurathaqi]

Dear community, 

I have an FMC Virtual, which is stuck at "System Processes are starting, please wait.". 

I tried to manually restart the services, start tomcat etc, based on community suggestions, but non of them was able to start the services which are down as following: 

mysqld (system,gui,mysql) - Running 
httpsd (system,gui) - Running 
sybase_arbiter (system,gui) - Waiting
vmsDbEngine (system,gui) - Down
ESS (system,gui) - Running 
DCCSM (system,gui) - Down
Tomcat (system,gui) - Down
VmsBackendServer (system,gui) - Down
mojo_server (system,gui) - Running 

I did also delete the process in the database, and tried to start the httpd service, but no success. 

Any idea how to proceed without having to reimage? 

I found error messages as following: 

Dec 3 13:41:10 firepower SF-IMS[5541]: [5541] pm:process [INFO] Killing DCCSM with /var/sf/bin/dccsmstop.pl
Dec 3 13:41:13 firepower SF-IMS[5541]: [5541] pm:process [INFO] Killing DCCSM with /var/sf/bin/dccsmstop.pl
Dec 3 13:41:14 firepower SF-IMS[5541]: [5541] pm:process [INFO] Started vmsDbEngine (11537)
Dec 3 13:41:14 firepower SF-IMS[11537]: [11537] pm:CGroups [INFO] Apply: memory=System <p11537/vmsDbEngine>.
Dec 3 13:41:14 firepower SF-IMS[11537]: [11537] pm:process [INFO] Starting pre-command: /usr/bin/perl -MSF::SFDBI -e 'SF::SFDBI::get_sybase_password("true")'
Dec 3 13:41:15 firepower SF-IMS[11537]: [11537] pm:process [INFO] Starting pre-command: /var/opt/CSCOpx/bin/perl /var/opt/CSCOpx/bin/tuneVmsDbCache.pl
Dec 3 13:41:15 firepower SF-IMS[5541]: [5541] pm:process [INFO] Killing DCCSM with /var/sf/bin/dccsmstop.pl
Dec 3 13:41:15 firepower SF-IMS[11176]: [11176] (none):SybaseDatastore [INFO] SybaseDatastore.c:497:clientmsg_callback(): Client Library error: severity(0) number(4) origin(3) layer(5): ct_connect(): network packet layer: internal net library error: Net-Lib protocol driver call to connect two endpoints failed
Dec 3 13:41:15 firepower SF-IMS[11176]: [11176] (none):SybaseDatastore [ERROR] SybaseDatastore.c:514:clientmsg_callback(): Operating system error number(111): Socket connect failed - errno 111 Connection refused
Dec 3 13:41:15 firepower SF-IMS[11176]: [11176] (none):SybaseDatastore [ERROR] SybaseDatastore.c:810:Connect(): Failed to connect to Sybase service after 60 seconds
Dec 3 13:41:15 firepower SF-IMS[11176]: [11176] Event Streamer:DatastoreClient [ERROR] Unable to connect to datastore: Not connected
Dec 3 13:41:15 firepower SF-IMS[11176]: [11176] Event Streamer:ConnectionHandler [ERROR] Unable to load sync datastore module: Not connected
Dec 3 13:41:15 firepower SF-IMS[11176]: [11176] Event Streamer:ConnectionHandler [INFO] Parent Exiting: Not connected
Dec 3 13:41:15 firepower SF-IMS[5541]: [5541] pm:process [INFO] Process estreamer-sftunnel (11176) exited cleanly
Dec 3 13:41:15 firepower SF-IMS[5541]: [5541] pm:log [INFO] Process 'estreamer-sftunnel' closed output.

Thank you,

Laura

[balaji.bandi]

May be Looks like bug :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt18051

as per the below thread :

We have a fix!

My ESXI host crashed and I was having the same error as you when I opened up the FMC web GUI I could not login and the error message "System processes are starting, please wait." was present. Apparently there is a unsolved bug with FMC CSCvt18051 that upon reloading the GUI processes do not start up on there own. However there is a work around. Console into your firepower linux OS and Delete the /var/lib/rabbitmq/mnesia/rabbit@localhost/recovery.dets file & restart RabbitMQ from pmtool. The command for that is pmtool restartbyid RabbitMQ. Give that a try and my gui started operating normally afterward. Hope it works for you as well.

https://community.cisco.com/t5/network-security/fmc-login-page-hung-at-quot-system-processes-are-starting-please/m-p/4512731#M1085532

[laurathaqi]

Hi @balaji.bandi 

Thank you for your feedback! 

I tried it and it did not solve the issue!

I am working on the backup restore, as I did quite some but none of them worked. 

To note: Forgot to note that it have enough storage. 

Thank you,

Laura 

[shehreyarkhan]

Hi,

I had the same issue and it was because disk becoming full of daily backup copies , if you run "df -h" command on the shell of FMC, it will show you the disk usage status and all you have to delete the copy of old backups from the FMC disk.

When disk becomes full or insufficient disk space the web services of FMC will stop.

Try cleaning up the disk.

Regards

[shehreyarkhan]

some sample out put of disk status

root@sfr-mgmt:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.7G 3.7G 0 100% /
devtmpfs 5.9G 72K 5.9G 1% /dev
/dev/sda1 88M 40M 41M 50% /boot
/dev/sda7 266G 77G 177G 31% /var
none 5.9G 172K 5.9G 1% /dev/shm
tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup
tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup/pm

root@sfr-mgmt:~# find / -xdev -size +1000000 -ls try with size
root@sfr-mgmt:~# find / -xdev -size +100000 -ls or this size

it will locate the backup location , some thing like below

18740 67712 -rw-r--r-- 1 root root 69335040 Dec 27 17:01 /mnt/remote-storage/sf-storage/79d7cef2-51bc-11e7-bf2b-8952ff38fd15/remote-backups/BACKUP_FTD-DC1-4012_20201227170008.tar

follow the path and delete the old files.

If the works you then just reload the FMC to initialize the web-service

[laurathaqi]

Hi all,

Thank you for your great input. Due to the many things I tried, and had no TAC Support available. I recovered the last backup in a new VM, and its working find now. 

Thank you,

Laura