topic Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night in Network Security

FMC login page hung at "System processes are starting, please wait." Remained hung over night

garyewhite — Wed, 12 Aug 2020 18:35:14 GMT

gui status follows...

admin@Sourcefire3d: sudo pmtool status | grep -i gui

mysqld (system,gui,mysql) - Running 3853
httpsd (system,gui) - Running 3856
sybase_arbiter (system,gui) - Waiting
vmsDbEngine (system,gui) - Down
ESS (system,gui) - Running 3858
DCCSM (system,gui) - Down
Tomcat (system,gui) - Down
VmsBackendServer (system,gui) - Down
mojo_server (system,gui) - Running 3881

Any ideas?

Thanks!

-G

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

balaji.bandi — Wed, 12 Aug 2020 19:13:01 GMT

Tomcat (system,gui) - Down - this service down so you would not get any GUI Page

what version is this ? is this new setup or working one failed?

Is this VM ? then what kind of compute you allocated for this VM

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

garyewhite — Wed, 12 Aug 2020 19:26:23 GMT

Version 6.1
Old, working VM system that failed to restart fully after power loss.
8GB RAM
4 CPU’s
2 HD’s, 250GB and 750GB

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

balaji.bandi — Wed, 12 Aug 2020 20:07:17 GMT

STOP the VM( shutdown guest properly.) wait for 10min, and then start and let us know the outcome.

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

crstephenson — Wed, 12 Aug 2020 23:48:29 GMT

I have exactly this same issue, after my ESXi server lost power last night.

Currently, both my FMC and FTD appear to have issues with services hanging or not starting. Will be really interested to hear anyone's in put on this, especially if there is a fix.

bt, i am running 6.6

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

balaji.bandi — Thu, 13 Aug 2020 06:37:56 GMT

Personally, these are built on Linux based platform, it's like any other OS abnormally power loss has side effects.

Like OS Crash -

These are very critical in a real-world environment at least expect to have 15-20min backup time of USP, so you can shutdown required VM necessary part of the shutdown process, or any Power Failure UPS can give a decent amount of backup time until the main recovery.

FMC/ FTD more rely on and DB and other stuff, so it's hard to recover some time, rather re-installing (personally I believe that what happens when Linux crashes).

Options1.

As suggested, turn Off VM

wait for few Minutes 10min

Turn on the VM gives more time to recovery

also if you have a screen capture, capture the console of VM to understand what process it got stuck. ?

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

crstephenson — Thu, 13 Aug 2020 09:40:49 GMT

Yeah, I have tried all of that, but the same problem persists. I even de-registered the VM an re-registered to ESXI, but no joy.

The service status is exactly the same as experienced by the OP (Gary). It's almost as if the FMC and FTD VMs are waiting to attach to a database of some kind, but the service has not started. Is there a way to force start the database engine?

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

Mohammed al Baqari — Thu, 13 Aug 2020 09:43:18 GMT

Hi,

Jump on expert mode of FMC and issue this command

tial -f -n xx /var/log/message

change 'xx' to the number of lines you want to see from the bottom. This
should help you to see the logs from FMC and what is happening.

If the DB is locked or corrupted, then you need to rebuild (and you can
restore your backup if present)

**** please remember to rate useful potss

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

garyewhite — Thu, 13 Aug 2020 13:47:54 GMT

tail on /var/log/messages doesn't show any indication that the DB is locked or corrupt:

root@Sourcefire3D:/var/log# tail -f -n 20 /var/log/messages
Aug 13 13:44:05 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published Identity Service for peer 10.134.32.60
Aug 13 13:44:05 Sourcefire3D SF-IMS[3817]: [3817] pm:process [INFO] Started Syncd (19219)
Aug 13 13:44:05 Sourcefire3D SF-IMS[19219]: [19219] pm:CGroups [INFO] Apply: process <p19219/Syncd>.
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:07 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:07 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:07 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3817]: [3817] pm:log [INFO] Process 'Syncd' closed output.
Aug 13 13:44:08 Sourcefire3D SF-IMS[3817]: [3817] pm:process [INFO] Process Syncd (19219) exited cleanly
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published Identity Service for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer X.X.X.X.
Aug 13 13:44:09 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published Identity Service for peer X.X.X.X
Aug 13 13:44:12 Sourcefire3D SF-IMS[3861]: [3907] sftunneld:sf_peers [INFO] Peer X.X.X.X needs a single connection
Aug 13 13:44:12 Sourcefire3D SF-IMS[3861]: [3907] sftunneld:sf_peers [INFO] Peer X.X.X.X needs a single connection
Aug 13 13:44:12 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer X.X.X.X.
Aug 13 13:44:13 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published Identity Service for peer X.X.X.X
Aug 13 13:44:14 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer X.X.X.X.
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

garyewhite — Thu, 13 Aug 2020 16:37:25 GMT

Shutdown for 30 minutes. Restarted. No change. Still hung at 'System processes are starting, please wait'.

-G

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

Marvin Rhoads — Thu, 13 Aug 2020 18:21:20 GMT

I had a customer recently experience an unplanned power outage and we ended up having to reimage several Firepower service modules after consultation with TAC confirmed unrecoverable corruption.

Similar problems could happen with FMC. If you have backups you could restore from them after reimage (need to make sure version and VDB match).

TAC consultation is definitely recommended either way.

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

balaji.bandi — Thu, 13 Aug 2020 18:32:10 GMT

Look at the logs - required TAC assistance here I guess. as suggested these devices never expect to power down like this. ( as this is unexpected) so future make sure you arrange any UPS for best outcome.

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

garyewhite — Thu, 13 Aug 2020 20:52:58 GMT

Anyone know if you can restore a backup through the CLI? Can't log in to the GUI... but the CLI works like a champ. And multiple backups are available in \var\sf\backup...

Thanks,

-Gary

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

shehreyarkhan — Fri, 08 Jan 2021 14:39:48 GMT

Hi,

I have the same issue, FMC was giving disk usage full warning and then I could not log in it after logging out, and it is stuck in

"System processes are starting, please wait.", I am not using VM its appliance

mysqld (system,gui,mysql) - Running 8363
httpsd (system,gui) - Running 19091
sybase_arbiter (system,gui) - Waiting
vmsDbEngine (system,gui) - Running 8367
ESS (system,gui) - Running 8385
DCCSM (system,gui) - Running 9238
Tomcat (system,gui) - Running 9425
VmsBackendServer (system,gui) - Running 9426
mojo_server (system,gui) - Running 19103

Please advise how to solve this issue?

Regards

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

garyewhite — Fri, 08 Jan 2021 15:22:01 GMT

Hi shehreyarkhan,
The end solution for this, in my case, was to reinstall the FMC and restore the settings from the backup.
It was not the preferred solution, but after being passed from one junior tech to another in the Cisco TAC pool, I relented and just reimaged.

My suggestion to you is to open a ticket with Cisco. Perhaps your situation will be more easily remedied.

Best of luck!

-garyewhite

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

kylemiller452225788 — Fri, 18 Jun 2021 22:11:30 GMT

We have a fix!

My ESXI host crashed and I was having the same error as you when I opened up the FMC web GUI I could not login and the error message "System processes are starting, please wait." was present. Apparently there is a unsolved bug with FMC CSCvt18051 that upon reloading the GUI processes do not start up on there own. However there is a work around. Console into your firepower linux OS and Delete the /var/lib/rabbitmq/mnesia/rabbit@localhost/recovery.dets file & restart RabbitMQ from pmtool. The command for that is pmtool restartbyid RabbitMQ. Give that a try and my gui started operating normally afterward. Hope it works for you as well.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt18051

currently there is no fix for this bug as up 6/15/2021 when Im writing this.

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

Marvin Rhoads — Sat, 19 Jun 2021 03:33:39 GMT

@kylemiller452225788

Thanks for sharing the BugID.

Re: FMC login page hung at "System processes are starting, please wait." Remained hung over night

WilliamMalta19560 — Sun, 11 Jul 2021 01:47:47 GMT

Hi, Team.

I was facing the same issue 'system processes are starting' message in the FMC Gui.

I noticed that some services does not start.
root@FMC:/Volume/home/admin# pmtool status | grep -i gui
mysqld (system,gui,mysql) - Running 21366
httpsd (system,gui) - Running 21369
sybase_arbiter (system,gui) - Waiting
vmsDbEngine (system,gui) - Running 21370
ESS (system,gui) - Running 21412
DCCSM (system,gui) - Running 22241
Tomcat (system,gui) - Down
VmsBackendServer (system,gui) - Down
mojo_server (system,gui) - Running 21860
root@FMC:/Volume/home/admin#

I have tried to restart services, but did not worked
pmtool restartbyid Tomcat
pmtool restartbyid VmsBackendServer

My solution was reboot FMC with reboot command under root user.

root@FMC:/Volume/home/admin# uptime
21:52:44 up 1 min, 1 user, load average: 4.49, 1.41, 0.49
root@FMC:/Volume/home/admin# pmtool status | grep -i gui
mysqld (system,gui,mysql) - Running 5851
httpsd (system,gui) - Running 5854
sybase_arbiter (system,gui) - Waiting
vmsDbEngine (system,gui) - Running 5855
ESS (system,gui) - Running 5896
DCCSM (system,gui) - Running 6765
Tomcat (system,gui) - Running 6902
VmsBackendServer (system,gui) - Running 6903
mojo_server (system,gui) - Running 5961
root@FMC:/Volume/home/admin#

Re: FMC login page hung at "System processes are starting, please

laurathaqi — Fri, 03 Dec 2021 13:53:33 GMT

Hi community,

Does someone has a solution for this problem? Happening on 6.6.1 Virtual on my Environment.

Looking forward to hearing from you.

Best,

Laura

Re: FMC login page hung at "System processes are starting, please

Kasper Elsborg — Sun, 21 Aug 2022 08:06:58 GMT

I had the same problem after an uncontrolled power outage.

I don't have any recovery.dets in /var/lib/rabbitmq/mnesia/rabbit@localhost/

but the "pmtool restartbyid RabbitMQ" did the trick.

Br. Kasper