FMC login page hung at "System processes are starting, please wait." Remained hung over night

[garyewhite]

Login page hung saying system processes are starting... it's been hours...

gui status follows...

admin@Sourcefire3d:  sudo pmtool status | grep -i gui

mysqld (system,gui,mysql) - Running 3853
httpsd (system,gui) - Running 3856
sybase_arbiter (system,gui) - Waiting
vmsDbEngine (system,gui) - Down
ESS (system,gui) - Running 3858
DCCSM (system,gui) - Down
Tomcat (system,gui) - Down
VmsBackendServer (system,gui) - Down
mojo_server (system,gui) - Running 3881

Any ideas?

Thanks!

-G

[balaji.bandi]

Tomcat (system,gui) - Down  - this service down so you would not get any GUI Page

what version is this ? is this new setup or working one failed?

Is this VM ? then what kind of compute you allocated for this VM

[garyewhite]

Version 6.1
Old, working VM system that failed to restart fully after power loss.
8GB RAM
4 CPU’s
2 HD’s, 250GB and 750GB

[balaji.bandi]

STOP the VM( shutdown guest properly.) wait for 10min, and then start and let us know the outcome.

[garyewhite]

Shutdown for 30 minutes.  Restarted.  No change.  Still hung at 'System processes are starting, please wait'.

-G

[crstephenson]

I have exactly this same issue, after my ESXi server lost power last night.

Currently, both my FMC and FTD appear to have issues with services hanging or not starting. Will be really interested to hear anyone's in put on this, especially if there is a fix.

bt, i am running 6.6

[balaji.bandi]

Personally, these are built on Linux based platform, it's like any other OS abnormally power loss has side effects.

Like OS Crash -

These are very critical in a real-world environment at least expect to have 15-20min backup time of USP, so you can shutdown required VM necessary part of the shutdown process, or any Power Failure UPS can give a decent amount of backup time until the main recovery.

FMC/ FTD more rely on and DB and other stuff, so it's hard to recover some time, rather re-installing (personally I believe that what happens when Linux crashes).

Options1. 

As suggested, turn Off VM

wait for few Minutes 10min

Turn on the VM gives more time to recovery

also if you have a screen capture, capture the console of VM to understand what process it got stuck. ?

[crstephenson]

Yeah, I have tried all of that, but the same problem persists. I even de-registered the VM an re-registered to ESXI, but no joy.

The service status is exactly the same as experienced by the OP (Gary). It's almost as if the FMC and FTD VMs are waiting to attach to a database of some kind, but the service has not started. Is there a way to force start the database engine?

[Mohammed al Baqari]

Hi,

Jump on expert mode of FMC and issue this command

tial -f -n xx /var/log/message

change 'xx' to the number of lines you want to see from the bottom. This
should help you to see the logs from FMC and what is happening.

If the DB is locked or corrupted, then you need to rebuild (and you can
restore your backup if present)

**** please remember to rate useful potss

[garyewhite]

tail on /var/log/messages doesn't show any indication that the DB is locked or corrupt:

root@Sourcefire3D:/var/log# tail -f -n 20 /var/log/messages
Aug 13 13:44:05 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published Identity Service for peer 10.134.32.60
Aug 13 13:44:05 Sourcefire3D SF-IMS[3817]: [3817] pm:process [INFO] Started Syncd (19219)
Aug 13 13:44:05 Sourcefire3D SF-IMS[19219]: [19219] pm:CGroups [INFO] Apply: process <p19219/Syncd>.
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:06 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:07 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:07 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:07 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3817]: [3817] pm:log [INFO] Process 'Syncd' closed output.
Aug 13 13:44:08 Sourcefire3D SF-IMS[3817]: [3817] pm:process [INFO] Process Syncd (19219) exited cleanly
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published Identity Service for peer X.X.X.X
Aug 13 13:44:08 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer X.X.X.X.
Aug 13 13:44:09 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published Identity Service for peer X.X.X.X
Aug 13 13:44:12 Sourcefire3D SF-IMS[3861]: [3907] sftunneld:sf_peers [INFO] Peer X.X.X.X needs a single connection
Aug 13 13:44:12 Sourcefire3D SF-IMS[3861]: [3907] sftunneld:sf_peers [INFO] Peer X.X.X.X needs a single connection
Aug 13 13:44:12 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer X.X.X.X.
Aug 13 13:44:13 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published Identity Service for peer X.X.X.X
Aug 13 13:44:14 Sourcefire3D SF-IMS[3861]: [3915] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer X.X.X.X.
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3913] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3908] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3912] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 1 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X
Aug 13 13:44:16 Sourcefire3D SF-IMS[3861]: [3916] sftunneld:sf_heartbeat [INFO] Received message for not published UE Priority Event Channel 0 for peer X.X.X.X

[Marvin Rhoads]

I had a customer recently experience an unplanned power outage and we ended up having to reimage several Firepower service modules after consultation with TAC confirmed unrecoverable corruption.

Similar problems could happen with FMC. If you have backups you could restore from them after reimage (need to make sure version and VDB match).

TAC consultation is definitely recommended either way.

[garyewhite]

Anyone know if you can restore a backup through the CLI?  Can't log in to the GUI... but the CLI works like a champ.  And multiple backups are available in \var\sf\backup...

Thanks,

-Gary

[balaji.bandi]

Look at the logs - required TAC assistance here I guess. as suggested these devices never expect to power down like this. ( as this is unexpected) so future make sure you arrange any UPS for best outcome.

[shehreyarkhan]

Hi,

I have the same issue, FMC was giving disk usage full warning and then I could not log in it after logging out, and it is stuck in

"System processes are starting, please wait.", I am not using VM its appliance 

mysqld (system,gui,mysql) - Running 8363
httpsd (system,gui) - Running 19091
sybase_arbiter (system,gui) - Waiting
vmsDbEngine (system,gui) - Running 8367
ESS (system,gui) - Running 8385
DCCSM (system,gui) - Running 9238
Tomcat (system,gui) - Running 9425
VmsBackendServer (system,gui) - Running 9426
mojo_server (system,gui) - Running 19103

Please advise how to solve this issue?

Regards

[garyewhite]

Hi shehreyarkhan,
The end solution for this, in my case, was to reinstall the FMC and restore the settings from the backup.
It was not the preferred solution, but after being passed from one junior tech to another in the Cisco TAC pool, I relented and just reimaged.

My suggestion to you is to open a ticket with Cisco. Perhaps your situation will be more easily remedied.

Best of luck!

-garyewhite