03-07-2021 01:40 PM
Hello Guys,
Today we just experienced an ambiguous behavior. We've a Cisco IPS 7120 sensor from the old days just after rebooting, it freezed that is, all interfaces are up, ping is working fine from the sensor to FMC and vice versa but can't apply any change. Also when logging via CLI and issue "show manager" it said no managers configured and any show command displays a blank output however, before the reload we checked all show commands and all outputs were correct.
Any ideas on what went wrong during the reload?
Thanks everyone.
Solved! Go to Solution.
03-09-2021 06:53 AM - edited 03-09-2021 07:47 AM
Since if you decided to re-image
try below command
sudo su
/etc/rc.d/init.d/mysqld status
/etc/rc.d/init.d/mysqld restart
check the logs
If this device managed by FMC. FMC should have all the information, check if you can take any local backup and settings
sure re-image should work, since do you do not have contract to open a TAC case.
03-11-2021 05:54 AM
Hi balaji,
I managed to fix the issue by Re-Imaging the device. One thing that was a nightmare is that the interactive menu couldn't found the ISO image on FTP / SCP.
Finally solve it by using HTTP server.
Revert to the base image then register with FMC and patch update sequentially.
Thanks ☺️☺️☺️
03-07-2021 01:45 PM - edited 03-07-2021 01:49 PM
what is the version of code 7120 and FMC ?
> sftunnel-status - what is the status ?
03-07-2021 01:47 PM
The version is 5.4
Thank you
03-07-2021 01:49 PM
both ?
03-07-2021 01:53 PM
Yes both are running 5.4 code.
From the FMC in the device management area it shows the sensor with green icon with status "Recovered".
When editing the device it shows all interfaces status as "no link" but actually they are up/up.
03-07-2021 02:03 PM
> sftunnel-status - what is the status ?
$ netstat -na | grep 8305
03-07-2021 02:09 PM
I believe sftunnel-status command is not supported since I couldn't execute it.
I'll try netstat - na and provide the output.
Thank you
Rami
03-08-2021 01:09 AM
Hi balaji,
here is the output of netstat -na | grep 8305
On Sensor:
tcp 0 0 10.20.100.140:59150 10.30.200.50:8305 ESTABLISHED
tcp 0 0 10.20.100.140:45953 10.30.200.50:8305 ESTABLISHED
On FMC:
tcp 0 0 10.30.200.50:8305 10.20.100.140:59150 ESTABLISHED
tcp 0 0 10.30.200.50:8305 10.20.100.140:45953 ESTABLISHED
I couldn't apply sftunnel-status it seems to be not-supported but I used pmtool status instead and make point of the below:
pmtool status
Received status (0):
Global Environment:
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin
Daemons:
mysqld (system,gui) - Down
Command: /usr/bin/mysqld --defaults-file=/etc/my.cnf --user=mysql --basedir=/usr --datadir=/var/lib/mysql --pid-file=/var/run/mysql/mysqld.pid --skip-external-locking
PID File: /var/run/mysql/mysqld.pid
Stop Timeout: 300
Next start: Mon Mar 8 08:32:59 2021
sftunnel (system) - Running 3805
Command: /usr/local/sf/bin/sftunnel -d -f /etc/sf/sftunnel.conf
PID File: /var/sf/run/sftunnel.pid
Enable File: /etc/sf/sftunnel.conf
Next start: Sun Mar 7 17:22:02 2021
it seems that the sftunnel is up but DB service is down also below the show manager output
> show managers
no managers configured
Thank you.
03-08-2021 01:46 AM
check the Logs messages /var/log/message and see any abnormal logs and also suggest checking the space issue - df -h
is this kit working as expected and do not have access - may try rebooting one more time. Mysql DB required to be up to get VDB
03-08-2021 02:20 AM
Hello balaji,
Thanks for your advise.
Actually this box was working as expected but suddenly we weren't able to apply ACP to it so reboot it and result in this behavior moreover, we tried to reboot it twice but with no luck.
I checked log messages and notice some Errors regarding the DB
> tail -f /var/log/messages
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[20123]: [20123] fpcollect:InitDatabase [ERROR] Unable to connect to datastore: Unhandled database error
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[20123]: [20123] fpcollect:fpcollect [ERROR] Exiting with code -1
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'fpcollect' closed output.
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'fpcollect' closed output.
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process fpcollect (20123) exited cleanly
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started fpcollect (20204)
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[20204]: [20204] fpcollect:Config [INFO] Loaded datastore 'MySQL'
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[20204]: [20204] fpcollect:config [INFO] Configuration read
Mar 8 10:08:27 SOURCEFIRE-SENSOR1 SF-IMS[3805]: [3846] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer 10.30.200.50.
Mar 8 10:08:49 SOURCEFIRE-SENSOR1 last message repeated 2 times
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[20162]: [20162] IDSEventAlerter:config [ERROR] Unable to connect to datastore: Unhandled database error
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ad9769a2-4907-11e4-bd51-5b852c85c85c-alert' closed output.
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process ad9769a2-4907-11e4-bd51-5b852c85c85c-alert (20162) exited cleanly
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started ad9769a2-4907-11e4-bd51-5b852c85c85c-alert (20242)
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ad9769a2-4907-11e4-bd51-5b852c85c85c-alert' closed output.
Mar 8 10:08:59 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started mysqld (20243)
Mar 8 10:08:59 SOURCEFIRE-SENSOR1 SF-IMS[20243]: [20243] pm:process [INFO] Starting pre-command: /bin/mkdir /var/run/mysql
Mar 8 10:08:59 SOURCEFIRE-SENSOR1 SF-IMS[20243]: [20243] pm:process [INFO] Starting pre-command: /bin/chmod 0755 /var/run/mysql
Mar 8 10:08:59 SOURCEFIRE-SENSOR1 SF-IMS[20243]: [20243] pm:process [INFO] Starting pre-command: /bin/chown mysql:mysql /var/run/mysql
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3805]: [3846] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer 10.30.200.50.
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process mysqld (20243) exited cleanly
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [ERROR] Process 20243 not found from log monitor.
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'TSS_Daemon' closed output.
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'TSS_Daemon' closed output.
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process TSS_Daemon (20179) exited cleanly
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started ntpd (20258)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started TSS_Daemon (20259)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started expire-session (20260)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started Pruner (20261)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started ActionQueueScrape (20262)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started SFTop10Cacher (20263)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started run_hm (20264)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'SFTop10Cacher' closed output.
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'SFTop10Cacher' closed output.
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process SFTop10Cacher (20263) exited cleanly
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ActionQueueScrape' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ActionQueueScrape' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process ActionQueueScrape (20262) exited cleanly
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'Pruner' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'Pruner' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process Pruner (20261) exited cleanly
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'run_hm' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'run_hm' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process run_hm (20264) exited cleanly
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'expire-session' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'expire-session' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process expire-session (20260) exited cleanly
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[20191]: [20191] SFDataCorrelator:MySQLDatastore [ERROR] Unable to connect to database after 60 seconds: Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (2)
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[20191]: [20191] SFDataCorrelator:DCE_DB [ERROR] Unable to connect to datastore: Unhandled database error
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[20191]: [20191] SFDataCorrelator:SFDataCorrelator [ERROR] Failed to process DB configuration
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'SFDataCorrelator' closed output.
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process SFDataCorrelator (20191) exited cleanly
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'SFDataCorrelator' closed output.
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started SFDataCorrelator (20271)
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] pm:process [INFO] Starting pre-command: /usr/local/sf/bin/check_sfd_shutdown.pl
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:main [INFO] Start SFDataCorrelator v5.4.0.8-23
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:License [INFO] Virtual 3D Sensors licenses found? Setting number to 0
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Correlator [INFO] Host limit set to 0
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Correlator [INFO] User limit set to 0
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Loaded datastore 'MySQL'
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] File storage path is /var/tmp
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] File sandbox top level domain is https://intel.api.sourcefire.com
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Validate Configuration /etc/sf/SFDataCorrelator.conf
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] UNIX socket configured
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Listening at: /var/sf/run/SFDataCorrelator.sock
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Unified2 archive output
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Event FileProcess
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] RNA event window set to 50 events
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] Affinity Configuration
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] NUMA Nodes:
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] Node 0
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPUs:
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPU 1 (Node 0)
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPU 2 (Node 0)
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPU 3 (Node 0)
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] Use NUMA: Yes
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPUs Per Node: 3
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:FileExtractCloud [INFO] Sandbox rate limit is 7
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Sandbox rate limit is 7
Mar 8 10:09:14 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:MySQLDatastore [WARN] Trying to connect to database server after error 2002: Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (2)
Mar 8 10:09:14 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ntpd' closed output.
Mar 8 10:09:14 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ntpd' closed output.
Mar 8 10:09:14 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process ntpd (20258) exited cleanly
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[20204]: [20204] fpcollect:InitDatabase [ERROR] Unable to connect to datastore: Unhandled database error
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[20204]: [20204] fpcollect:fpcollect [ERROR] Exiting with code -1
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process fpcollect (20204) exited cleanly
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started fpcollect (20284)
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'fpcollect' closed output.
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'fpcollect' closed output.
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[20284]: [20284] fpcollect:Config [INFO] Loaded datastore 'MySQL'
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[20284]: [20284] fpcollect:config [INFO] Configuration read
Do you recommend to shutdown the appliance physically from the power button then powered it up? can this force the DB to initialize normally?
Appreciate you help.
Thank you.
Rami
03-08-2021 03:06 AM
in a normal situation, you can restart MySQL DB process.
check disk space as suggested or du -h ( i may be thinking space issue or DB run file locked)
> show disk
i would suggest to proper shutdown the kit and bring back online check if that fix the issue ( do you have cisco contract to open a TAC case if is this not resolved)
03-08-2021 03:28 AM
Here is the disk output, everything seem to be good.
> show disk
Filesystem Size Used Avail Use% Mounted on
/dev/root 2.9G 1.2G 1.6G 44% /
devtmpfs 7.4G 64K 7.4G 1% /dev
/dev/sda1 99M 30M 64M 32% /boot
/dev/sda7 67G 17G 47G 27% /var
none 7.4G 70M 7.3G 1% /dev/shm
# du -h
20K ./.ssh
404K .
Unfortunately we don't have a valid contract right now so we can't open a TAC so we might shutdown the appliance and boot it up again to see if anything changed.
Thank you.
Rami
03-08-2021 10:13 PM
Hello balaji,
I've shutdown the appliance and bring it back up again but that didn't solve the issue. I tried so many things
/etc/rc.d/init.d/network restart
manage_procs.pl with option 3 (Restart Comm. Channel)
I even re-configure the manager statically using the "configure manager" command and that didn't work, it returns an error
getPeersByRole: unable to connect to db at /usr/local/sf/lib/perl/5.10.1/SF/PeerManager/Peers.pm line 107 -
Is there a way to check the database and fix it?
I am getting frustrated
03-09-2021 05:59 AM
At this stage not sure what direction i can suggest, we need to fix MySQL DB to run - see if that works.
03-09-2021 06:15 AM
Hi Balaji, Thanks for the response.
Actually, I tried many things to fix the DB , though I am not that much experienced with SQL but I'm planning to Re-image the device and that should delete the DB and put a new one.
Am I correct?
Regards,
Rami.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide