Solved: Re: Cisco IPS 7120 Very Strange behavior

Rami Ibrahim · ‎03-07-2021

Hello Guys,

Today we just experienced an ambiguous behavior. We've a Cisco IPS 7120 sensor from the old days just after rebooting, it freezed that is, all interfaces are up, ping is working fine from the sensor to FMC and vice versa but can't apply any change. Also when logging via CLI and issue "show manager" it said no managers configured and any show command displays a blank output however, before the reload we checked all show commands and all outputs were correct.

Any ideas on what went wrong during the reload?

Thanks everyone.

balaji.bandi · ‎03-09-2021

Since if you decided to re-image

try below command

sudo su

/etc/rc.d/init.d/mysqld status
/etc/rc.d/init.d/mysqld restart

check the logs

If this device managed by FMC. FMC should have all the information, check if you can take any local backup and settings

sure re-image should work, since do you do not have contract to open a TAC case.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

View solution in original post

Rami Ibrahim · ‎03-11-2021

Hi balaji,

I managed to fix the issue by Re-Imaging the device. One thing that was a nightmare is that the interactive menu couldn't found the ISO image on FTP / SCP.

Finally solve it by using HTTP server.

Revert to the base image then register with FMC and patch update sequentially.

Thanks ☺️☺️☺️

View solution in original post

balaji.bandi · ‎03-07-2021

what is the version of code 7120 and FMC ?

> sftunnel-status - what is the status ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Rami Ibrahim · ‎03-07-2021

The version is 5.4

Thank you

balaji.bandi · ‎03-07-2021

both ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Rami Ibrahim · ‎03-07-2021

Yes both are running 5.4 code.

From the FMC in the device management area it shows the sensor with green icon with status "Recovered".

When editing the device it shows all interfaces status as "no link" but actually they are up/up.

balaji.bandi · ‎03-07-2021

> sftunnel-status - what is the status ?

$ netstat -na | grep 8305

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Rami Ibrahim · ‎03-07-2021

I believe sftunnel-status command is not supported since I couldn't execute it.

I'll try netstat - na and provide the output.

Thank you

Rami

Rami Ibrahim · ‎03-08-2021

Hi balaji,

here is the output of netstat -na | grep 8305

On Sensor:

tcp 0 0 10.20.100.140:59150 10.30.200.50:8305 ESTABLISHED
tcp 0 0 10.20.100.140:45953 10.30.200.50:8305 ESTABLISHED

On FMC:

tcp 0 0 10.30.200.50:8305 10.20.100.140:59150 ESTABLISHED
tcp 0 0 10.30.200.50:8305 10.20.100.140:45953 ESTABLISHED

I couldn't apply sftunnel-status it seems to be not-supported but I used pmtool status instead and make point of the below:

pmtool status

Received status (0):

Global Environment:
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin

Daemons:
mysqld (system,gui) - Down
Command: /usr/bin/mysqld --defaults-file=/etc/my.cnf --user=mysql --basedir=/usr --datadir=/var/lib/mysql --pid-file=/var/run/mysql/mysqld.pid --skip-external-locking
PID File: /var/run/mysql/mysqld.pid
Stop Timeout: 300
Next start: Mon Mar 8 08:32:59 2021

sftunnel (system) - Running 3805
Command: /usr/local/sf/bin/sftunnel -d -f /etc/sf/sftunnel.conf
PID File: /var/sf/run/sftunnel.pid
Enable File: /etc/sf/sftunnel.conf
Next start: Sun Mar 7 17:22:02 2021

it seems that the sftunnel is up but DB service is down also below the show manager output

> show managers

no managers configured

Thank you.

balaji.bandi · ‎03-08-2021

check the Logs messages /var/log/message and see any abnormal logs and also suggest checking the space issue - df -h

is this kit working as expected and do not have access - may try rebooting one more time. Mysql DB required to be up to get VDB

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Rami Ibrahim · ‎03-08-2021

Hello balaji,

Thanks for your advise.

Actually this box was working as expected but suddenly we weren't able to apply ACP to it so reboot it and result in this behavior moreover, we tried to reboot it twice but with no luck.

I checked log messages and notice some Errors regarding the DB

> tail -f /var/log/messages
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[20123]: [20123] fpcollect:InitDatabase [ERROR] Unable to connect to datastore: Unhandled database error
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[20123]: [20123] fpcollect:fpcollect [ERROR] Exiting with code -1
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'fpcollect' closed output.
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'fpcollect' closed output.
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process fpcollect (20123) exited cleanly
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started fpcollect (20204)
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[20204]: [20204] fpcollect:Config [INFO] Loaded datastore 'MySQL'
Mar 8 10:08:17 SOURCEFIRE-SENSOR1 SF-IMS[20204]: [20204] fpcollect:config [INFO] Configuration read
Mar 8 10:08:27 SOURCEFIRE-SENSOR1 SF-IMS[3805]: [3846] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer 10.30.200.50.
Mar 8 10:08:49 SOURCEFIRE-SENSOR1 last message repeated 2 times
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[20162]: [20162] IDSEventAlerter:config [ERROR] Unable to connect to datastore: Unhandled database error
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ad9769a2-4907-11e4-bd51-5b852c85c85c-alert' closed output.
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process ad9769a2-4907-11e4-bd51-5b852c85c85c-alert (20162) exited cleanly
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started ad9769a2-4907-11e4-bd51-5b852c85c85c-alert (20242)
Mar 8 10:08:52 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ad9769a2-4907-11e4-bd51-5b852c85c85c-alert' closed output.
Mar 8 10:08:59 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started mysqld (20243)
Mar 8 10:08:59 SOURCEFIRE-SENSOR1 SF-IMS[20243]: [20243] pm:process [INFO] Starting pre-command: /bin/mkdir /var/run/mysql
Mar 8 10:08:59 SOURCEFIRE-SENSOR1 SF-IMS[20243]: [20243] pm:process [INFO] Starting pre-command: /bin/chmod 0755 /var/run/mysql
Mar 8 10:08:59 SOURCEFIRE-SENSOR1 SF-IMS[20243]: [20243] pm:process [INFO] Starting pre-command: /bin/chown mysql:mysql /var/run/mysql
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3805]: [3846] sftunneld:sf_heartbeat [INFO] Received message for not published Malware Lookup Service for peer 10.30.200.50.
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process mysqld (20243) exited cleanly
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [ERROR] Process 20243 not found from log monitor.
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'TSS_Daemon' closed output.
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'TSS_Daemon' closed output.
Mar 8 10:09:00 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process TSS_Daemon (20179) exited cleanly
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started ntpd (20258)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started TSS_Daemon (20259)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started expire-session (20260)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started Pruner (20261)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started ActionQueueScrape (20262)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started SFTop10Cacher (20263)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started run_hm (20264)
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'SFTop10Cacher' closed output.
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'SFTop10Cacher' closed output.
Mar 8 10:09:03 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process SFTop10Cacher (20263) exited cleanly
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ActionQueueScrape' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ActionQueueScrape' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process ActionQueueScrape (20262) exited cleanly
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'Pruner' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'Pruner' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process Pruner (20261) exited cleanly
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'run_hm' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'run_hm' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process run_hm (20264) exited cleanly
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'expire-session' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'expire-session' closed output.
Mar 8 10:09:04 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process expire-session (20260) exited cleanly
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[20191]: [20191] SFDataCorrelator:MySQLDatastore [ERROR] Unable to connect to database after 60 seconds: Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (2)
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[20191]: [20191] SFDataCorrelator:DCE_DB [ERROR] Unable to connect to datastore: Unhandled database error
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[20191]: [20191] SFDataCorrelator:SFDataCorrelator [ERROR] Failed to process DB configuration
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'SFDataCorrelator' closed output.
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process SFDataCorrelator (20191) exited cleanly
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'SFDataCorrelator' closed output.
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started SFDataCorrelator (20271)
Mar 8 10:09:08 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] pm:process [INFO] Starting pre-command: /usr/local/sf/bin/check_sfd_shutdown.pl
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:main [INFO] Start SFDataCorrelator v5.4.0.8-23
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:License [INFO] Virtual 3D Sensors licenses found? Setting number to 0
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Correlator [INFO] Host limit set to 0
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Correlator [INFO] User limit set to 0
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Loaded datastore 'MySQL'
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] File storage path is /var/tmp
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] File sandbox top level domain is https://intel.api.sourcefire.com
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Validate Configuration /etc/sf/SFDataCorrelator.conf
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] UNIX socket configured
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Listening at: /var/sf/run/SFDataCorrelator.sock
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Unified2 archive output
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Event FileProcess
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] RNA event window set to 50 events
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] Affinity Configuration
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] NUMA Nodes:
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] Node 0
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPUs:
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPU 1 (Node 0)
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPU 2 (Node 0)
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPU 3 (Node 0)
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] Use NUMA: Yes
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:Affinity [INFO] CPUs Per Node: 3
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:FileExtractCloud [INFO] Sandbox rate limit is 7
Mar 8 10:09:09 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:config [INFO] Sandbox rate limit is 7
Mar 8 10:09:14 SOURCEFIRE-SENSOR1 SF-IMS[20271]: [20271] SFDataCorrelator:MySQLDatastore [WARN] Trying to connect to database server after error 2002: Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (2)
Mar 8 10:09:14 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ntpd' closed output.
Mar 8 10:09:14 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'ntpd' closed output.
Mar 8 10:09:14 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process ntpd (20258) exited cleanly
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[20204]: [20204] fpcollect:InitDatabase [ERROR] Unable to connect to datastore: Unhandled database error
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[20204]: [20204] fpcollect:fpcollect [ERROR] Exiting with code -1
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Process fpcollect (20204) exited cleanly
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3788] pm:process [INFO] Started fpcollect (20284)
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'fpcollect' closed output.
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[3788]: [3791] pm:log [INFO] Process 'fpcollect' closed output.
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[20284]: [20284] fpcollect:Config [INFO] Loaded datastore 'MySQL'
Mar 8 10:09:17 SOURCEFIRE-SENSOR1 SF-IMS[20284]: [20284] fpcollect:config [INFO] Configuration read

Do you recommend to shutdown the appliance physically from the power button then powered it up? can this force the DB to initialize normally?

Appreciate you help.

Thank you.

Rami

balaji.bandi · ‎03-08-2021

in a normal situation, you can restart MySQL DB process.

check disk space as suggested or du -h ( i may be thinking space issue or DB run file locked)

> show disk

i would suggest to proper shutdown the kit and bring back online check if that fix the issue ( do you have cisco contract to open a TAC case if is this not resolved)

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Rami Ibrahim · ‎03-08-2021

Here is the disk output, everything seem to be good.

> show disk
Filesystem Size Used Avail Use% Mounted on
/dev/root 2.9G 1.2G 1.6G 44% /
devtmpfs 7.4G 64K 7.4G 1% /dev
/dev/sda1 99M 30M 64M 32% /boot
/dev/sda7 67G 17G 47G 27% /var
none 7.4G 70M 7.3G 1% /dev/shm

# du -h
20K ./.ssh
404K .

Unfortunately we don't have a valid contract right now so we can't open a TAC so we might shutdown the appliance and boot it up again to see if anything changed.

Thank you.

Rami

Rami Ibrahim · ‎03-08-2021

Hello balaji,

I've shutdown the appliance and bring it back up again but that didn't solve the issue. I tried so many things

/etc/rc.d/init.d/network restart

manage_procs.pl with option 3 (Restart Comm. Channel)

I even re-configure the manager statically using the "configure manager" command and that didn't work, it returns an error

getPeersByRole: unable to connect to db at /usr/local/sf/lib/perl/5.10.1/SF/PeerManager/Peers.pm line 107 -

Is there a way to check the database and fix it?

I am getting frustrated

balaji.bandi · ‎03-09-2021

At this stage not sure what direction i can suggest, we need to fix MySQL DB to run - see if that works.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Rami Ibrahim · ‎03-09-2021

Hi Balaji, Thanks for the response.

Actually, I tried many things to fix the DB , though I am not that much experienced with SQL but I'm planning to Re-image the device and that should delete the DB and put a new one.

Am I correct?

Regards,

Rami.