Error running VSOM in MSP RHEL Server

[Ritchie Nasayao]

Hi,

I have encountered problem in loading VSOM from MSP RHEL Server. Login page does not show value. Error loading /media1 during bootup process of RHEL. It says "mounting local filesystem error" and "mount: structure needs cleaning". After checking service cisco status, VSOM is down and at the end "/media1 is not mounted".

Anyone has idea or has encountered this kind of problem, please advise. Thank you.

[Ritchie Nasayao]

Hi Shanky,

The Cisco services started well. My problem now is that after restarting the physical server, the RHEL is not booting up maybe due to the error as shown in the screenshoot I attached. The physical server is not able finish or complete the boot process. It just hangs. Please see the image I attached for reference.

Thank you very much again.

[Ritchie Nasayao]

Hi Shanky/Scott,

Now I have tried to restart cisco services but failed to load database server:

>>>>

[root@VSOM10 ~]# df -hk
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 7749180 2766424 4582760 38% /
/dev/sdb5 7749200 2964920 4384280 41% /usr/BWhttpd
/dev/sdb3 30997316 1812964 27584352 7% /var
/dev/sdb7 54173584 1405500 49971772 3% /mysql/data
/dev/sda1 147764 11458 128677 9% /boot
tmpfs 2069352 0 2069352 0% /dev/shm
/dev/sdc1 7680363008 2996700 7677366308 1% /media1
[root@VSOM10 ~]#
[root@VSOM10 ~]#
[root@VSOM10 ~]#
[root@VSOM10 ~]# service cisco status
Checking for VSOM database server ... Down.
Checking for tomcat... Down.
Checking for VSOM... down.
Checking for database server ... Down.
Checking for event engine daemon... Down.
Checking for snmp daemon... Down.
Checking for snmp server... Down.
Checking for lighttpd... Down.
Checking for msi rest... Down.
Checking for msi daemon... Down.
Checking for pending upgrade request
No Pending Upgrade Request
Checking for httpserver... Down.
Checking for LocalAdmin Server... Down.
Checking for vsrecorder
Checking for xvcrmanMonitor... Down.
Checking for xvcrman... Down.
Checking for gap server... Down.
Checking for vsms daemons... down.
Checking for scheduler daemon... Down.
Checking for failover server... Down.
Checking for StorageMonitor server... Down.
Checking for mediaout... Down.
Checking for mrtg... Down.
Checking for discovery server... Down.
Checking for backup server... Down.
Checking for Cmapi Server... Down.
Checking for Analytics Server... Down.
Checking for external trigger server... Down.
Checking for Device Info Server... Down.
Checking for SystemMonitor server... Down.
[root@VSOM10 ~]#
[root@VSOM10 ~]#
[root@VSOM10 ~]# service cisco restart
Stopping scheduler daemon... done.
Stopping httpserver... done.
Stopping LocalAdmin Server... done.
Stopping Cmapi Server... done.
Stopping Analytics Server... done.
Stopping external trigger server... done.
Stopping DeviceInfo Server... done.
Stopping mediaout... done.
Stopping StorageMonitor server... done.
Stopping vsrecorder... done.
Stopping failover server... done.
Stopping mrtg... done.
Stopping discovery server... done.
Stopping backup server... done.
Stopping SystemMonitor server... done.
init_upgrade : nothing to do
Stopping lighttpd... done.
Stopping msi rest... done.
Stopping msi daemon... done.
Stopping snmp server... done.
Stopping event engine daemon... done.
Stopping database server... not running.
Stopping VSOM... done
Stopping tomcat ... done
Stopping VSOM database server... not running.

Starting VSOM database server... done.
Starting tomcat ... done
Waiting for ActiveMQ ... done
Starting VSOM... done
Starting database server... /usr/BWhttpd/bin/init_mysql: line 743: local: `Startup of mysql server failed. Please check /usr/BWhttpd/logs/mysql.log': not a valid identifier

Error starting mysql. Aborting cisco start
cisco stop in progress. Aborting
[root@VSOM10 ~]# Stopping scheduler daemon... done.
Stopping httpserver... done.
Stopping LocalAdmin Server... done.
Stopping Cmapi Server... done.
Stopping Analytics Server... done.
Stopping external trigger server... done.
Stopping DeviceInfo Server... done.
Stopping mediaout... done.
Stopping StorageMonitor server... done.
Stopping vsrecorder... done.
Stopping failover server... done.
Stopping mrtg... done.
Stopping discovery server... done.
Stopping backup server... done.
Stopping SystemMonitor server... done.
init_upgrade : nothing to do
Stopping lighttpd... done.
Stopping msi rest... done.
Stopping msi daemon... done.
Stopping snmp server... done.
Stopping event engine daemon... done.
Stopping database server... not running.
Stopping VSOM... done
Stopping tomcat ... done
Killing tomcat ... done.
Stopping VSOM database server... done.

[root@VSOM10 ~]#

>>>>

Please advise. Thank you again.

[Ritchie Nasayao]

I have also tried the following as per known bug fixes but still unable to start database server:

>>>>

[root@VSOM10 ~]# /etc/init.d/cisco stop
Stopping scheduler daemon... done.
Stopping httpserver... done.
Stopping LocalAdmin Server... done.
Stopping Cmapi Server... done.
Stopping Analytics Server... done.
Stopping external trigger server... done.
Stopping DeviceInfo Server... done.
Stopping mediaout... done.
Stopping StorageMonitor server... done.
Stopping vsrecorder... done.
Stopping failover server... done.
Stopping mrtg... done.
Stopping discovery server... done.
Stopping backup server... done.
Stopping SystemMonitor server... done.
init_upgrade : nothing to do
Stopping lighttpd... done.
Stopping msi rest... done.
Stopping msi daemon... done.
Stopping snmp server... done.
Stopping event engine daemon... done.
Stopping database server... not running.
Stopping VSOM... done
Stopping tomcat ... done
Stopping VSOM database server... not running.
[root@VSOM10 ~]#
[root@VSOM10 ~]#
[root@VSOM10 ~]# cd /usr/BWhttpd/mysql/data
[root@VSOM10 data]# chown mysql: UMS ib_logfile0 ib_logfile1 ibdata1 mysql performance_schema
[root@VSOM10 data]# chmod 660 ib_logfile0 ib_logfile1 ibdata1
[root@VSOM10 data]# chmod 700 UMS mysql performance_schema
[root@VSOM10 data]# /etc/init.d/cisco start

Starting VSOM database server... done.
Starting tomcat ... done
Waiting for ActiveMQ ... done
Starting VSOM... done
Starting database server... /usr/BWhttpd/bin/init_mysql: line 743: local: `Startup of mysql server failed. Please check /usr/BWhttpd/logs/mysql.log': not a valid identifier

Error starting mysql. Aborting cisco start
cisco stop in progress. Aborting
[root@VSOM10 data]# Stopping scheduler daemon... done.

>>>>

[Ritchie Nasayao]

you might also have known or encountered this in the log file:

>>>>

151127 17:19:13 mysqld_safe Starting mysqld daemon with databases from /usr/BWhttpd/mysql/data
151127 17:19:13 [Note] Plugin 'FEDERATED' is disabled.
151127 17:19:14 InnoDB: The InnoDB memory heap is disabled
151127 17:19:14 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
151127 17:19:14 InnoDB: Compressed tables use zlib 1.2.3
151127 17:19:14 InnoDB: Using Linux native AIO
151127 17:19:14 InnoDB: Initializing buffer pool, size = 16.0M
151127 17:19:14 InnoDB: Completed initialization of buffer pool
151127 17:19:14 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 2274881017
151127 17:19:14 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2274883935
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1 row operations to undo
InnoDB: Trx id counter is C47C00
151127 17:19:14 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Starting in background the rollback of uncommitted transactions
151127 17:19:15 InnoDB: Rolling back trx with id C47A2D, 1 rows to undo
151127 17:19:15 InnoDB: Waiting for the background threads to start
151127 17:19:15 InnoDB: Assertion failure in thread 2963983248 in file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:19:15 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=2000
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 188751 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/BWhttpd/mysql/bin/mysqld(my_print_stacktrace+0x33)[0x83ec593]
/usr/BWhttpd/mysql/bin/mysqld(handle_fatal_signal+0x41c)[0x82adf5c]
[0xfb8420]
/lib/libc.so.6(abort+0x101)[0x178701]
/usr/BWhttpd/mysql/bin/mysqld[0x853fb85]
/usr/BWhttpd/mysql/bin/mysqld[0x853fd36]
/usr/BWhttpd/mysql/bin/mysqld[0x84b9de5]
/usr/BWhttpd/mysql/bin/mysqld[0x84d132c]
/usr/BWhttpd/mysql/bin/mysqld[0x84cd91c]
/usr/BWhttpd/mysql/bin/mysqld[0x84cddc6]
/usr/BWhttpd/mysql/bin/mysqld[0x84c25ea]
/usr/BWhttpd/mysql/bin/mysqld[0x8589323]
/usr/BWhttpd/mysql/bin/mysqld[0x8589550]
/usr/BWhttpd/mysql/bin/mysqld[0x84c425c]
/usr/BWhttpd/mysql/bin/mysqld[0x84c4a0b]
/lib/libpthread.so.0[0x2e1852]
/lib/libc.so.6(clone+0x5e)[0x22084e]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
151127 17:19:15 mysqld_safe mysqld from pid file /usr/BWhttpd/mysql/data/mysql.pid ended

>>>>

[Scott Olsen]

Ouch.  Something has really damaged your mySQL installation as far as I can tell. Your best course of action (in my opinion) at this point would be to try and locate a valid VSOM backup and refresh the server.

I dont know your particular situation.  If blowing away the server and restoring from recovery media & backup isn't an option, you *might* be able to manually try repairing the mySQL install... but it's likely to be a rough go :-S

What chassis is this server running on?  It sounds as though the server went down *very* hard at a most inopportune time.

Wish I had an easy fix for you.

[Ritchie Nasayao]

I need to put the server back into action. I need to resort with the quickest restore action. This is running on MSP server. I am not sure how it went through during a shutdown whether if it was proper or not.

[Scott Olsen]

Understood.  The quickest path is likely to be a factory restore of the chassis.  I recommend reviewing (and bookmarking!) all the information mapped out here:

http://www.cisco.com/c/en/us/td/docs/security/physical_security/video_surveillance/network/vsm/roadmap/vsm_7_roadmap.html#pgfId-48992

... in particular, the documentation on restoring an MSP platform back to a functional state.

There was a much older, and more complicated restore image that you might see referenced throughout documentation.  Cisco has since created a superior recovery image that can be written to a flashdrive with Win32DiskImager.  You can get it here:

*Be sure to read the Release Notes!

All the best.  Cheers!

[Shanky Burman · Accepted solution]

Hi Ritchie,

I think the option we are trying to do repair is not  helping us here,

Please re-run the below command to repair, also follow the above the above the above steps get the partition up.

++ Run xfs_repair for /media1

  #  xfs_repair -L /dev/sdc1 -l /dev/sdc2

-L represent Force Log Zeroing, let me know if any query.

Thanks

Shanky

[Scott Olsen · Accepted solution]

I'm not sure why you are deleting any files anywhere, Ritchie.  The mrtg.cfg file is completely unrelated to the issue.

If you've run the xfs_repair binary with the -n flag, it's only going to report on the issues it locates, and not actually modify or repair them.  I was hoping you'd share the output with the forum before proceeding, but ultimately that's up to you.

Shanky's post also mentions the xfs_repair command, but it *will* modify the filesystem (and likely correct your problem).