VSM 7.5 Upgrade Application Failed Still initializing

[cwhill]

After upgrading from 7.1 to 7.5 and all reports appeared fine when trying to login to the VSOM page I get the attached error. It never actually initializes after 24 hours. I can login to VSM and see 2 of 3 service sucessfully started. The Operations Manager overall status report critical. Image 2 attached. Has anyone done and upgrade from 7.1 and seen this and come up with a fix?

Thanks for any input.

[cwhill]

Some more info in case this helps for advice. When I looked at the catalina.log 

/usr/BWhttpd/tomcat/logs/catalina.log

I see this at the tail of the log.

caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation failed:

[Branden Varney]

in the CLI run:

sudo su -

service cisco status

------------------------

You will most likely see multiple services not running. If you do see some services not running

run:

service cisco restart

---------------------------

this can take up to ten minutes. Please let us know the outcome.

[cwhill]

[localadmin@VSOM01 ~]$ sudo su -
Thank you so much for some help on this. Its driving me crazy. I'm suprised others haven't seen this issue as well. I have a fairly new, clean install and this was the first upgrade so there isn't alot of baggage on this server yet. I thought it would have went clean. It looks like all services are running. I pasted the console output below.

[root@VSOM01 ~]# service cisco status
Checking for VSOM database server ... Running.
Checking for tomcat...Running.
Checking for VSOM... running.
Checking for database server ... Running.
Checking for event engine daemon... Running.
Checking for snmp daemon... Running.
Checking for Snmp Server... Running.
Checking for http server... Running.
Checking for LocalAdmin Server... Running.
Checking for lighttpd... Running.
Checking for msi rest... Running.
Checking for msi daemon... Running.
Checking for pending upgrade request
    No Pending Upgrade Request
Checking for vsrecorder
    Checking for xvcrmanMonitor... Running.
    Checking for xvcrman... Running.
    Checking for gap server... Running.
Checking for vsms daemons... running.
Checking for scheduler daemon... Running.
Checking for FailoverServer... Running.
Checking for Storage Monitor... Running.
Checking for MediaOut Handler... Running.
Checking for DiscoveryServer... Running.
Checking for Backup Server... Running.
Checking for CmapiServer... Running.
Checking for Analytics Server... Running.
Checking for Analytics Metadata Handler... Running.
Checking for External Trigger Server... Running.
Checking for Device Info Server... Running.
Checking for mrtg... Running.
Checking for System Monitor... Running.

[cwhill]

Anyone else with thoughts on the issue? I'm going to try tech support at Cisco today to see if I can get anywhere with that. I was hoping to avoid that.  =(

[tsenden866]

Dear All,

I tried the same given in this link  https://bst.cloudapps.cisco.com/bugsearch/bug/CSCun40024

Also restarted the cisco service but to no avail. Please do help me.

I version of softwares installed is

 rpm -aq | grep Cisco
Cisco_VSBase-7.5.1-043d
Cisco_VSMS-7.5.1-043d
Cisco_VSDrivers-7.5.1-043d
Cisco_Tomcat-7.0.37-2
Cisco_VSRecorder-7.5.1-043d
Cisco_MPClient-7.5.0-24
Cisco_SASD-7.5.0-26
Cisco_VSOM-7.5.1-4
Cisco_AMQBroker-7.5.1-1
Cisco_VSMC-7.5.1-004
Cisco_VSTools-7.5.1-043

When checking the logs catalina.log, I get to see the following

Caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation failed: Application is still initializing. Try again after some time
        ... 25 more

Please help me.

[tsenden866]

In detail, I get the error in catalina.log file as follows

May 17, 2017 4:33:00 PM org.apache.cxf.binding.http.interceptor.DispatchInterceptor handleMessage
INFO: Invoking POST on /getServiceStatus
May 17, 2017 4:33:00 PM org.apache.cxf.phase.PhaseInterceptorChain doIntercept
INFO: Interceptor has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Operation failed: Application is still initializing. Try again after some time
        at com.cisco.psbu.vs.ism.interceptors.LoginInterceptor.createFault(LoginInterceptor.java:218)
        at com.cisco.psbu.vs.ism.interceptors.LoginInterceptor.handleMessage(LoginInterceptor.java:108)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
        at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)
        at com.cisco.psbu.vs.ism.main.VsomServletController.invokeDestination(VsomServletController.java:551)
        at com.cisco.psbu.vs.ism.main.VsomServletController.invoke(VsomServletController.java:136)
        at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:163)
        at org.apache.cxf.transport.servlet.AbstractCXFServlet.doPost(AbstractCXFServlet.java:141)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation failed: Application is still initializing. Try again after some time

Please help.

[tsenden866]

error in vsom_be.log

2017-05-17 16:02:06,600 [Thread-38] ERROR c.c.p.v.i.s.u.AuditLogMgmtUtil: Internal Error
org.springframework.jdbc.UncategorizedSQLException: CallableStatementCallback; uncategorized SQLException for SQL [{call partitionAuditLogs(?, ?)}]; SQL state [HY000]; error code [1526]; Table has no partition for value 1495015326000; nested exception is java.sql.SQLException: Table has no partition for value 1495015326000
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:952) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:985) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.object.StoredProcedure.execute(StoredProcedure.java:117) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at com.cisco.psbu.vs.ism.services.auditlog.impl.AuditLogMgmt.partitionAuditLogs(AuditLogMgmt.java:101) ~[AuditLogMgmt.class:na]
        at com.cisco.psbu.vs.ism.services.util.AuditLogMgmtUtil.partitionAuditLogs(AuditLogMgmtUtil.java:335) ~[AuditLogMgmtUtil.class:na]
        at com.cisco.psbu.vs.ism.services.internal.VsomServicesInitializer.vsomRestarted(VsomServicesInitializer.java:153) [VsomServicesInitializer.class:na]
        at com.cisco.psbu.vs.ism.main.BootServlet$VsomInitializer.run(BootServlet.java:78) [BootServlet$VsomInitializer.class:na]
        at java.lang.Thread.run(Unknown Source) [na:1.7.0_17]
Caused by: java.sql.SQLException: Table has no partition for value 1495015326000
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3491) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3423) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.CallableStatement.execute(CallableStatement.java:859) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mchange.v2.c3p0.impl.NewProxyCallableStatement.execute(NewProxyCallableStatement.java:3044) ~[c3p0-0.9.1.jar:0.9.1]
        at org.springframework.jdbc.core.JdbcTemplate$5.doInCallableStatement(JdbcTemplate.java:987) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:936) ~[spring-jdbc-2.5.6.jar:2.5.6]
        ... 7 common frames omitted
2017-05-17 16:02:06,602 [Thread-38] ERROR c.c.p.v.i.m.BootServlet: Error in handling vsomRestarted
com.cisco.psbu.vs.ism.model.services.VsomException: System error: Server error. Cause org.springframework.jdbc.UncategorizedSQLException CallableStatementCallback; uncategorized SQLException for SQL [{call partitionAuditLogs(?, ?)}]; SQL state [HY000]; error code [1526]; Table has no partition for value 1495015326000; nested exception is java.sql.SQLException: Table has no partition for value 1495015326000
        at com.cisco.psbu.vs.ism.services.util.AuditLogMgmtUtil.partitionAuditLogs(AuditLogMgmtUtil.java:350) ~[AuditLogMgmtUtil.class:na]
        at com.cisco.psbu.vs.ism.services.internal.VsomServicesInitializer.vsomRestarted(VsomServicesInitializer.java:153) ~[VsomServicesInitializer.class:na]
        at com.cisco.psbu.vs.ism.main.BootServlet$VsomInitializer.run(BootServlet.java:78) ~[BootServlet$VsomInitializer.class:na]
        at java.lang.Thread.run(Unknown Source) [na:1.7.0_17]

Please help.

[Scott Olsen]

My initial concern is that your VSOM DB is corrupt.  Do you have a recent backup you could restore with via the VSMC/CDAF console?

[tsenden866]

Hi Scott,

Thank you for the advice. Well ! I don't have any backup of VSOM DB. What I am thinking is to reset the whole lot and begin the fresh recording/DB. Or can I have the backup of the VSOM DB, whether it's corrupted or not ?

Please advice.

Regards

[tsenden866]

Dear Team,

I am still with the same issue. Please do help.

Or else, please do help me to reset the DB from scratch so that I can do away with the corrupted DB and have a fresh one please.

Regards

[Branden Varney · Accepted solution]

cwhill,

   I did some additional digging for you and it looks like yesterday 3/15 a bug was filed:

https://tools.cisco.com/bugsearch/bug/CSCun40024

The work around is locted in the link if you feel comfortable executing it. If not feel free to open up a TAC case and we will be happy to get you up and running.

[cwhill]

Thanks so much. I'll call an open up a case on it! I aprpeciate the help and the response.

[Scott Olsen]

This is terrific feedback!

I'm also amazed someone managed to find an actual public facing bug where the contents aren't restricted.  

Could there actually be positive change happening?  Here's to hoping!

[cwhill]

Thank you again Branden for the help in resolving this. Things are running great again.