03-12-2014 09:50 AM
After upgrading from 7.1 to 7.5 and all reports appeared fine when trying to login to the VSOM page I get the attached error. It never actually initializes after 24 hours. I can login to VSM and see 2 of 3 service sucessfully started. The Operations Manager overall status report critical. Image 2 attached. Has anyone done and upgrade from 7.1 and seen this and come up with a fix?
Thanks for any input.
Solved! Go to Solution.
03-16-2014 06:13 AM
cwhill,
I did some additional digging for you and it looks like yesterday 3/15 a bug was filed:
https://tools.cisco.com/bugsearch/bug/CSCun40024
The work around is locted in the link if you feel comfortable executing it. If not feel free to open up a TAC case and we will be happy to get you up and running.
03-12-2014 08:09 PM
Some more info in case this helps for advice. When I looked at the catalina.log
/usr/BWhttpd/tomcat/logs/catalina.log
I see this at the tail of the log.
caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation failed:
03-12-2014 09:37 PM
in the CLI run:
sudo su -
service cisco status
------------------------
You will most likely see multiple services not running. If you do see some services not running
run:
service cisco restart
---------------------------
this can take up to ten minutes. Please let us know the outcome.
03-13-2014 09:33 AM
[localadmin@VSOM01 ~]$ sudo su -
Thank you so much for some help on this. Its driving me crazy. I'm suprised others haven't seen this issue as well. I have a fairly new, clean install and this was the first upgrade so there isn't alot of baggage on this server yet. I thought it would have went clean. It looks like all services are running. I pasted the console output below.
[root@VSOM01 ~]# service cisco status
Checking for VSOM database server ... Running.
Checking for tomcat...Running.
Checking for VSOM... running.
Checking for database server ... Running.
Checking for event engine daemon... Running.
Checking for snmp daemon... Running.
Checking for Snmp Server... Running.
Checking for http server... Running.
Checking for LocalAdmin Server... Running.
Checking for lighttpd... Running.
Checking for msi rest... Running.
Checking for msi daemon... Running.
Checking for pending upgrade request
No Pending Upgrade Request
Checking for vsrecorder
Checking for xvcrmanMonitor... Running.
Checking for xvcrman... Running.
Checking for gap server... Running.
Checking for vsms daemons... running.
Checking for scheduler daemon... Running.
Checking for FailoverServer... Running.
Checking for Storage Monitor... Running.
Checking for MediaOut Handler... Running.
Checking for DiscoveryServer... Running.
Checking for Backup Server... Running.
Checking for CmapiServer... Running.
Checking for Analytics Server... Running.
Checking for Analytics Metadata Handler... Running.
Checking for External Trigger Server... Running.
Checking for Device Info Server... Running.
Checking for mrtg... Running.
Checking for System Monitor... Running.
03-14-2014 06:53 AM
Anyone else with thoughts on the issue? I'm going to try tech support at Cisco today to see if I can get anywhere with that. I was hoping to avoid that. =(
05-17-2017 02:41 AM
Dear All,
I tried the same given in this link https://bst.cloudapps.cisco.com/bugsearch/bug/CSCun40024
Also restarted the cisco service but to no avail. Please do help me.
I version of softwares installed is
rpm -aq | grep Cisco
Cisco_VSBase-7.5.1-043d
Cisco_VSMS-7.5.1-043d
Cisco_VSDrivers-7.5.1-043d
Cisco_Tomcat-7.0.37-2
Cisco_VSRecorder-7.5.1-043d
Cisco_MPClient-7.5.0-24
Cisco_SASD-7.5.0-26
Cisco_VSOM-7.5.1-4
Cisco_AMQBroker-7.5.1-1
Cisco_VSMC-7.5.1-004
Cisco_VSTools-7.5.1-043
When checking the logs catalina.log, I get to see the following
Caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation failed: Application is still initializing. Try again after some time
... 25 more
Please help me.
05-17-2017 03:34 AM
In detail, I get the error in catalina.log file as follows
May 17, 2017 4:33:00 PM org.apache.cxf.binding.http.interceptor.DispatchInterceptor handleMessage
INFO: Invoking POST on /getServiceStatus
May 17, 2017 4:33:00 PM org.apache.cxf.phase.PhaseInterceptorChain doIntercept
INFO: Interceptor has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Operation failed: Application is still initializing. Try again after some time
at com.cisco.psbu.vs.ism.interceptors.LoginInterceptor.createFault(LoginInterceptor.java:218)
at com.cisco.psbu.vs.ism.interceptors.LoginInterceptor.handleMessage(LoginInterceptor.java:108)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)
at com.cisco.psbu.vs.ism.main.VsomServletController.invokeDestination(VsomServletController.java:551)
at com.cisco.psbu.vs.ism.main.VsomServletController.invoke(VsomServletController.java:136)
at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:163)
at org.apache.cxf.transport.servlet.AbstractCXFServlet.doPost(AbstractCXFServlet.java:141)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation failed: Application is still initializing. Try again after some time
Please help.
05-17-2017 03:36 AM
error in vsom_be.log
2017-05-17 16:02:06,600 [Thread-38] ERROR c.c.p.v.i.s.u.AuditLogMgmtUtil: Internal Error
org.springframework.jdbc.UncategorizedSQLException: CallableStatementCallback; uncategorized SQLException for SQL [{call partitionAuditLogs(?, ?)}]; SQL state [HY000]; error code [1526]; Table has no partition for value 1495015326000; nested exception is java.sql.SQLException: Table has no partition for value 1495015326000
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83) ~[spring-jdbc-2.5.6.jar:2.5.6]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-2.5.6.jar:2.5.6]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-2.5.6.jar:2.5.6]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:952) ~[spring-jdbc-2.5.6.jar:2.5.6]
at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:985) ~[spring-jdbc-2.5.6.jar:2.5.6]
at org.springframework.jdbc.object.StoredProcedure.execute(StoredProcedure.java:117) ~[spring-jdbc-2.5.6.jar:2.5.6]
at com.cisco.psbu.vs.ism.services.auditlog.impl.AuditLogMgmt.partitionAuditLogs(AuditLogMgmt.java:101) ~[AuditLogMgmt.class:na]
at com.cisco.psbu.vs.ism.services.util.AuditLogMgmtUtil.partitionAuditLogs(AuditLogMgmtUtil.java:335) ~[AuditLogMgmtUtil.class:na]
at com.cisco.psbu.vs.ism.services.internal.VsomServicesInitializer.vsomRestarted(VsomServicesInitializer.java:153) [VsomServicesInitializer.class:na]
at com.cisco.psbu.vs.ism.main.BootServlet$VsomInitializer.run(BootServlet.java:78) [BootServlet$VsomInitializer.class:na]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_17]
Caused by: java.sql.SQLException: Table has no partition for value 1495015326000
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055) ~[mysql-connector-java-5.1.6.jar:na]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956) ~[mysql-connector-java-5.1.6.jar:na]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3491) ~[mysql-connector-java-5.1.6.jar:na]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3423) ~[mysql-connector-java-5.1.6.jar:na]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936) ~[mysql-connector-java-5.1.6.jar:na]
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060) ~[mysql-connector-java-5.1.6.jar:na]
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542) ~[mysql-connector-java-5.1.6.jar:na]
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734) ~[mysql-connector-java-5.1.6.jar:na]
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995) ~[mysql-connector-java-5.1.6.jar:na]
at com.mysql.jdbc.CallableStatement.execute(CallableStatement.java:859) ~[mysql-connector-java-5.1.6.jar:na]
at com.mchange.v2.c3p0.impl.NewProxyCallableStatement.execute(NewProxyCallableStatement.java:3044) ~[c3p0-0.9.1.jar:0.9.1]
at org.springframework.jdbc.core.JdbcTemplate$5.doInCallableStatement(JdbcTemplate.java:987) ~[spring-jdbc-2.5.6.jar:2.5.6]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:936) ~[spring-jdbc-2.5.6.jar:2.5.6]
... 7 common frames omitted
2017-05-17 16:02:06,602 [Thread-38] ERROR c.c.p.v.i.m.BootServlet: Error in handling vsomRestarted
com.cisco.psbu.vs.ism.model.services.VsomException: System error: Server error. Cause org.springframework.jdbc.UncategorizedSQLException CallableStatementCallback; uncategorized SQLException for SQL [{call partitionAuditLogs(?, ?)}]; SQL state [HY000]; error code [1526]; Table has no partition for value 1495015326000; nested exception is java.sql.SQLException: Table has no partition for value 1495015326000
at com.cisco.psbu.vs.ism.services.util.AuditLogMgmtUtil.partitionAuditLogs(AuditLogMgmtUtil.java:350) ~[AuditLogMgmtUtil.class:na]
at com.cisco.psbu.vs.ism.services.internal.VsomServicesInitializer.vsomRestarted(VsomServicesInitializer.java:153) ~[VsomServicesInitializer.class:na]
at com.cisco.psbu.vs.ism.main.BootServlet$VsomInitializer.run(BootServlet.java:78) ~[BootServlet$VsomInitializer.class:na]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_17]
Please help.
05-17-2017 07:01 AM
My initial concern is that your VSOM DB is corrupt. Do you have a recent backup you could restore with via the VSMC/CDAF console?
05-17-2017 10:01 PM
Hi Scott,
Thank you for the advice. Well ! I don't have any backup of VSOM DB. What I am thinking is to reset the whole lot and begin the fresh recording/DB. Or can I have the backup of the VSOM DB, whether it's corrupted or not ?
Please advice.
Regards
05-24-2017 05:31 AM
Dear Team,
I am still with the same issue. Please do help.
Or else, please do help me to reset the DB from scratch so that I can do away with the corrupted DB and have a fresh one please.
Regards
03-16-2014 06:13 AM
cwhill,
I did some additional digging for you and it looks like yesterday 3/15 a bug was filed:
https://tools.cisco.com/bugsearch/bug/CSCun40024
The work around is locted in the link if you feel comfortable executing it. If not feel free to open up a TAC case and we will be happy to get you up and running.
03-16-2014 06:43 PM
Thanks so much. I'll call an open up a case on it! I aprpeciate the help and the response.
03-17-2014 06:09 AM
This is terrific feedback!
I'm also amazed someone managed to find an actual public facing bug where the contents aren't restricted.
Could there actually be positive change happening? Here's to hoping!
03-19-2014 09:02 AM
Thank you again Branden for the help in resolving this. Things are running great again.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide