This document was generated from CDN thread
Created by: Julio Rodriguez on 14-05-2012 02:25:47 PM
Hello,
We are running into an issue where the CVPReporting.tmp file is getting written too. When it reaches the max (we have changed it to 250mb) the server goes in to partial state. Even whem moving it to 250mb, the file gets filled up within about 1 1/2 hrs. I have to be shutting down the call server and renaming the file in order to bring the server back up.
What I have read is that this file is written to when the DB disconnects and should delete after the DB reconnects. It is constantly being written to indicating to me that the DB is not finctioning properly.
I am able to change the passwords for the users and still have the same issue.
I cannot create a new user or run the DB purge.
According to the ORM log, I get the following messages:
tryint to create user
70: 10.9.5.41: May 14 2012 14:14:07.935 -0400: %CVP_7_0_ORM-3-ORM_ERROR_GENERAL: Script createUser execution failed, reason=-1
com.cisco.cvp.oamp.ocom.mbean.ScriptException: Script execution failed
:\Cisco\CVP\bin\cvp4dbBackend
at com.cisco.cvp.oamp.orm.utility.ScriptUtil.executeScript(ScriptUtil.java:185)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.executeDBOperation(SubsystemReportingImpl.java:1827)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.addUser(SubsystemReportingImpl.java:1882)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414)
at javax.management.StandardMBean.invoke(StandardMBean.java:323)
at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410)
at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1343)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)
[id:8001]
Trying to purge DB
74: 10.9.5.41: May 14 2012 14:24:19.872 -0400: %CVP_7_0_ORM-3-ORM_ERROR_GENERAL: Script schedulePurge execution failed, reason=-1
com.cisco.cvp.oamp.ocom.mbean.ScriptException: Script execution failed:\Cisco\CVP\bin\cvp4dbBackend
at com.cisco.cvp.oamp.orm.utility.ScriptUtil.executeScript(ScriptUtil.java:185)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.executeDBOperation(SubsystemReportingImpl.java:1827)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.schedulePurge(SubsystemReportingImpl.java:2219)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414)
at javax.management.StandardMBean.invoke(StandardMBean.java:323)
at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410)
at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1343)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)
[id:8001]
Anyone run into this before?
Thanks
Subject: RE: CVP reporting server error
Replied by: Ryan Hilfers on 15-05-2012 10:45:53 AM
Hey there, I have recently encountered this exact issue. You will need to apply ES6 and also ET_11 (ET_11 is a fix for a bug introduced in ES6... ridiculous I know..) Looks like you're hitting this defect (CSCtu43570 - Reporting server failover file
grows out of control exceeding setting) and possibly caused by (CSCto54083 - Reporting server fails with -710
resulting in out of memory condition)
I would consult TAC and get this resolved... definitely a bug if your tmp file is exceeding it's max size!
-Ryan
Subject: RE: CVP reporting server error
Replied by: Ryan Hilfers on 15-05-2012 10:50:25 AM
Aaack, I may have read your question a bit too quickly.. might not be the exact issue I was referencing, nonetheless consult TAC, especially if this is CVP8x since there are known defects.
-Ryan
Subject: RE: CVP reporting server error
Replied by: Julio Rodriguez on 15-05-2012 12:39:22 PM
I appreciate your response Ryan,
I was told by TAC that there is a fix for the file size causing the server to go into partial state in version 8.5 ES6. We are currently on 7.0(2) and need to do quite a bit to get to 8.x. My thing is that my server should not be writing to this file unless the DB is disconnected and should be deleteing the file once thje DB connets. Thsi is never happening. I do not know much about informix to know how to check this. The services are all running. I
Whenever I restart the server, I get this in the logs.
3: 10.9.5.41: May 15 2012 08:54:57.585 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
4: 10.9.5.41: May 15 2012 08:54:57.585 -0400: %CVP_7_0_RPT-1-REPORTING_STATE_CHANGE: REPORTING Subsystem state changed to RPT SS RPT1 changes its state to Partial Service cause Unfinished calls recovery started [id:4001]
6: 10.9.5.41: May 15 2012 08:54:57.601 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 0's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
5: 10.9.5.41: May 15 2012 08:54:59.882 -0400: %CVP_7_0_MessagingLayer-1-PLUGIN_INITIALIZATION_SUCCESS: SYS_RPT2 Connection Successful: host=Socket QoS Level set to: 0 Plugin initialization and configuration succeeded [id:10]
9: 10.9.5.41: May 15 2012 08:54:59.882 -0400: %CVP_7_0_MessagingLayer-1-PLUGIN_INITIALIZATION_SUCCESS: Received an ACK from the server. Plugin initialization and configuration succeeded [id:10]
10: 10.9.5.41: May 15 2012 08:54:59.913 -0400: %CVP_7_0_MessagingLayer-1-PLUGIN_INITIALIZATION_SUCCESS: SYS_RPT1 Connection Successful: host=Socket QoS Level set to: 0 Plugin initialization and configuration succeeded [id:10]
14: 10.9.5.41: May 15 2012 08:54:59.913 -0400: %CVP_7_0_MessagingLayer-1-PLUGIN_INITIALIZATION_SUCCESS: Received an ACK from the server. Plugin initialization and configuration succeeded [id:10]
18: 10.9.5.41: May 15 2012 08:55:07.601 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 1's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
21: 10.9.5.41: May 15 2012 08:55:17.601 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 2's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
24: 10.9.5.41: May 15 2012 08:55:27.617 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 3's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
27: 10.9.5.41: May 15 2012 08:55:37.696 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 4's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
30: 10.9.5.41: May 15 2012 08:55:47.696 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 5's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
33: 10.9.5.41: May 15 2012 08:55:57.696 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 6's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
36: 10.9.5.41: May 15 2012 08:56:07.697 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 7's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
39: 10.9.5.41: May 15 2012 08:56:17.697 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 8's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
42: 10.9.5.41: May 15 2012 08:56:27.697 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 9's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
45: 10.9.5.41: May 15 2012 08:56:37.697 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 10's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
48: 10.9.5.41: May 15 2012 08:56:47.698 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 11's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
51: 10.9.5.41: May 15 2012 08:56:57.698 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 12's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
54: 10.9.5.41: May 15 2012 08:57:07.698 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 13's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
57: 10.9.5.41: May 15 2012 08:57:17.699 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 14's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
60: 10.9.5.41: May 15 2012 08:57:27.699 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 15's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
63: 10.9.5.41: May 15 2012 08:57:37.699 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 16's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
66: 10.9.5.41: May 15 2012 08:57:47.700 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 17's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
69: 10.9.5.41: May 15 2012 08:57:57.700 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 18's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
72: 10.9.5.41: May 15 2012 08:58:07.700 -0400: %CVP_7_0_RPT-3-REPORTING_DB_ERROR_MSG: getConnectionForDatabaseInit -- Cannot open system catalog (systables). [id:4012]
100: 10.9.5.41: May 15 2012 08:58:07.732 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
104: 10.9.5.41: May 15 2012 08:58:07.732 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
134: 10.9.5.41: May 15 2012 08:58:07.747 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
140: 10.9.5.41: May 15 2012 08:58:07.747 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
144: 10.9.5.41: May 15 2012 08:58:07.763 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
161: 10.9.5.41: May 15 2012 08:58:07.763 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
200: 10.9.5.41: May 15 2012 08:58:07.778 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
238: 10.9.5.41: May 15 2012 08:58:07.794 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
268: 10.9.5.41: May 15 2012 08:58:07.794 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
277: 10.9.5.41: May 15 2012 08:58:07.810 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
311: 10.9.5.41: May 15 2012 08:58:07.810 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
443: 10.9.5.41: May 15 2012 08:58:07.825 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
445: 10.9.5.41: May 15 2012 08:58:07.841 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
490: 10.9.5.41: May 15 2012 08:58:07.872 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
500: 10.9.5.41: May 15 2012 08:58:07.872 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
630: 10.9.5.41: May 15 2012 08:58:07.903 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
642: 10.9.5.41: May 15 2012 08:58:07.903 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
724: 10.9.5.41: May 15 2012 08:58:07.919 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
734: 10.9.5.41: May 15 2012 08:58:07.919 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
864: 10.9.5.41: May 15 2012 08:58:07.935 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
1012: 10.9.5.41: May 15 2012 08:58:07.950 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
1027: 10.9.5.41: May 15 2012 08:58:07.950 -0400: %CVP_7_0_RPT-1-REPORTING_STATE_CHANGE: REPORTING Subsystem state changed to RPT SS RPT1 changes its state to In Service cause Unfinished calls recovery ended [id:4001]
It is not until the 18's try that the server comes back up in oamp.
I think I have a bigger issue which is causing the ftemp file to be writtne to.
Subject: RE: CVP reporting server error
Replied by: Ryan Hilfers on 15-05-2012 01:18:41 PM
Julio, two things:
Check to be sure your "Informix"/"cvp_dbadmin"/"cvp_dbuser" user's passwords are not set to expire on the machine... Check in Local Users and Groups.
Given this is a 7x CVP Server I would not expect issues to be happening.. I can see your message bus connections establish fine with the Call Servers but there is definitely an issue when trying to write to the database... although it claims to be back in service due to "completed the failover processing" it looks like the recovery entries failed as well for the same SQL exceptions... I wish I had some better advice for you, in my currenty environment we didn't have any issues until we upgraded to 8.5 and eventually found out we needed ES6. Do you see it making database writes via OAMP statistics? I'd assume not. Run TCPView or netstat and see if Informix is listening on expected ports. have you looked at the diag page on that CVP? (there isn't a lot to gleem from a reporting server but you can dump RPT). Perhaps you need to escalate to a different TAC engineer, they should really help you get this resolved (especially a JDBC connection failure).
Also, side note... you can upgrade the reporting server first to 8.5 without changing any other components, check out the upgrade guide for CVP8. Perhaps that is a possiblity and an upgrade may resolve the connection issues but as you said there is something else happening here.
Subject: RE: CVP reporting server error
Replied by: Julio Rodriguez on 15-05-2012 01:30:38 PM
This is great info. Did not know that the reporting server could be upgraded on its own.
TAC has escalated it to the BU, which I don't understand why, but I am just waiting for their response and trying to figure it out in the meantime. I am tempted to do a database restore but not sure what I can lose by doing this. The statistics shows no writing to the DB so I may not be losing much, plus I have all these temp files that I may be able to load to the DB after wards.
Created by: Julio Rodriguez on 14-05-2012 02:25:47 PM
Hello,
We are running into an issue where the CVPReporting.tmp file is getting written too. When it reaches the max (we have changed it to 250mb) the server goes in to partial state. Even whem moving it to 250mb, the file gets filled up within about 1 1/2 hrs. I have to be shutting down the call server and renaming the file in order to bring the server back up.
What I have read is that this file is written to when the DB disconnects and should delete after the DB reconnects. It is constantly being written to indicating to me that the DB is not finctioning properly.
I am able to change the passwords for the users and still have the same issue.
I cannot create a new user or run the DB purge.
According to the ORM log, I get the following messages:
tryint to create user
70: 10.9.5.41: May 14 2012 14:14:07.935 -0400: %CVP_7_0_ORM-3-ORM_ERROR_GENERAL: Script createUser execution failed, reason=-1
com.cisco.cvp.oamp.ocom.mbean.ScriptException: Script execution failed
:\Cisco\CVP\bin\cvp4dbBackendat com.cisco.cvp.oamp.orm.utility.ScriptUtil.executeScript(ScriptUtil.java:185)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.executeDBOperation(SubsystemReportingImpl.java:1827)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.addUser(SubsystemReportingImpl.java:1882)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414)
at javax.management.StandardMBean.invoke(StandardMBean.java:323)
at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410)
at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1343)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)
[id:8001]
Trying to purge DB
74: 10.9.5.41: May 14 2012 14:24:19.872 -0400: %CVP_7_0_ORM-3-ORM_ERROR_GENERAL: Script schedulePurge execution failed, reason=-1
com.cisco.cvp.oamp.ocom.mbean.ScriptException: Script execution failed
:\Cisco\CVP\bin\cvp4dbBackendat com.cisco.cvp.oamp.orm.utility.ScriptUtil.executeScript(ScriptUtil.java:185)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.executeDBOperation(SubsystemReportingImpl.java:1827)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.schedulePurge(SubsystemReportingImpl.java:2219)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414)
at javax.management.StandardMBean.invoke(StandardMBean.java:323)
at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410)
at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1343)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)
[id:8001]
Anyone run into this before?
Thanks
Subject: RE: CVP reporting server error
Replied by: Ryan Hilfers on 15-05-2012 10:45:53 AM
Hello,
We are running into an issue where the CVPReporting.tmp file is getting written too. When it reaches the max (we have changed it to 250mb) the server goes in to partial state. Even whem moving it to 250mb, the file gets filled up within about 1 1/2 hrs. I have to be shutting down the call server and renaming the file in order to bring the server back up.
What I have read is that this file is written to when the DB disconnects and should delete after the DB reconnects. It is constantly being written to indicating to me that the DB is not finctioning properly.
I am able to change the passwords for the users and still have the same issue.
I cannot create a new user or run the DB purge.
According to the ORM log, I get the following messages:
tryint to create user
70: 10.9.5.41: May 14 2012 14:14:07.935 -0400: %CVP_7_0_ORM-3-ORM_ERROR_GENERAL: Script createUser execution failed, reason=-1
com.cisco.cvp.oamp.ocom.mbean.ScriptException: Script execution failed
at com.cisco.cvp.oamp.orm.utility.ScriptUtil.executeScript(ScriptUtil.java:185)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.executeDBOperation(SubsystemReportingImpl.java:1827)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.addUser(SubsystemReportingImpl.java:1882)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414)
at javax.management.StandardMBean.invoke(StandardMBean.java:323)
at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410)
at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1343)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)
[id:8001]
Trying to purge DB
74: 10.9.5.41: May 14 2012 14:24:19.872 -0400: %CVP_7_0_ORM-3-ORM_ERROR_GENERAL: Script schedulePurge execution failed, reason=-1
com.cisco.cvp.oamp.ocom.mbean.ScriptException: Script execution failed
at com.cisco.cvp.oamp.orm.utility.ScriptUtil.executeScript(ScriptUtil.java:185)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.executeDBOperation(SubsystemReportingImpl.java:1827)
at com.cisco.cvp.oamp.orm.mbean.pc.impl.SubsystemReportingImpl.schedulePurge(SubsystemReportingImpl.java:2219)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414)
at javax.management.StandardMBean.invoke(StandardMBean.java:323)
at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410)
at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1343)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)
[id:8001]
Anyone run into this before?
Thanks
Hey there, I have recently encountered this exact issue. You will need to apply ES6 and also ET_11 (ET_11 is a fix for a bug introduced in ES6... ridiculous I know..) Looks like you're hitting this defect (CSCtu43570 - Reporting server failover file
grows out of control exceeding setting) and possibly caused by (CSCto54083 - Reporting server fails with -710
resulting in out of memory condition)
I would consult TAC and get this resolved... definitely a bug if your tmp file is exceeding it's max size!
-Ryan
Subject: RE: CVP reporting server error
Replied by: Ryan Hilfers on 15-05-2012 10:50:25 AM
Aaack, I may have read your question a bit too quickly.. might not be the exact issue I was referencing, nonetheless consult TAC, especially if this is CVP8x since there are known defects.
-Ryan
Subject: RE: CVP reporting server error
Replied by: Julio Rodriguez on 15-05-2012 12:39:22 PM
I appreciate your response Ryan,
I was told by TAC that there is a fix for the file size causing the server to go into partial state in version 8.5 ES6. We are currently on 7.0(2) and need to do quite a bit to get to 8.x. My thing is that my server should not be writing to this file unless the DB is disconnected and should be deleteing the file once thje DB connets. Thsi is never happening. I do not know much about informix to know how to check this. The services are all running. I
Whenever I restart the server, I get this in the logs.
3: 10.9.5.41: May 15 2012 08:54:57.585 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
4: 10.9.5.41: May 15 2012 08:54:57.585 -0400: %CVP_7_0_RPT-1-REPORTING_STATE_CHANGE: REPORTING Subsystem state changed to RPT SS RPT1 changes its state to Partial Service cause Unfinished calls recovery started [id:4001]
6: 10.9.5.41: May 15 2012 08:54:57.601 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 0's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
5: 10.9.5.41: May 15 2012 08:54:59.882 -0400: %CVP_7_0_MessagingLayer-1-PLUGIN_INITIALIZATION_SUCCESS: SYS_RPT2 Connection Successful: host=Socket QoS Level set to: 0 Plugin initialization and configuration succeeded [id:10]
9: 10.9.5.41: May 15 2012 08:54:59.882 -0400: %CVP_7_0_MessagingLayer-1-PLUGIN_INITIALIZATION_SUCCESS: Received an ACK from the server. Plugin initialization and configuration succeeded [id:10]
10: 10.9.5.41: May 15 2012 08:54:59.913 -0400: %CVP_7_0_MessagingLayer-1-PLUGIN_INITIALIZATION_SUCCESS: SYS_RPT1 Connection Successful: host=Socket QoS Level set to: 0 Plugin initialization and configuration succeeded [id:10]
14: 10.9.5.41: May 15 2012 08:54:59.913 -0400: %CVP_7_0_MessagingLayer-1-PLUGIN_INITIALIZATION_SUCCESS: Received an ACK from the server. Plugin initialization and configuration succeeded [id:10]
18: 10.9.5.41: May 15 2012 08:55:07.601 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 1's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
21: 10.9.5.41: May 15 2012 08:55:17.601 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 2's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
24: 10.9.5.41: May 15 2012 08:55:27.617 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 3's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
27: 10.9.5.41: May 15 2012 08:55:37.696 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 4's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
30: 10.9.5.41: May 15 2012 08:55:47.696 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 5's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
33: 10.9.5.41: May 15 2012 08:55:57.696 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 6's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
36: 10.9.5.41: May 15 2012 08:56:07.697 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 7's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
39: 10.9.5.41: May 15 2012 08:56:17.697 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 8's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
42: 10.9.5.41: May 15 2012 08:56:27.697 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 9's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
45: 10.9.5.41: May 15 2012 08:56:37.697 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 10's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
48: 10.9.5.41: May 15 2012 08:56:47.698 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 11's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
51: 10.9.5.41: May 15 2012 08:56:57.698 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 12's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
54: 10.9.5.41: May 15 2012 08:57:07.698 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 13's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
57: 10.9.5.41: May 15 2012 08:57:17.699 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 14's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
60: 10.9.5.41: May 15 2012 08:57:27.699 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 15's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
63: 10.9.5.41: May 15 2012 08:57:37.699 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 16's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
66: 10.9.5.41: May 15 2012 08:57:47.700 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 17's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
69: 10.9.5.41: May 15 2012 08:57:57.700 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: recoverCallRegistry() 18's try -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
72: 10.9.5.41: May 15 2012 08:58:07.700 -0400: %CVP_7_0_RPT-3-REPORTING_DB_ERROR_MSG: getConnectionForDatabaseInit -- Cannot open system catalog (systables). [id:4012]
100: 10.9.5.41: May 15 2012 08:58:07.732 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
104: 10.9.5.41: May 15 2012 08:58:07.732 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
134: 10.9.5.41: May 15 2012 08:58:07.747 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
140: 10.9.5.41: May 15 2012 08:58:07.747 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
144: 10.9.5.41: May 15 2012 08:58:07.763 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
161: 10.9.5.41: May 15 2012 08:58:07.763 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
200: 10.9.5.41: May 15 2012 08:58:07.778 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
238: 10.9.5.41: May 15 2012 08:58:07.794 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
268: 10.9.5.41: May 15 2012 08:58:07.794 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
277: 10.9.5.41: May 15 2012 08:58:07.810 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
311: 10.9.5.41: May 15 2012 08:58:07.810 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
443: 10.9.5.41: May 15 2012 08:58:07.825 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
445: 10.9.5.41: May 15 2012 08:58:07.841 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
490: 10.9.5.41: May 15 2012 08:58:07.872 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
500: 10.9.5.41: May 15 2012 08:58:07.872 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
630: 10.9.5.41: May 15 2012 08:58:07.903 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
642: 10.9.5.41: May 15 2012 08:58:07.903 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
724: 10.9.5.41: May 15 2012 08:58:07.919 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
734: 10.9.5.41: May 15 2012 08:58:07.919 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: Cannot open system catalog (systables). [id:4003]
864: 10.9.5.41: May 15 2012 08:58:07.935 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
1012: 10.9.5.41: May 15 2012 08:58:07.950 -0400: %CVP_7_0_RPT-3-REPORTING_INTERNAL_ERROR: initAllConnections -- java.sql.SQLException: Cannot open system catalog (systables). [id:4003]
1027: 10.9.5.41: May 15 2012 08:58:07.950 -0400: %CVP_7_0_RPT-1-REPORTING_STATE_CHANGE: REPORTING Subsystem state changed to RPT SS RPT1 changes its state to In Service cause Unfinished calls recovery ended [id:4001]
It is not until the 18's try that the server comes back up in oamp.
I think I have a bigger issue which is causing the ftemp file to be writtne to.
Subject: RE: CVP reporting server error
Replied by: Ryan Hilfers on 15-05-2012 01:18:41 PM
Julio, two things:
Check to be sure your "Informix"/"cvp_dbadmin"/"cvp_dbuser" user's passwords are not set to expire on the machine... Check in Local Users and Groups.
Given this is a 7x CVP Server I would not expect issues to be happening.. I can see your message bus connections establish fine with the Call Servers but there is definitely an issue when trying to write to the database... although it claims to be back in service due to "completed the failover processing" it looks like the recovery entries failed as well for the same SQL exceptions... I wish I had some better advice for you, in my currenty environment we didn't have any issues until we upgraded to 8.5 and eventually found out we needed ES6. Do you see it making database writes via OAMP statistics? I'd assume not. Run TCPView or netstat and see if Informix is listening on expected ports. have you looked at the diag page on that CVP? (there isn't a lot to gleem from a reporting server but you can dump RPT). Perhaps you need to escalate to a different TAC engineer, they should really help you get this resolved (especially a JDBC connection failure).
Also, side note... you can upgrade the reporting server first to 8.5 without changing any other components, check out the upgrade guide for CVP8. Perhaps that is a possiblity and an upgrade may resolve the connection issues but as you said there is something else happening here.
Subject: RE: CVP reporting server error
Replied by: Julio Rodriguez on 15-05-2012 01:30:38 PM
This is great info. Did not know that the reporting server could be upgraded on its own.
TAC has escalated it to the BU, which I don't understand why, but I am just waiting for their response and trying to figure it out in the meantime. I am tempted to do a database restore but not sure what I can lose by doing this. The statistics shows no writing to the DB so I may not be losing much, plus I have all these temp files that I may be able to load to the DB after wards.