LMS 3.2(solaris 10) - User Tracking: again "ogs_server_urn" Not found! errors

[Martin Ermel]

LMS 3.2 on a solaris 10 server that had problems with ctm_config.txt in the past (https://supportforums.cisco.com/thread/2033941) now again has problems with the device selctor in UT. There are FATAL errors of the below type logged to CampusOGSServer.log:

[ Tue Oct 19  18:27:22 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /Campus@lms-server/Campus@sdeu1121/System Defined Groups/All Unreachable Devices
[ Tue Oct 19  18:30:52 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /Campus@lms-server
[ Tue Oct 19  18:33:03 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /Campus@lms-server/Campus@sdeu1121/System Defined Groups/All Unreachable Devices
[ Tue Oct 19  18:35:04 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /RME@lms-server/Pre-deployed
[ Tue Oct 19  18:35:05 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /RME@lms-server/User Defined Groups

To create the failure I go to UT > Reports > Report Generator => Switch Port Report > Switch Port Summary
     The Device Selector that opens does not contain any devices and clicking on the Search Button produces the following error:

Problem with File /WEB-INF/screens/deviceselector/DeviceFilter.jsp!!!URN_NOT_FOUND : urn "ogs_server_urn" : Not found !!

also the following error is found in /opt/CSCOpx/MDC/tomcat/logs/stdout.log:

Oct 18, 2010 2:18:49 PM org.apache.catalina.core.ApplicationDispatcher invoke

SEVERE: Servlet.service() for servlet jsp threw exception

com.cisco.nm.xms.ctm.common.CTMException: URN_NOT_FOUND : urn "ogs_server_urn" : Not found !!

        at com.cisco.nm.xms.ctm.client.CTMCall.establishIPC(CTMCall.java:238)

        at com.cisco.nm.xms.ctm.client.CTMCall.<init>(CTMCall.java:218)

        at com.cisco.nm.xms.ctm.client.CTMClientProxy.<init>(CTMClientProxy.java:64)

        at com.cisco.nm.xms.ctm.client.CTMClientProxy.getProxy(CTMClientProxy.java:180)

        at com.cisco.nm.xms.ogs.client.OGSServerProxy.init(OGSServerProxy.java:179)

        at com.cisco.nm.xms.ogs.client.OGSServerProxy.init(OGSServerProxy.java:98)

        at com.cisco.nm.xms.ogs.client.OGSServerProxy.<init>(OGSServerProxy.java:85)

        at com.cisco.nm.xms.ogs.client.mgmt.OGSRuleExpressionUtil.getServerProxy(OGSRuleExpressionUtil.java:102)

        at com.cisco.nm.xms.ogs.client.mgmt.OGSRuleExpressionUtil.getClassDefUtil(OGSRuleExpressionUtil.java:123)

        at com.cisco.nm.xms.ogs.client.mgmt.OGSRuleExpressionUtil.getInternalClasses(OGSRuleExpressionUtil.java:153)

        at com.cisco.nm.xms.ogs.client.mgmt.OGSRuleExpressionUtil.getClasses(OGSRuleExpressionUtil.java:174)

        at org.apache.jsp.WEB_002dINF.screens.deviceselector.DeviceFilter_jsp._jspService(DeviceFilter_jsp.java:158)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)

        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:692)

        at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:594)

        at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:506)

        at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:966)

        at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:614)

        at com.cisco.nm.uii.taglib.framework.ContentAreaTag.doStartTag(ContentAreaTag.java:76)

        at org.apache.jsp.WEB_002dINF.screens.popup_jsp._jspx_meth_embu_contentarea_0(popup_jsp.java:876)

        at org.apache.jsp.WEB_002dINF.screens.popup_jsp._jspService(popup_jsp.java:693)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)

        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:692)

        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:470)

        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:405)

        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)

        at org.apache.struts.action.ActionServlet.processActionForward(ActionServlet.java:1758)

        at com.cisco.nm.uii.UIIController.processActionForward(UIIController.java:380)

        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1595)

        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:491)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)

        at com.cisco.nm.cmf.util.AccessLogFilter.doFilter(AccessLogFilter.java:128)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)

        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)

        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)

        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)

        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)

        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)

        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)

        at java.lang.Thread.run(Thread.java:595)

I did check the ctm_config.txt file in campus and cmapps directory, removed the ctmregistry and ctmregistry.backup files from /opt/CSCOpx/MDC/tomcat/webapps/campus/WEB-INF/lib, removed all files and directories below /opt/CSCOpx/MDC/tomcat/work/Standalone/localhost/ and restarted the services;
No change...

What I noticed when looking directly into CMF database is that "CampusOgsGroupCacheTable" is completely empty where as the following tables contain data:
    CampusOgsGroupPropertiesTable
    CampusOGSTagTable
    CampusUserGroupAssociationTable

What could be the reason for this behaviour?

I yet thought about the option to export all the groups from CMF (or at least the campus groups) and drop all tables from OGS and re-importing the groups but when I try to do this I get the following error:

root@lms-server # ./perl ./OGSCli.sh -u admin
Enter CiscoWorks password:
Authentication succeeded.
INFO: Enter EXIT to quit from OGSCli prompt any time.
Enter an OGSCli task <export | import> [export]:export
Enter a filename to export groups:./OGSGroupExport.info
WARN: The group details in the selected file will be overwritten with the new export groups information.Do you want to continue? (y/n) [n]:y
Enter a User-defined Group hierarchy name to export or All to export all User-defined Groups from all applications [All]:All
INFO: The hierarchies of the User-defined Groups in all applications that are installed in all servers, will be exported to ./OGSGroupExport.info.
Authorization succeeded.
ERROR: Could not export the specified User-defined Groups hierarchies to ./OGSGroupExport.info.
ERROR: This could be because of one or more of the  following reasons:
ERROR: * OGSServer process may not be running
ERROR: * Application or remote Grouping Server may not be reachable
ERROR: * Free disk space may be low
ERROR: * You do not have the required file permissions to create the file.
ERROR: See /var/adm/CSCOpx/log/CMFOGSClient.log for more details.
root@lms-server # ./perl ./OGSCli.sh -u admin -d
Enter CiscoWorks password:
Authentication failed.
Verify the username and password that you have entered.
root@lms-server #

just to be sure I upload the 2 ctm_config.txt files ....

[Joe Clarke]

The ctm_config files look okay.  Post the output of the pdshow command.  What happened on this server recently?  Was the hostname changed?

[Martin Ermel]

according to the customer, "nothing" happens to the server, i.e he is not aware of anything that could have interfere with this service. The hostname did not change in the past, the databases were imported from LMS 3.1 but were running and restarted several times without any problems (or perhaps better "without obvious problems.."); then the problem with ctm_config.txt arises (now for the 3rd time, even with DYNAMIC_PORT_ALLOCATION=0 being set after the 2nd)

The only thing I would bring in touch with this is that they are using the servlet technique to get UT data from custom UT reports.

I add the pdshow along with the output of netstat -aa

What about the issue with "CampusOgsGroupCacheTable" in the cmf database ? How or when is this table being populated? Is this just a subsequent issue because of the problems with CTM or Is there a possibility to force a trigger or task so that this table will be populated again?

I just checked the CampusOgsGroupCacheTable on another LMS system and found that this table is empty also ...- so it seems this is a normal state (even when the other GroupCacheTables in CMF are populated...)

[Martin Ermel]

I was wondering about the fact that even CampusOGSServer was running, CTM could not allocate the necessary ports (netstat -a).Because everything seems to be ok I commented out
    DYNAMIC_PORT_ALLOCATION=0
in ..../campus/WEB-INF/lib/ctm_config.txt

after stopping and starting CampusOGSServer (pdterm/pdexec) the device selector in UT was up and running again ...
I did a test with setting
    DYNAMIC_PORT_ALLOCATION=1

result: UT device selector was broken
It looks like this line (no matter if set to 0 or 1) seems to break "campus CTM" on this server...

It is running for now, but there are still 3 issues open and I do not know if they are directly associated

1)  OGSCli is still not working and finishes with this message:

root@lms-server # ls -al | grep -i ogs
-rwxrwxr-x   1 casuser  casusers    3288 Nov 20  2008 OGSCli.sh
-rwxrwxrwx   1 root     casusers       0 Oct 19 11:34 OGSGroupExport.info
root@lms-server # chown casuser:casusers OGSGroupExport.info
root@lms-server # ./OGSCli.sh -u admin
Enter CiscoWorks password:
Authentication succeeded.
INFO: Enter EXIT to quit from OGSCli prompt any time.
Enter an OGSCli task [export]:export
Enter a filename to export groups:OGSGroupExport.info
WARN: The group details in the selected file will be overwritten with the new export groups information.Do you want to continue? (y/n) [n]:y
Enter a User-defined Group hierarchy name to export or All to export all User-defined Groups from all applications [All]:All
INFO: The hierarchies of the User-defined Groups in all applications that are installed in all servers, will be exported to OGSGroupExport.info.
Authorization succeeded.
ERROR: Could not export the specified User-defined Groups hierarchies to OGSGroupExport.info.
ERROR: This could be because of one or more of the  following reasons:
ERROR: * OGSServer process may not be running
ERROR: * Application or remote Grouping Server may not be reachable
ERROR: * Free disk space may be low
ERROR: * You do not have the required file permissions to create the file.
ERROR: See /var/adm/CSCOpx/log/CMFOGSClient.log for more details.
root@lms-server #

the /var/adm/CSCOpx/log/CMFOGSClient.log is attached and there seems to be some certification issues; I recreated the certificate and in the GUI it shows to be set and be valid...

2) these FATAL messages appear in CampusOGSServer.log and I am not sure if they are harmless:

[ Thu May 20  16:16:34 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /Campus@lms-server/Campus@sdeu
1121/System Defined Groups/All Unreachable Devices
[ Thu May 20  16:20:10 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /Campus@lms-server
[ Thu May 20  16:22:24 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /Campus@lms-server/Campus@sdeu
1121/System Defined Groups/All Unreachable Devices
[ Thu May 20  16:24:23 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /RME@lms-server/Pre-deployed
[ Thu May 20  16:24:23 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /RME@lms-server/User Defined G
roups
[ Thu May 20  16:24:23 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /RME@lms-server/User Defined G
roups/All-CE-NM
[ Thu May 20  16:24:47 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups/Customizable Groups/Customizable Group 2
[ Thu May 20  16:24:50 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups/IN-Devices
[ Thu May 20  16:24:55 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups/Customizable Groups
[ Thu May 20  16:25:03 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups/Customizable Groups/Customizable Group 3
[ Thu May 20  16:29:01 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /RME@lms-server
[ Thu May 20  16:29:02 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /RME@lms-server/All Devices
[ Thu May 20  16:29:02 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /RME@lms-server/Normal Devices
[ Thu May 20  16:29:13 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups/Customizable Groups/Customizable Group 4
[ Thu May 20  16:29:14 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups/Customizable Groups/Customizable Group A
[ Thu May 20  16:29:34 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups/Customizable Groups/Customizable Group B
[ Thu May 20  16:29:35 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups/Customizable Groups/Customizable Group C
[ Thu MOGSGroups.infoay 20  16:30:34 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /RME@lms-server/User Defined G
roups/DE-Switche-IOS
[ Thu May 20  16:30:39 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server
[ Thu May 20  16:30:39 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups
[ Thu May 20  16:30:39 CEST 2010 ]FATAL  com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy  No top-level cache for: /DFM@lms-server/User Defined G
roups/Customizable Groups/Customizable Group 1

3) I found CampusOgsGroupCacheTable in cmf database to be empty on 2 different lms 3.2 installations (both solaris) but I cannot believe that this is ok due to some stored procedures pointing to this table to keep it on sync with CampusOgsGroupPropertiesTable

[Joe Clarke]

First, DYNAMIC_PORT_ALLOCATION=0 should disable the automatic modification of these files.  However, if the TCP port in the file is in use, CampusOGSServer will just not work, and you would get the URN Not Found errors.  So, you can try picking another random, high TCP port, and turn port allocation off.  See if that makes things more reliable.

As for OGSCli.sh, that script will not work due to CSCtd31272 if SSL is enabled.  Export groups from the Common Services group admin screen.

These errors aren't really that bad.  The OGS Server is still running, and it sounds like it's serving requests now that you have fixed the ctm_config file.

The cache tables are typically empty.  The real table is the GroupProperties table.

[Martin Ermel]

When dmgtd is stopped no port in the 5500x range is in use.

I did a test with ctm_config.txt and only one change:
1) DYNAMIC_PORT_ALLOCATION=0 is active as per recommendation ==>> results in CampusOGSServer starting but UT Device Selector is not working (netstat -a does not show any used port in the 5500x range)
2) DYNAMIC_PORT_ALLOCATION=0 is commented out like this #DYNAMIC_PORT_ALLOCATION=0 ==>> results in CampusOGSServer starting and UT Device Selector is working; also 55001 is allocated;

I get it working with this setting but we had problems 3 times in the past with this file so customer would like to have this fixed (if possible), because other teams rely on it.

I also tried to change SERVER_PORT=57000 (checked with netstat -a that there are no used ports in this range with and without dmgtd started). CampusOGSServer did start (from the ports it seems to be ok) but UT device selector had the "ogs_urn_server" error. So I was not able to change the default port successfully. I also tried to clear the browser cache before accessing LMS  ( what I did not do was deleting the tree under ..../Standalone/localhost ). This is the netstat output with SERVER_PORT=57000 and requesting the dev selector:
===========
root@lms-server # netstat -a | egrep '5500|5700'
lms-server.45500       lms-server.45007       49152      0 49152      0 ESTABLISHED
lms-server.45007       lms-server.45500       49152      0 49152      0 ESTABLISHED
*.57000              *.*                0      0 49152      0 LISTEN
localhost.57000      localhost.48197      49152      0 49152      0 TIME_WAIT
localhost.57000      localhost.48216      49152      0 49152      0 TIME_WAIT
===========

for OGSCLI.sh:

The server was integrated with ACS but SSL for LMS is not enabled on that server and also the FQDN is in /etc/hosts.

But, I don' t care about that, as I succesfully used the GUI - thanks.

Thanks, so I will suppress the existence of those errors in my mind ;-)

Ok, I assume the cache tables exists for performance reasons and CampusOGSServer is not that slow on this server, so it is ok. But this seems to be the only cache table not being populated. And if I really read the fatal errors I could think they are related to this fact of an empty cache table (com.cisco.nm.xms.ogs.server.GroupCacheImpl getHierarchy) ...

[Joe Clarke]

When you changed the port, did you update both ctm_config.txt files?  They both need to point to the same port.  If you put back DYNAMIC_PORT_ALLOCATION=0, and reproduce the error, post the outputs of "pfiles PID", where PID is the PID of CampusOGSServer and ANIServer.

[Martin Ermel]

I did the test and the output is attached; the explicit lines you requested are [1016 - 1279];

I collected also info about the working scenario, roughly these steps, but comments are inline of the attached file

=================

LMS is started, port 55000
NO ERROR with UT Device Selector
pfiles
netsat -a |egrep -i '5500|5700'

pdterm CampusOGSServer, changing ctm_config.txt (57000)
pdexec CampusOGSServer >> Error with UT Device Selector
pfiles
stopping LMS, starting LMS >> ERROR with UT Device Selector
pfiles
netstat

pdterm CampusOGSServer, changing ctm_config.txt (to 55000)
pdexec CampusOGSServer
pfiles
netstat
>>ERROR with UT Device Selectot

stopping LMS, starting LMS
netstat
pfiles
NO ERROR with UT Device Selector
pfiles
netsat

======

With the customer, I yet discussed a reinstallation of LMS on that server without importing the old databases; - if there is no obvious failure here we still can stop troublshooting and do the reinstall....

[Joe Clarke · Accepted solution]

Instead of setting DYNAMIC_PORT_ALLOCATION=0, change the perms on the two ctm_config.txt files to 0444, use the original port, and see if UT works.  If so, run in that configuration for a while to see if corruption returns.  It shouldn't since casuser will not be able to write to the file.

[Martin Ermel]

finally customer decided to reinstall this server and I wasn't disposed to detain him ... :-)

I will keep the 0444 in mind in case of...

thank you joe for your help!