MSE Location Analytics stopped collecting data

jpop00001 · ‎10-15-2013

I'm using MSE 7.4 and using Location analytics. I got 5 days of data, and now its no longer collecting data.

When i go to the location analytics GUI the only day i can select is almost 2 weeks ago. When I run reports directly from the MSE, those work fine.

Any idea on where I can start looking?

tosormun · ‎10-15-2013

What are the contents of the /opt/mse/logs/analytics/mse.log.o file? What is displayed when typing df -k (checking the disk space for MSE?

jpop00001 · ‎10-17-2013

Ah finally posting! Here's my error log.

10/15/13 18:39:28.277 ERROR[location] [18] AesNmspServiceMessageHandler.processImpl(target,request,seqNum,locpMessage) 10.159.0.50 ServiceSubscribeRequest 0 ServiceSubscribeResponse

10/15/13 18:39:28.277 ERROR[location] [18] AesNmspServiceMessageHandler.processImpl Received failure for subscription request

10/15/13 18:39:32.624 ERROR[core-engine] [25] Error in getting licensing limits. Attempt: 1 com.aes.server.mse.MSEClientException: getLicensingCapabilities: Licensing is not available, please try again later

10/16/13 08:22:00.413 ERROR[config] [52] AesLogJULImpl- syslog method : Exception while trying to log to external SyslogServer.

10/16/13 08:22:00.413 ERROR[config] [52] java.io.IOException: Message too long

tosormun · ‎10-17-2013

At the end of the /opt/mse/logs/analytics/mse.log file (whichever is the newest and actually I believe that the path to this file in 7.4 may be /opt/mse/analytics/intellify/tools/MSEclient) it should look something like what is listed below. If there is something else, then the analytics process has stopped, you may wish to manually stop/start the mse process /etc/init.d/msed restart and/or reboot, the mse and analytics processes will restart as the box comes back up. Remember the MSE process and the Analytics process are seperate, the MSE process works by itself with no dependency on the Analytics process however if the MSE process is down the Analytics process cannot work.

Also what does getserverinfo display, I see some mention of exceeding license limits (the results of getserverinfo with tell us what those are. Also is it possible to upgrade to 1.4 Prime/7.5 MSE, the images and instructuctions can be found at cisco.com web site.

PROCESS MANAGER STARTING READDATA

--------------- ATOM STARTING ZoneReaderProcess : starting

--------------- ATOM COMPLETED IN 0 ms ZoneReaderProcess : 4 zones read.

PROCESS MANAGER COMPLETED READDATA

Starting write

PROCESS MANAGER STARTING STORE_DATA_2/2

--------------- ATOM STARTING PathWriterProcess : starting

--------------- ATOM COMPLETED IN 20 ms PathWriterProcess : Saved 2 paths (rejected 0 duplicates).

PROCESS MANAGER COMPLETED STORE_DATA_2/2

Stored 2 paths for 2 devices. Total of 2 points (avg: 1.00 points/path)

Clearing device cache

Previous point count: 224679, points added since last stats gathered: 42000

Finished

Pruning database...

Max points: 2000000

Reset points: 1900000

Pruned 0 points

Finished at 2013-10-17T08:42:50.321-05:00 ============================================================

jpop00001 · ‎10-17-2013

I have to be honest I'm not the Main Cisco guy so I'm not quite sure I know exactly how to get some of these, I've just been tasked with trying to get Analytics working. I have access to the cisco prime infrastructure and the logs I get are from there.

The license issue is because we're under licensed for the number of elements that are being collected. We peak at around 3000 elements but are only licensed for 500 currently. We're trying to prove out the accuracy and the validity of the data before we purchase more..

I'm not sure how to run the console commands, can i do that from the MSE? The logs are what I get when I use the download logs button under Services > Mobility Services Engines > cmse > system > logs

tosormun · ‎10-17-2013

To grab the logs I refer to you will need to have cli access to the MSE. This is typically accomplished by using an ssh session ssh -l root ipaddress or if the MSE is an appliance you may connect via the console port on the box. Once at the cli for the mse type getserverinfo, this will display a bunch of stuff that is useful in understanding what the state of the mse is. Also the retrieve the contents of the files I mention, from the cli on the MSE navigate to the directory and either using vi to grab the file contents or type cat filename (like this cat mse.log.0), this will display the entire contents of the log file which can be copied and pasted into some text edit progam. Remeber for this file we only care about what is at the bottom couple lines.

I would recommend moving to the latest code release (7.5 MSE/1.4 Prime) as there are many improvements over 7.4. Information, release notes, additional troubleshooting information, etc. may be found here ---

http://www.cisco.com/en/US/products/ps9742/index.html MSE

http://www.cisco.com/en/US/products/ps12239/index.html Prime

jpop00001 · ‎10-17-2013

why do my comments sometime require approval by moderator and sometimes not?

jpop00001 · ‎10-17-2013

Here's the bottom of the mse.log file:

PROCESS MANAGER STARTING STORE_DATA_3/3

--------------- ATOM STARTING PathWriterProcess : starting

java.sql.SQLException: ORA-01654: unable to extend index ANALYTICS.PATH_PK by 8 in tablespace ANALYTICS

ORA-06512: at "ANALYTICS.PATH_PACKAGE", line 9

ORA-06512: at line 1

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:439)

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:395)

at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:802)

at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:436)

at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)

at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)

at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:202)

at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1005)

at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1307)

at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3449)

at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3550)

at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4710)

at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1374)

at ie.thinksmarttechnologies.intellify.locationintelligence.db.processes.PathWriterProcess.insertPaths(PathWriterProcess.java:511)

at ie.thinksmarttechnologies.intellify.locationintelligence.db.processes.PathWriterProcess.executeLinkedAtom(PathWriterProcess.java:2091)

at ie.thinksmarttechnologies.intellify.chaining.process.LinkedProcess.executeAtom(LinkedProcess.java:44)

at ie.thinksmarttechnologies.intellify.chaining.process.AbstractProcessAtom.execute(AbstractProcessAtom.java:21)

at ie.thinksmarttechnologies.intellify.chaining.process.ProcessManager.executeAtom(ProcessManager.java:116)

at ie.thinksmarttechnologies.intellify.chaining.process.ProcessManager.execute(ProcessManager.java:105)

at ie.thinksmarttechnologies.com.mseconnector.mseapi.Scraper.sendPathsToOracle(Scraper.java:650)

at ie.thinksmarttechnologies.com.mseconnector.mseapi.Scraper.run(Scraper.java:258)

at ie.thinksmarttechnologies.com.mseconnector.Main.run(Main.java:184)

at ie.thinksmarttechnologies.com.mseconnector.Main.main(Main.java:484)

PROCESS MANAGER FAIL STORE_DATA_3/3 : PathWriterProcess : cause = ORA-01654: unable to extend index ANALYTICS.PATH_PK by 8 in tablespace ANALYTICS

ORA-06512: at "ANALYTICS.PATH_PACKAGE", line 9

ORA-06512: at line 1

jpop00001 · ‎10-17-2013

Here is getserverinfo with the IPs replaced. I've got to do this with multiple posts. it wont let me add the whole thing at once

Health Monitor is running

Starting MSE Platform, Waiting to check the status.

MSE services are up, getting the status

-------------

Server Config

-------------

Product name: Cisco Mobility Service Engine

Version: 7.4.110.0

Health Monitor Ip Address: 1.1.1.1

High Availability Role: 1

Hw Version: V01

Hw Product Identifier: AIR-MSE-VA-K9

Hw Serial Number: xxxxxx_62909fa0-1f05-11e3-80c0-005056837292

Use HTTP: false

Legacy HTTPS: false

Legacy Port: 8001

Log Modules: 24516

Log Level: INFO

Days to keep events: 2

Session timeout in mins: 30

DB backup in days: 2

jpop00001 · ‎10-17-2013

-------------

Services

-------------

Service Name: Context Aware Service

Service Version: 7.4.0.45

Admin Status: Enabled

Operation Status: Up

Service Name: WIPS

Service Version: 1.0.4041.0

Admin Status: Disabled

Operation Status: Down

Service Name: Mobile Concierge Service

Service Version: 2.0.0.37

Admin Status: Disabled

Operation Status: Down

Service Name: Location Analytics Service

Service Version: 1.0.0.12

Admin Status: Enabled

Operation Status: Up

jpop00001 · ‎10-17-2013

Server Monitor

--------------

Server start time: Tue Oct 15 18:39:25 EDT 2013

Server current time: Thu Oct 17 11:33:33 EDT 2013

Server timezone: US/Michigan

Server timezone offset: -18000000

Restarts: 6

Used Memory (bytes): 181712032

Allocated Memory (bytes): 532873216

Max Memory (bytes): 954466304

DB virtual memory (kbytes): 0

DB virtual memory limit (bytes): 0

DB disk memory (bytes): 33657628288

DB free size (kbytes): 0

jpop00001 · ‎10-17-2013

Could this be the issue under Server monitor? DB free size (kbytes): 0

jpop00001 · ‎10-17-2013

Server Monitor

--------------

Server start time: Tue Oct 15 18:39:25 EDT 2013

Server current time: Thu Oct 17 11:33:33 EDT 2013

Server timezone: US/Michigan

Server timezone offset: -18000000

Restarts: 6

Used Memory (bytes): 181712032

Allocated Memory (bytes): 532873216

Max Memory (bytes): 954466304

DB virtual memory (kbytes): 0

DB virtual memory limit (bytes): 0

DB disk memory (bytes): 33657628288

DB free size= 0 <---***is this the problem? No free DB space?***

jpop00001 · ‎10-17-2013

Server Monitor

--------------

Server start time: Tue Oct 15 18:39:25 EDT 2013

Server current time: Thu Oct 17 11:33:33 EDT 2013

Server timezone: US

Server timezone offset: -18000000

Restarts: 6

Used Memory: 181712032

Allocated Memory : 532873216

Max Memory : 954466304

DB virtual memory : 0

DB virtual memory limit: 0

DB disk memory : 33657628288

DB free size: 0 <---***is this the problem? No free DB space?***

jpop00001 · ‎10-17-2013

Server Monitor

--------------

Server start time: Tue Oct 15 18:39:25 EDT 2013

Server current time: Thu Oct 17 11:33:33 EDT 2013

Server timezone: US/Michigan

Server timezone offset: -18000000

Restarts: 6

Used Memory (bytes): 181712032

Allocated Memory (bytes): 532873216

Max Memory (bytes): 954466304

DB virtual memory (kbytes): 0

DB virtual memory limit (bytes): 0

DB disk memory (bytes): 33657628288

DB free size (kbytes): 0 <---***is this the problem? No free DB space?***