The Basics of Working with Logs
Take a look at the logs. Generally, I start with the ipics.log, but there are several important logs in the system.
SSH into the server as root
(Since 2.2 or maybe 4.0) In root's home directory, there is a folder called logs (/root/logs). It contains symlinks to the most common logs we use. The most common are found at the top of the folder, while the less common ones are found in the sub-folders below. This makes it really easy to watch the system in near real-time with the command:
tail -F ~/logs/*
To get a general idea what when wrong, let's look for unique error messages. The following command will distill a 7000 line log file down to a short list of error messages with the option to further filter the results using egrep:
grep ERROR ipics.log | egrep -i -v "(ignoreLinesWithThis|OrThis)" | cut -d" " -f4- | sort -u
Then use grep to look for interesting keywords you got from the first query to understand the context of the error and if it is worth investigating. The following command will show you all occurrences of "InterestingKeyword" plus 5 lines of context above and below the matching line:
grep -C5 "InterestingKeyword" ipics.log
There is a lot of noise in the logs, so it is nice to compare with another server to see what is "normal".
If you don't see anything related to the symptom you are debugging, expand your search to the other logs:
o ~/logs/catalina.out - This is where your thread dumps go, as well as general tomcat (usually non-IPICS) errors show up.
o ~/logs/stdout.log and ~/logs/stderr.log - Sometimes you find stuff in here, but since it isn't timestamped, it is difficult to correlate with the other logs.
o ~/logs/edr.log - If your problem is related to HA setup or tear-down.
o Then start looking at subsystem logs like nm.log, rcs.log, dms.log, engine.out, and the MIVR logs.
o ~/logs/os/latest/ - The install logs
o Use get_all_logs to take a snapshot of all the system logs before they roll-over. If you are on an HA system, remember to run it on the secondary too.
·Be careful not to modify the ipics.log while the server is running. Always copy the log to /tmp before you make any changes or filter-out lines. To prevent accidental changes, open logs with "vi -R ipics.log"
Where to begin?
·There are a number of useful diagnostic tools you can use, such as:
o versions - Dumps the version data for the OS, WAR, and IPICS RPM's
o service ipics status Overall system status
o service ipics ha-status HA status
o top-ipics - Good for watching performance of IPICS related processes
o /opt/cisco/ipics/security/checksec - Gives SSH/SSL trust status and diagnoses some common problems.
o /opt/cisco/ippe/bin/systat - Gives overall system status and diagnoses some common problems.
