Introduction
It’s always a good practice to collect important and relevant information and logs from the system, whenever some issue is reported.
This information will help to understand about the CPS logs and information which is always required to debug the issue.
Please collect the following details around the time when issue occur to analyse the issue:
Initial Analysis
1. The output for about.sh. This will tell us the version being used and other details like the Policy Builder and Control Center links.
2. The output for diagnostics.sh will give you basic system health check outputs .
E.g : diagnostic.sh -q will give only the failed process status. diagnostics.sh will look similar as shown below. In this below example it illustrates that QNS service is not running and swap memory usage is more on pcrfclient01
Validating hostnames...[Pass]
Checking basic TCP ports (80, 11211, 7070, 8080, 27017, 9091, 9092)...[Pass]
Checking QNS UDP (RADIUS) ports (1812, 1813)... Could not connect to port 1812 on qns01 (Radius access requests)...[FAIL]
Could not connect to port 1813 on qns01 (Radius accounting requests)...[FAIL]
Checking qns passwordless logins on all boxes...[Pass]
Checking swap space... Checking swap memory usage on pcrfclient01...[FAIL]
Swap usage is 2833 MB. This will likely lead to a slowdown in your system! Please ensure your memory is provisioned properly.
If systems memory usage is no longer high, you can reset swap with:
/opt/broadhop/installer/diag/support/swap2ram.sh
Checking disk usage...[PASS]
Running advanced diagnostics on qns01:9045...[FAIL]
[Warn]: Using POC/Development license (100 session limit).
To use a license file, remove -Dcom.broadhop.developer.mode from /etc/broadhop/qns.conf Checking CPU and memory allocation...
Checking memory on pcrfclient01...[Warn]
Memory on pcrfclient01 is currently 3948 MB (~4168 with VMWare overhead). Recommended amount is 8192 MB (minimum 4096 MB).
Checking CPU Count on pcrfclient01...[Warn]
CPU Count on pcrfclient01 is currently 2. It is recommended to be 4 (minimum 2)
3. The output for list_installed_features command. This command will list only the features which are currently active in QPS system.
4. Consolidated-engine logs. These are present on pcrfclient VM in the path /var/log/broadhop .This is the QPS transaction log.In case of a GR setup, this information should be collected from the Active site(s)..
5. Consolidated-qns logs. These are present on pcrfclient VM in the path /var/log/broadhop. This is kind of QNS debug Logs.
6. The SVN dump(PB Config) being used. Contents of: /etc/broadhop. This will provide us the parameters set in qns.conf and the existing log levels that are set. In case of a GR setup, this information needs to be collected from the Active site(s).
Database
- The output for top_qps.sh. This will tell us about the transactions being made and their time etc.
- For DB related issues: we’ll need to ssh to all sessionmgr VMs and get the mongodb-*.log and mongodb-*.log.gz. These are present on the sessionmgr VMs at the location /var/log.
- The mongostat and mongotop output. The command to collect these outputs is:
mongostat --host <primary sessionmgr hostname>:<port>
mongotop --host <primary sessionmgr hostname>:<port>
4. The output for rs.status() and rs.conf(). This will give us the status of the replica sets. These can be collected by logging into the sessionmgr VMs, ssh to any sessionmgr and give the above commands.
Memory
- Heapdump in case of memory or VM not responding. The jvm.conf needs to have XX:+HeapDumpOnOutOfMemoryError. jvm.conf is present in the location /etc/broadhop/pcrf
- HPROF files would need to be zipped and collected from /tmp/*.hprof location. Also these can be as huge as the memory configured on the QNS VM so make sure to have adequate disk space.
Servers & Packet Captures
netstat and tcpdump outputs for connectivity related issues.
tcpdump -i <any port> <p#>-s 0 -vvv -w /tmp/tcpdump.pcap;
netstat -anp | grep <port>;
Firewall
Iptables output. Needed for networking or firewall related issues.
ssh to any sessionmgr.
give the command: iptables –L
Garbage Collection
Service-qns*.log, service-qns*.log.gz and qns logs. These logs are required to see the activity of a particular node if it is causing any issue and also if the issue is related to GC then it will contain the GC and Full GC log statements as well. These are present on the LB and QNS VMs at the location
/var/log/broadhop
Grafana graphs for CPU usage, GC etc is also useful to detect issue.
