Problem Statement
NSO can engage out of memory issue from time to time. If an issue is reproduced and there isn't a observability tool in the environment, it will be a hassle to get a higher level overview on what has been happened before and during the issue. In this case, we introduce "NSO Memory Utilization Measurement Tool" to help the customer and Cisco internal teams to collect the memory consumption(allocation and used) on all NSO related process. These process include NSO Core, Java VM and Python VM. For Python VM, the "NSO Memory Utilization Measurement Tool" will first collect per VM memory consumption and than total memory consumption across all the VMs.
Feature Introduction
With single command
sh plot.sh <Time Consumption>this tool will collect 4 type of data as shown below for NSO Core, Java VM and Python VM.
- Allocated memory from "writeable/private" from the "pmap" command. This gives us the actually allocated memory for each process without shared memory. (Suggested by @Magnus Thoang from BU Architect Support Team).
- The Physical Memory Used is from "VmRSS" in "/proc/$pid/status"
- "CommitLimit" from "/proc/meminfo" as memory allocation limit.
- "Committed_AS" from "/proc/meminfo" for global allocated memory across the system.
Eventually generate Diagrams for each of them like the one shown below
The diagram above we simulate a scenario of NSO OOM during the booting phase. The allocated memory exceed the limit which cause NSO to crash due to OOM. With this tool, this can be easily indicate at which moment the NSO crash. Since the Tool will create similar diagram for all NSO related process. By cross comparing them will provide a clue on what process or which package might cause the issue. This tool is based on simple Bash for Data Collection and Processing and Gnuplot for Data Visualization.
Diagram Generated
All the diagram generated have a red warning line to indicate where is the CommitLimit except per VM PythonVM measurment.
- Allocated Memory per Process vs Total Allocated Memory(Commited_AS) vs Physical Memory Usage(RSS)
- NSO Core(ncs.smp)
- JavaVM
- PythonVM
- per VM (Without CommitLimit Warning Line to increase visiblity)
- Total for all the VMs
- Comparision between ncs.smp, JavaVM and PythonVM
- Allocated Memory per Process vs Total Allocated Memory(Commited_AS)
- Total Allocated Memory(Commited_AS) vs Physical Memory Usage(RSS)
Tool Repository
2 Comments