Troubleshooting for Sessmgr WARN state due to over Memory Usage
Go to hidden mode and run this commands for sessmgr instance which is in WARN or OVER state
show messenger proclet facility sessmgr instance <> heap depth 9
show messenger proclet facility sessmgr instance <> heap verbose
show messenger proclet facility sessmgr instance <> heap
show messenger proclet facility sessmgr instance <> system heap verbose
show messenger proclet facility sessmgr instance <> system heap depth 9
show snx sessmgr instance <> memory mblks
show snx sessmgr instance <> memory ldbuf
show snx sessmgr instance <> memory sdbuf
show session subsystem facility sessmgr instance <instance-value> debug-info verbose
task core facility sessmgr instance <> (upload the corefile to SR)
or
you can also chose to follow below action plan:
1. Collect SSD before restarting the SESSMGR instance in warn state.
2. Collect the core dump for any of the affected sessmgr using "task core facility sessmgr instance <instance-value>" .
3. Collect the output of the following commands in hidden mode,
- show messenger proclet facility sessmgr instance <instance-value> heap depth 9
- show messenger proclet facility sessmgr instance <instance-value> system heap depth 9
- show messenger proclet facility sessmgr instance <instance-value> heap
- show messenger proclet facility sessmgr instance <instance-value> system
- show snx sessmgr instance <instance-value> memory ldbuf
- show snx sessmgr instance <instance-value> memory mblk
- show session subsystem facility sessmgr instance <instance-value> debug-info verbose
- show task resources facility sessmgr instance <instance-value>
- show messenger proclet facility sessmgr instance <instance-value> graphs heap
4. After this restart the sessmgr task using "task kill facility sessmgr instance <instance-value>"
5. Then execute the following CLI command "show task resources facility sessmgr instance <instance-value> and make sure it is back in "good" state.
6. Collect another SSD.
7. Collect the output of all CLI commands mentioned in Step#3.
8. Collect the core dump for any of the healthy SESSMGR instance using the command mentioned in Step#2.
9. Wait 15 minutes and collect another set of SSD
10. After SSD get again output of all CLI commands mentioned in Step#3.
——-
Troubleshooting for sessmgr WARN state to high CPU
show profile facility sessmgr active depth 1 head 26
show profile facility sessmgr active depth 4 head 101
show profile facility sessmgr active depth 8 head 201
-Show task info facility sessmgr instance <x>
-Task snap facility sessmgr instance x (multiple times) ( This command generates below outputs and we can profile output)
Nonfatal Signal 10: User defined signal 1
PC: [075c825d/X] acsmgr_match_rule()
Note: User-initiated state dump.
Signal from: sitmain pid=3625 uid=0
Process: card=4 cpu=0 arch=X pid=4320 cpu=~100% argv0=sessmgr
Crash time: 2015-MAY-07+10:53:17 UTC
Recent errno: 11 Resource temporarily unavailable
Stack (39928@0xffff4000):
[075c825d/X] acsmgr_match_rule() sp=0xffff45c8
[075c2321/X] acsmgr_trm_match_rule() sp=0xffff45f8
[07ad7eaf/X] _do_acsmgr_process_packet() sp=0xffff58a8
[07b03349/X] acsmgr_process_packet() sp=0xffff5948
[07abd6c9/X] acsmgr_analyze_packet() sp=0xffff5b68
[0761fc40/X] acsmgr_data_receive_merge_mode() sp=0xffff60a8
[07a60bb6/X] acs_handle_datapath_events_from_sm_interface() sp=0xffff60f8
[0343f62d/X] sessmgr_sef_initiate_data_packet_ind() sp=0xffff6168
[03420fd9/X] sessmgr_pcc_intf_send_data_packet_ind() sp=0xffff61a8
[0598fb88/X] sessmgr_pcc_fwd_packet() sp=0xffff61f8
[05973b8d/X] sessmgr_ipv4_process_user_pkt() sp=0xffffad88
[03c0591f/X] sessmgr_pgw_handle_inbound_data_packet() sp=0xffffae28
[058ce0c2/X] sessmgr_egtpu_receive_gtpu_packet() sp=0xffffb3e8
[058d499a/X] sessmgr_med_data_receive() sp=0xffffd618
[0a203a77/X] sn_loop_run() sp=0xffffdba8
[0a02b3f4/X] main() sp=0xffffdbe8
…
Profile depth=1:
10.8% 111 actsm_search()
10.1% 103 acsmgr_match_rule()
4.2% 43 acs_control_charge()
3.4% 35 sessmgr_egtpu_dispatch_routine()
3.3% 34 libc.so.6/poll()
3.2% 33 sn_slist_lookup_by_key()
2.3% 24 acsmgr_rule_compare_ip_addr()
2.1% 22 libc.so.6/memcpy()
2.1% 21 _do_acsmgr_process_packet()
2.1% 21 acsmgr_dcca_charge()
When we do explicit profiling, the debug profiler takes us to an interactive screen, which accepts keystrokes like UP/DOWN to dynamically increase/decrease sample timer values and dynamically adjust the depth of the function-stack that will be used for sorting the samples (number keys 1-8). In addition it dynamically displays the current sorted list of samples that could fit the screen. We get to the see the total number of samples collected so far and the list of top occurrences, again both the absolute number of times and as a percentage to the total samples. When we exit the debug profiler's interactive output, it presents a complete summary of all samples seen.
debug profiler facility sessmgr instance <>
