cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

How to Troubleshoot High CPU/Memory utilization issue for sessmgr

1099
Views
5
Helpful
0
Comments

Troubleshooting for Sessmgr WARN state due to over Memory Usage

 

Go to hidden mode and run this commands for sessmgr instance which is in WARN or OVER state

 

show messenger proclet facility sessmgr instance <> heap depth 9

show messenger proclet facility sessmgr instance <> heap verbose

show messenger proclet facility sessmgr instance <> heap

show messenger proclet facility sessmgr instance <> system heap verbose

show messenger proclet facility sessmgr instance <> system heap depth 9

show snx sessmgr instance <> memory mblks

show snx sessmgr instance <> memory ldbuf

show snx sessmgr instance <> memory sdbuf

show session subsystem facility sessmgr instance <instance-value> debug-info verbose

task core facility sessmgr instance <> (upload the corefile to SR)

 

or

 

you can also chose to follow below action plan:

 

1. Collect SSD before restarting the SESSMGR instance in warn state.

 

2. Collect the core dump for any of the affected sessmgr using "task core facility sessmgr instance <instance-value>" . 

3. Collect the output of the following commands in hidden mode,         

  - show messenger proclet facility sessmgr instance <instance-value> heap depth 9         

- show messenger proclet facility sessmgr instance <instance-value> system heap depth 9         

- show messenger proclet facility sessmgr instance <instance-value> heap       

   - show messenger proclet facility sessmgr instance <instance-value> system         

  - show snx sessmgr instance <instance-value> memory ldbuf         

- show snx sessmgr instance <instance-value> memory mblk         

- show session subsystem facility sessmgr instance <instance-value> debug-info verbose         

- show task resources facility sessmgr instance <instance-value>        

-  show messenger proclet facility sessmgr instance <instance-value> graphs heap

 4. After this restart the sessmgr task using "task kill facility sessmgr instance <instance-value>" 

5. Then execute the following CLI command "show task resources facility sessmgr instance <instance-value> and make sure it is back in "good" state.

6. Collect another SSD.

7. Collect the output of all CLI commands mentioned in Step#3. 

8. Collect the core dump for any of the healthy SESSMGR instance using the command mentioned in Step#2.

 9. Wait 15 minutes and collect another set of SSD 

10. After SSD get again output of all CLI commands mentioned in Step#3.

——-

Troubleshooting for sessmgr WARN state to high CPU

 

show profile facility sessmgr active depth 1 head 26

show profile facility sessmgr active depth 4 head 101

show profile facility sessmgr active depth 8 head 201

-Show task info facility sessmgr instance <x>

-Task snap facility sessmgr instance x (multiple times) ( This command generates below outputs and we can profile output)

 

Nonfatal Signal 10: User defined signal 1

  PC: [075c825d/X] acsmgr_match_rule()

  Note: User-initiated state dump.

  Signal from: sitmain pid=3625 uid=0

  Process: card=4 cpu=0 arch=X pid=4320 cpu=~100% argv0=sessmgr

  Crash time: 2015-MAY-07+10:53:17 UTC

  Recent errno: 11 Resource temporarily unavailable

  Stack (39928@0xffff4000):

    [075c825d/X] acsmgr_match_rule() sp=0xffff45c8

    [075c2321/X] acsmgr_trm_match_rule() sp=0xffff45f8

    [07ad7eaf/X] _do_acsmgr_process_packet() sp=0xffff58a8

    [07b03349/X] acsmgr_process_packet() sp=0xffff5948

    [07abd6c9/X] acsmgr_analyze_packet() sp=0xffff5b68

    [0761fc40/X] acsmgr_data_receive_merge_mode() sp=0xffff60a8

    [07a60bb6/X] acs_handle_datapath_events_from_sm_interface() sp=0xffff60f8

    [0343f62d/X] sessmgr_sef_initiate_data_packet_ind() sp=0xffff6168

    [03420fd9/X] sessmgr_pcc_intf_send_data_packet_ind() sp=0xffff61a8

    [0598fb88/X] sessmgr_pcc_fwd_packet() sp=0xffff61f8

    [05973b8d/X] sessmgr_ipv4_process_user_pkt() sp=0xffffad88

    [03c0591f/X] sessmgr_pgw_handle_inbound_data_packet() sp=0xffffae28

    [058ce0c2/X] sessmgr_egtpu_receive_gtpu_packet() sp=0xffffb3e8

    [058d499a/X] sessmgr_med_data_receive() sp=0xffffd618

    [0a203a77/X] sn_loop_run() sp=0xffffdba8

    [0a02b3f4/X] main() sp=0xffffdbe8

  …

  Profile depth=1:

     10.8%        111  actsm_search()

     10.1%        103  acsmgr_match_rule()

      4.2%         43  acs_control_charge()

      3.4%         35  sessmgr_egtpu_dispatch_routine()

      3.3%         34  libc.so.6/poll()

      3.2%         33  sn_slist_lookup_by_key()

      2.3%         24  acsmgr_rule_compare_ip_addr()

      2.1%         22  libc.so.6/memcpy()

      2.1%         21  _do_acsmgr_process_packet()

      2.1%         21  acsmgr_dcca_charge()

 

 

 

When we do explicit profiling, the debug profiler takes us to an interactive screen, which accepts keystrokes like UP/DOWN to dynamically increase/decrease sample timer values and dynamically adjust the depth of the function-stack that will be used for sorting the samples (number keys 1-8). In addition it dynamically displays the current sorted list of samples that could fit the screen. We get to the see the total number of samples collected so far and the list of top occurrences, again both the absolute number of times and as a percentage to the total samples. When we exit the debug profiler's interactive output, it presents a complete summary of all samples seen.

 

debug profiler facility sessmgr instance <>