On several 4.x releases we

witani · ‎02-24-2016

Dears,

We have using an ASR9K for 3 years. Commit and update used to take 1 or 2 sec. Suddenly it started to take 20 sec to finish.

CPU is around 9 %. Any idea how to approach this problem ?

Regards

Eddie Chami · ‎02-24-2016

Dear its probably due to a leak, or a process blocked, or some busted verifier. What release is this on? TO check these you normally start with show log, you'll see noise there, followed by show process blocked, then show proc mem..

if your running old release you may want to upgrade.

If you want to stay on your current release, start looking at the above to give you some leads.

Regards

Eddie.

witani · ‎02-24-2016

Thank you Eddy.

My version i Cisco IOS XR Software, Version 4.2.1[Default]

I see processes in reply state for a long time, is that what you are talking about ?

RP/0/RSP0/CPU0:cvas-2#sh processes blocked
Thu Feb 25 09:27:08.311 EET
Jid       Pid Tid            Name State   TimeInState    Blocked-on
65549     16397   1             ksh Reply 31175:43:10:0070   16395 devc-conaux
65554     65554   2      devb-umass Reply    0:00:00:0120   53284 io-usb
   97     49187   2      umass-enum Reply 31175:43:13:0250       1 kernel
   97     49187   6      umass-enum Reply 31175:43:07:0854   53284 io-usb
   97     49187   7      umass-enum Reply 31175:43:10:0372       1 kernel
   97     49187   8      umass-enum Reply 31175:43:09:0191       1 kernel
   97     49187   9      umass-enum Reply 31175:43:07:0855       1 kernel
   52     65574   2         attachd Reply 31175:43:11:0191   57381 eth_server
   52     65574   3         attachd Reply 31175:43:11:0189   24595 mqueue
   88     65580   6            qnet Reply    0:00:00:0037   57381 eth_server
   51     65584   2   attach_server Reply 31175:43:11:0000   24595 mqueue
65588     65588   2      devb-umass Reply    0:00:02:0553   53284 io-usb
65593     77881   2      devb-umass Reply    0:00:50:0963   53284 io-usb
435    233568   1     tftp_server Reply 13748:49:31:0186   24595 mqueue
317    295142   2         lpts_fm Reply    0:00:00:0074 282764 lpts_pa
65868 128377164   1            more Reply    0:00:00:0063   24593 pipe
65882 128377178   1 show_processes Reply    0:00:00:0000       1 kernel
65892 128323940   1            exec Reply    0:00:00:0161       1 kernel

Eddie Chami · ‎02-25-2016

Nothing there looks odd, you probably want to look at this during a commit when its hanging, or syslog/memory leak. You have a pretty old release here, you should upgrade.

witani · ‎02-27-2016

Hi Eddie,

As you suggested, I looked at blocked processes during commit hanging. Four extra processes appeared. Appreciate if you have a look.

Regards,

Output before commit command:

Jid Pid Tid Name State TimeInState Blocked-on

65549 16397 1 ksh Reply 31223:55:29:0824 16395 devc-conaux

65554 65554 2 devb-umass Reply 0:00:33:0568 53284 io-usb

97 49187 2 umass-enum Reply 31223:55:33:0004 1 kernel

97 49187 6 umass-enum Reply 31223:55:27:0608 53284 io-usb

97 49187 7 umass-enum Reply 31223:55:30:0126 1 kernel

97 49187 8 umass-enum Reply 31223:55:28:0945 1 kernel

97 49187 9 umass-enum Reply 31223:55:27:0609 1 kernel

52 65574 2 attachd Reply 31223:55:30:0945 57381 eth_server

52 65574 3 attachd Reply 31223:55:30:0943 24595 mqueue

88 65580 6 qnet Reply 0:00:00:0048 57381 eth_server

51 65584 2 attach_server Reply 31223:55:30:0755 24595 mqueue

65588 65588 2 devb-umass Reply 0:00:03:0573 53284 io-usb

65593 77881 2 devb-umass Reply 0:00:13:0939 53284 io-usb

435 233568 1 tftp_server Reply 13797:01:50:0940 24595 mqueue

317 295142 2 lpts_fm Reply 0:00:00:0661 282764 lpts_pa

65882 179966298 1 exec Reply 0:00:04:0826 295133 devc-vty

65888 179970400 1 exec Reply 0:00:00:0149 1 kernel

65889 179974497 1 more Reply 0:00:00:0064 24593 pipe

65890 179974498 1 show_processes Reply 0:00:00:0000 1 kernel

----------------------------------------------------------------------------------------------------------

Output after commit command:

Jid Pid Tid Name State TimeInState Blocked-on

65549 16397 1 ksh Reply 31223:55:43:0739 16395 devc-conaux

65554 65554 2 devb-umass Reply 0:00:04:0533 53284 io-usb

97 49187 2 umass-enum Reply 31223:55:46:0921 1 kernel

97 49187 6 umass-enum Reply 31223:55:41:0525 53284 io-usb

97 49187 7 umass-enum Reply 31223:55:44:0043 1 kernel

97 49187 8 umass-enum Reply 31223:55:42:0862 1 kernel

97 49187 9 umass-enum Reply 31223:55:41:0526 1 kernel

52 65574 2 attachd Reply 31223:55:44:0862 57381 eth_server

52 65574 3 attachd Reply 31223:55:44:0860 24595 mqueue

88 65580 6 qnet Reply 0:00:00:0005 57381 eth_server

51 65584 2 attach_server Reply 31223:55:44:0671 24595 mqueue

65588 65588 2 devb-umass Reply 0:00:17:0489 53284 io-usb

65593 77881 2 devb-umass Reply 0:00:27:0856 53284 io-usb

435 233568 1 tftp_server Reply 13797:02:04:0857 24595 mqueue

317 295142 2 lpts_fm Reply 0:00:01:0060 282764 lpts_pa

65882 179966298 1 exec Reply 0:00:06:0289 1 kernel

65888 179970400 1 exec Reply 0:00:00:0149 1 kernel

65889 179982689 1 rpl_edit Reply 0:00:05:0176 1 kernel

65890 179986786 1 config Reply 0:00:03:0545 233567 rdsfs_svr

65892 179986788 1 more Reply 0:00:00:0068 24593 pipe

65893 179986789 1 show_processes Reply 0:00:00:0000 1 kernel

Eddie Chami · ‎02-29-2016

That doesn't look problematic, you'll need to start looking at the configmgr traces to see why there is a delay, are you in a position to upgrade to 5.1.3 or 5.3.3?

Eddie.

witani · ‎03-04-2016

Thank you Eddie,

I read that moving from 4.2 to 5.x requires long outages, is that true ?

Regards,

Eddie Chami · ‎03-04-2016

The downtime should not be more than 15mins.

Regards

Eddie.

AARON WEINTRAUB · ‎02-24-2016

On several 4.x releases we had issues with the vic/ether_ctrl processes getting locked up and tied up with the license manager, sometimes causing commits to timeout or take a very long time. A show proc blocked (from another terminal) would be a good starting point.

Commit and Update taking too long on XR OS