02-24-2016 01:59 AM
Dears,
We have using an ASR9K for 3 years. Commit and update used to take 1 or 2 sec. Suddenly it started to take 20 sec to finish.
CPU is around 9 %. Any idea how to approach this problem ?
Regards
02-24-2016 06:23 AM
Dear its probably due to a leak, or a process blocked, or some busted verifier. What release is this on? TO check these you normally start with show log, you'll see noise there, followed by show process blocked, then show proc mem..
if your running old release you may want to upgrade.
If you want to stay on your current release, start looking at the above to give you some leads.
Regards
Eddie.
02-24-2016 11:43 PM
Thank you Eddy.
My version i Cisco IOS XR Software, Version 4.2.1[Default]
I see processes in reply state for a long time, is that what you are talking about ?
RP/0/RSP0/CPU0:cvas-2#sh processes blocked
Thu Feb 25 09:27:08.311 EET
Jid Pid Tid Name State TimeInState Blocked-on
65549 16397 1 ksh Reply 31175:43:10:0070 16395 devc-conaux
65554 65554 2 devb-umass Reply 0:00:00:0120 53284 io-usb
97 49187 2 umass-enum Reply 31175:43:13:0250 1 kernel
97 49187 6 umass-enum Reply 31175:43:07:0854 53284 io-usb
97 49187 7 umass-enum Reply 31175:43:10:0372 1 kernel
97 49187 8 umass-enum Reply 31175:43:09:0191 1 kernel
97 49187 9 umass-enum Reply 31175:43:07:0855 1 kernel
52 65574 2 attachd Reply 31175:43:11:0191 57381 eth_server
52 65574 3 attachd Reply 31175:43:11:0189 24595 mqueue
88 65580 6 qnet Reply 0:00:00:0037 57381 eth_server
51 65584 2 attach_server Reply 31175:43:11:0000 24595 mqueue
65588 65588 2 devb-umass Reply 0:00:02:0553 53284 io-usb
65593 77881 2 devb-umass Reply 0:00:50:0963 53284 io-usb
435 233568 1 tftp_server Reply 13748:49:31:0186 24595 mqueue
317 295142 2 lpts_fm Reply 0:00:00:0074 282764 lpts_pa
65868 128377164 1 more Reply 0:00:00:0063 24593 pipe
65882 128377178 1 show_processes Reply 0:00:00:0000 1 kernel
65892 128323940 1 exec Reply 0:00:00:0161 1 kernel
02-25-2016 12:03 AM
Nothing there looks odd, you probably want to look at this during a commit when its hanging, or syslog/memory leak. You have a pretty old release here, you should upgrade.
02-27-2016 01:36 AM
Hi Eddie,
As you suggested, I looked at blocked processes during commit hanging. Four extra processes appeared. Appreciate if you have a look.
Regards,
Output before commit command:
Jid Pid Tid Name State TimeInState Blocked-on
65549 16397 1 ksh Reply 31223:55:29:0824 16395 devc-conaux
65554 65554 2 devb-umass Reply 0:00:33:0568 53284 io-usb
97 49187 2 umass-enum Reply 31223:55:33:0004 1 kernel
97 49187 6 umass-enum Reply 31223:55:27:0608 53284 io-usb
97 49187 7 umass-enum Reply 31223:55:30:0126 1 kernel
97 49187 8 umass-enum Reply 31223:55:28:0945 1 kernel
97 49187 9 umass-enum Reply 31223:55:27:0609 1 kernel
52 65574 2 attachd Reply 31223:55:30:0945 57381 eth_server
52 65574 3 attachd Reply 31223:55:30:0943 24595 mqueue
88 65580 6 qnet Reply 0:00:00:0048 57381 eth_server
51 65584 2 attach_server Reply 31223:55:30:0755 24595 mqueue
65588 65588 2 devb-umass Reply 0:00:03:0573 53284 io-usb
65593 77881 2 devb-umass Reply 0:00:13:0939 53284 io-usb
435 233568 1 tftp_server Reply 13797:01:50:0940 24595 mqueue
317 295142 2 lpts_fm Reply 0:00:00:0661 282764 lpts_pa
65882 179966298 1 exec Reply 0:00:04:0826 295133 devc-vty
65888 179970400 1 exec Reply 0:00:00:0149 1 kernel
65889 179974497 1 more Reply 0:00:00:0064 24593 pipe
65890 179974498 1 show_processes Reply 0:00:00:0000 1 kernel
----------------------------------------------------------------------------------------------------------
Output after commit command:
Jid Pid Tid Name State TimeInState Blocked-on
65549 16397 1 ksh Reply 31223:55:43:0739 16395 devc-conaux
65554 65554 2 devb-umass Reply 0:00:04:0533 53284 io-usb
97 49187 2 umass-enum Reply 31223:55:46:0921 1 kernel
97 49187 6 umass-enum Reply 31223:55:41:0525 53284 io-usb
97 49187 7 umass-enum Reply 31223:55:44:0043 1 kernel
97 49187 8 umass-enum Reply 31223:55:42:0862 1 kernel
97 49187 9 umass-enum Reply 31223:55:41:0526 1 kernel
52 65574 2 attachd Reply 31223:55:44:0862 57381 eth_server
52 65574 3 attachd Reply 31223:55:44:0860 24595 mqueue
88 65580 6 qnet Reply 0:00:00:0005 57381 eth_server
51 65584 2 attach_server Reply 31223:55:44:0671 24595 mqueue
65588 65588 2 devb-umass Reply 0:00:17:0489 53284 io-usb
65593 77881 2 devb-umass Reply 0:00:27:0856 53284 io-usb
435 233568 1 tftp_server Reply 13797:02:04:0857 24595 mqueue
317 295142 2 lpts_fm Reply 0:00:01:0060 282764 lpts_pa
65882 179966298 1 exec Reply 0:00:06:0289 1 kernel
65888 179970400 1 exec Reply 0:00:00:0149 1 kernel
65889 179982689 1 rpl_edit Reply 0:00:05:0176 1 kernel
65890 179986786 1 config Reply 0:00:03:0545 233567 rdsfs_svr
65892 179986788 1 more Reply 0:00:00:0068 24593 pipe
65893 179986789 1 show_processes Reply 0:00:00:0000 1 kernel
02-29-2016 01:03 AM
That doesn't look problematic, you'll need to start looking at the configmgr traces to see why there is a delay, are you in a position to upgrade to 5.1.3 or 5.3.3?
Eddie.
03-04-2016 04:40 AM
Thank you Eddie,
I read that moving from 4.2 to 5.x requires long outages, is that true ?
Regards,
03-04-2016 05:06 AM
The downtime should not be more than 15mins.
Regards
Eddie.
02-24-2016 07:07 PM
On several 4.x releases we had issues with the vic/ether_ctrl processes getting locked up and tied up with the license manager, sometimes causing commits to timeout or take a very long time. A show proc blocked (from another terminal) would be a good starting point.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide