WS-c3750X-24T-S High CPU Utilization Auth Manager

[Oleksii Martyshchenko]

Yesterday I've upgraded the stack of two WS-c3750X-24T-S from c3750e-universalk9-mz.122-55.SE1.bin to c3750e-ipbasek9-mz.150-2.SE.bin. After upgrading it began to flood the console by 'Failed to send hrpc non blocking message'. I've read that "the message is just a symptom of some other root cause". After this I found out that CPU is loaded at 99%.

Sorting gave me this:

ho_c3750_dmz#sh proc cpu so 5s

CPU utilization for five seconds: 55%/3%; one minute: 58%; five minutes: 67%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

206    17223728     1968099       8751 18.88% 22.50% 27.61%   0 Auth Manager

110     6350053     1517346       4184 10.40%  9.20% 11.17%   0 HLFM address lea

  97     2553410     3923828        650  4.96%  3.93%  4.20%   0 hrpc <- response

362     1227655     1940350        632  2.55%  1.89%  2.03%   0 AAA SEND STOP EV

  88     1217078      232103       5243  2.08%  1.76%  1.74%   0 RedEarth Tx Mana

   4     1116012       48989      22780  1.91%  1.82%  1.74%   0 Check heaps

176     2214496     1512804       1463  0.95%  2.31%  3.60%   0 Hulc LED Process

153     1231720     1109876       1109  0.95%  1.70%  1.74%   0 HULC DOT1X Proce

....

Can't find any information about Auth Manager and corresponding troubleshooting.

Please help!

[InayathUlla Sharieff]

Alex,

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

Also I would appriciate if you could upgrade your IOS to 12.2(55)SE7 as this is more stable one and have all the high cpu issues fixed in it.

Rgards

Inayath

[Dale Miller · Accepted solution]

I am sure you hitting a known defect CSCuf32893. If you need 15.0 for specific feature try the workaround published in the release note. Otherwise, follow the advice provided and downgrade to 12.2(55).

Regards,

Dale

[Oleksii Martyshchenko]

The thing is that upgrading if for getting IPSec functionality. Otherwise, no sense of upgrading at all. Every thing was stable with c3750e-universalk9-mz.122-55.SE1.bin.

[InayathUlla Sharieff]

Alex,

So whats yoiur thought on this?

If you still wants this to be TS then get the TAC case raised?

Regards

Inayath

[Oleksii Martyshchenko]

No opportunity to open the case.

[Oleksii Martyshchenko]

I read the troubleshooting of High CPU Utilization before creating this discussion

here http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

and here http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml#SDM

And I havn't found any mention about Auth Manager.

[InayathUlla Sharieff · Accepted solution]

Unfortunately thats not there.

Anyways as suggested there are 2 workaround to get rid of this issue:

1- Software downgrade

2- Disable micro.

Please find the defect details as given below : CSCuf32893<http://cdets.cisco.com/apps/dumpcr?&content=summary&format=html&identifier=CSCuf32893> 3750 - memory depletion due to Auth Manager Symptom: Auth Manager continues to hold more memory in Processor Pool locking out access to the 3750 stack unless switch is rebooted. Conditions: Stack running 150-2.SE1.bin// note your current IOS version is impacted as well. Workaround: Downgrade to 12.2(55)SE or Disable device sensor by "no macro auto monitor "

HTH

Regards

Inayath

*Plz rate all usefull posts and mark the thread as answered if this has answered your query.

[Oleksii Martyshchenko]

Sorry, give me 10 minutes to read the link. I haven't done it yet.

[Oleksii Martyshchenko]

Can't open the page. I thought it was problems with Inet here in my office, but I asked to check from another place and it's the same result.

But I understood the variants. Can you explain what will disabling device sensor issue?

[Oleksii Martyshchenko]

I haven't wait till answer and enter the command.

Result

ho_c3750_dmz#sh proc cpu history

11          11111                         1111122222

007777788888111118888888888888889999988888333332222299999999

100

90

80

70

60

50

40

30

20                                                *****

10 **********************************************************

0....5....1....1....2....2....3....3....4....4....5....5....6

0    5    0    5    0    5    0    5    0    5    0

CPU% per second (last 60 seconds)

229999999999999999999999999999999999999999999999999999999999

209986888998877897978979986887578866887748888768797898999988

100   ************************************** *****************

90   *#*****####*#*#*######***#**#**#**####**####*****###**##

80   *############################*##########################

70   ########################################################

60   ########################################################

50   ########################################################

40   ########################################################

30   ########################################################

20 **########################################################

10 ##########################################################

0....5....1....1....2....2....3....3....4....4....5....5....6

0    5    0    5    0    5    0    5    0    5    0

CPU% per minute (last 60 minutes)

* = maximum CPU%   # = average CPU%

Like falling from a rock.

[InayathUlla Sharieff]

Congrats.

Regards

Inayath

*Plz rate all usefull posts and mark the thread as answered if this has answered your query.

[Bilal Nawaz]

Hello Alex, as Inayath has stated 12.2(55)SE7 is one of the most stable IOS versions available. It is strongly advised to downgrade to this version if possible.

You can see what difference it made for me to be on the stated version in terms of resource utilisation.

https://supportforums.cisco.com/thread/2222308

Hope this helps

Sent from Cisco Technical Support iPhone App

[Oleksii Martyshchenko]

Yesterday I couldn't finded any answer and I desided to proceed with upgrading. So I've upgraded up  to c3750e-ipbasek9-mz.150-2.SE2.bin.

As a result CPU utilization declined to about 60%, but now it's again

ho_c3750_dmz#sh proc cpu s 5s

CPU utilization for five seconds: 90%/2%; one minute: 86%; five minutes: 85%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

210     8255967     1555567       5307 51.27% 38.69% 37.95%   0 Auth Manager

112     3325956     1290135       2577 12.93% 16.37% 15.49%   0 HLFM address lea

178      935389     1193043        784  5.43%  6.27%  5.82%   0 Hulc LED Process

366      719826     1526283        471  2.55%  2.11%  2.12%   0 AAA SEND STOP EV

155      880331      805221       1093  1.75%  1.70%  1.71%   0 HULC DOT1X Proce

  99     1814891     3683140        492  1.27%  4.18%  4.09%   0 hrpc <- response

  90      943476      188086       5016  1.27%  1.70%  1.71%   0 RedEarth Tx Mana

   5      895908       39490      22686  0.95%  1.73%  1.72%   0 Check heaps

  89      409743      257847       1589  0.63%  1.07%  1.05%   0 RedEarth I2C dri

415       16098        7286       2209  0.63%  0.05%  0.01%   0 SNMP ENGINE

133      187978       48342       3888  0.63%  0.51%  0.54%   0 hpm counter proc

264       14741      479842         30  0.31%  0.15%  0.10%   0 UDLD

....

So as a total result - no changes.

[InayathUlla Sharieff]

As the problem is resolved now.

Could you mark the post as answered and rate all the usefull posts of this thread. This will help others if any facing the same issue in future and it would be helpfull for them if the thread is mark as answered so that they use the same action plan implemented.

Have a nice day.

REgards

Inayath