High NTP CPU utilization

[Shawnw4401]

Problem:

Reoccurring high utilization for NTP

Version:

Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9-M), Version 15.1(2)SY7, RELEASE SOFTWARE (fc4)

CPU:

CPU utilization for five seconds: 77%/1%; one minute: 76%; five minutes: 74%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 334   259753056   1344790     193158 70.28% 65.54% 63.05%   0 NTP
  22    61813604 234158717        263  0.62%  1.33%  1.20%   0 ARP Input
 330    46129644 490418664         94  0.48%  0.57%  0.64%   0 IP Input
 306    10776248  30406982        354  0.34%  0.16%  0.16%   0 QOS Stats Gather
 362    23073276   8172391       2823  0.34%  0.40%  0.40%   0 CEF: IPv4 proces
 634    15132776 383702572         39  0.34%  0.39%  0.39%   0 OSPF-221 Hello
 637    11561020 294704635         39  0.27%  0.27%  0.28%   0 OSPF-300 Hello
 551    41699500 229333176        181  0.27%  0.73%  0.72%   0 Port manager per
 616     2524228   3288406        767  0.27%  0.25%  0.23%   0 Syslog
 321     2089332   9222533        226  0.20%  0.15%  0.14%   0 CDP Protocol
 622     3667584  16965710        216  0.20%  0.20%  0.19%   0 MLSM Process
 354     7054084   1926202       3662  0.13%  0.09%  0.08%   0 XDR receive
 548     3513656  89625492         39  0.13%  0.08%  0.07%   0 IGMP Input
  54      939620   4606656        203  0.13%  0.09%  0.08%   0 Logger

NTP configurations:

ntp authentication-key 1 md5 ********************** 7
ntp authentication-key 500 md5 ********************** 7
ntp authenticate
ntp trusted-key 1
ntp trusted-key 500
ntp source Loopback3
ntp master 3
ntp server ************ key 500
ntp server ************* key 1 prefer

I've tried to re-add just one NTP server (either server) and it is immediately jumps up to 90% utilization.

[Mark Malone]

Hi

seems to be an awful lot NTP bugs in that release version your on , you may be hitting something there , seems very buggy that its hitting that high when pointed to 1 server only , I don't see anything exact that match's your issue but 1 close

Have your tried it without all the optional config security/keys etc just to see , if you only run the command ntp server x.x.x.x prefer does it show the same issue still spiking ?

CSCui60499

ntp

High CPU from Process NTP with "access-group serve" Configured

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/15-1SY/release_notes.html

[Shawnw4401]

Mark,

No, I have not tried without the authentication keys, because it is mandatory for us to have authentication.

[Mark Malone]

Yes understand that we use it too but was suggesting it as test to try and see if one of the commands was the trigger for the issue , as per the bug I posted from your version above the trigger is just one command causing it that's spiking the cpu for ntp for that problem it could be the same in your config , even though config is fine bugs can be triggered by anything in the syntax

there is no identical bug in the release notes for your version so could open it with TAC see if they have something listed that's not publicly available in the release notes or run the show tech through the cli analyser and see if it picks up something for the NTP

or run a netdr capture and see if you can see something being punted at cpu level in software or debug the ntp see if there something irregular showing

there the options I would use when chasing what looks like a bug as its only effecting ntp in cpu

[Shawnw4401]

Mark, 

I can try that tomorrow to see if it does fix anything. We've also thought about upgrading the firmware for the device, as it's only on the 6500 series switches that this is an issue with.

[Mark Malone]

Yes upgrades the quickest way to fix it but without knowing the exact bug id you can run the risk of hitting it again after the upgrade as some bugs are carried through multiple versions , on the website there saying this is the current recommended/suggested version for that supervisor so that may be a version to look at going to as the most stable ...151-2.SY10.bin

https://software.cisco.com/download/release.html?mdfid=280829702&softwareid=280805680&release=15.1.2-SY10&relind=AVAILABLE&rellifecycle=MD&reltype=latest