UCM v10.5.2 consumes significant network bandwidth causing network problems - a recent occurrence

[Carl Fitzsimmons]

Hello,

We have had a UCM. v10.5.2 running for several years using VMWare currently VMWare Fusion Professional Version 11.5.3. Recently UCM is consuming significant internal ethernet bandwidth to the point where email and web traffic is significantly affected. No changes to UCM has occurred. The only aspect associated would be the OS which is VMWare Fusion which was the same version 10 and I upgraded to the most recent to see if this would resolve this problem.

I have narrowed this down to UCM using our ASA5506. Looking at the outside interface Traffic usage we could see we exceeded 75000 kpbs. This is abnormally high and should be around 20000-25000 kbps. 

Looking at traffic the UCM IP was consuming the majority of bandwidth. When I reboot the UCM traffic becomes normal 20000-25000 kbps with immediate drop off occurring. I have looked at Location and LBM in UCM (we only have a PUB) and checked those settings which have not been changed.

We have also just had our ASA5506 replaced on RMA which was associated to this Field Notice: http://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64228.html however to the best of my knowledge that should not cause this problem. ASA Version 9.5(1) ASDM 7.5(1)

Currently our only connection is SIP through Vitelity and that has been solid for a few years.

Resetting UCM is not an advisable long term solution. I am looking for suggestions on what to look at to identify the problem and  work on a remedy. Suggestions are welcomed.

Thanks

Carl

[Carl Fitzsimmons]

If anyone searches on this for a similar problem this is what I found.

I setup RTMT to run with Server - Process being viewed and sorting on %CPU

When ASA showed high network traffic and looking at RTMT I observed ntpd CPU utilization was taking more than 82%. So I logged into UCM Cisco OS Unified Administration to the NTP list servers. In my case I had 4. I then deleted them and installed our local router and a public. Then I went to our local router and checked that it was able to get to its external NTP source. After some reading on Cisco Forum I changed our NTP Master to a NTP Master 1. In that reading and if I understood I found that this setting makes it an NTP source to use on our internal network over say a NTP Master 2. Originally that was set to 15. If that is not a correct understanding please advise.

When I deleted the NTP sources in UCM the %utilization went to less than 10%. I just verified our internal NTP settings on out internal router.

So for me it was NTPD service that caused the problem

[Carl Fitzsimmons]

Never claim victory too soon. UCM server pegged again and ntpd was using over 80% again. So I created a reference to 10.1.1.1 which points to another router internally which is not a NTP Master and deleted the reference to our NTP Master. CPU utilization went to below 10% again. Will wait to see what affect this has. I have attached a pic from the ASA showing the reduced ethernet traffic after making this change.