Symptoms
Network latency/sluggish performance during normal operations relying heavily upon UDP forward-protocols between normally segregated broadcast domains, internally and site-to-site
Frequent software application time synchronization losses resulting in visual data "freezes" on monitor screens; audio transmissions were unaffected
Diagnosis
Wireshark packet traces revealed frequent parity errors corresponding with UDP data disruptions
During a terminal session on the ISR4331, the command "show process cpu | exclude 0.00" revealed excssively high utilization of the "IP Input" process (typically around 55%) driving overall CPU utilization to 85% or higher during normal, high-volume data operations; a similarly configured ISR2921 on the distant end reported 43% overall utilization on average with only 21% IP Input under the same or similar operating conditions
SolarWinds Network Traffic Analyzer revealed the two greatest "chatterboxes" producing the highest volumes of UDP traffic; verified via the corresponding network switch by clearing interface counters and then by initiating the "show interfaces | include output" command
Solution
Apllied standard expanded (numbered) access control lists (ACL's) to router sub-interface "ip directed-broadcast" commands allowing only directed-broadcasts from specific sources
EXAMPLE:
Router4331(configure)#interface Gi0/0/0.XX
Router4331(configure-if)#ip directed-broacast 101
Router4331(configure-if)#exit
Router4331(configure)#access-list 101 permit udp host x.x.x.x any eq [UDP port]
...
Verified consistent "ip mtu" settings on physical interfaces (where needed), sub-interfaces and tunnels along affected paths
Verified required UDP forward-protocols and removed legacy/unneeded protocols from the global "ip forward-protocol udp" command list
Removed unnecessary "ip directed-broadcast" and "ip network-broadcast" commands from unaffected router sub-interfaces
Verified required (non-DCHP related) ip helper-addresses on router sub-interfaces and removed or changed outdated legacy helpers
The two "chatterbox" servers can be configured for unicast and multicast operations, but given the number of clients spread across the number of remote locations, a major "off-hours" maintenance effort would have to be coordinated for server host and client reconfigurations, so for now, the "Band-Aid" fix will do.
Overall CPU utilization on the ISR4331 dropped from 85% on average to less than 50%, with average "IP Input" process utilization sinking from 55% to 35% during high volume data transfers; not ideal, but certainly a step in the right direction as other configuration issues and options are explored. The bottom line is the CPU is not cooking like it was.
