Re: How to control unwanted UDP traffic (Broadcast & Multicast) on Cisco Router 2911?

Varun Luthra · ‎11-02-2018

Dear Experts,

Please suggest, we are getting UDP traffic either broadcast or multicast in router 2911 which cause 95%+ utilisation of the router. And we got error in link and business impact due to this. Company has stock exchange business and nano sec downtime worries for us. Please suggest how to control unwanted traffic coming in to cisco router 2911?

Other surprising thing, servers communicated with through LAN network only but why Router CPU utilisation got increased? Sharing router configuration if you found some missing or over configuration which helps to understand better. Highly appreciate your suggestion.

Current configuration : 6715 bytes

!

! Last configuration change at 09:16:50 IST Fri Nov 2 2018

! NVRAM config last updated at 15:08:12 IST Wed Oct 31 2018

version 15.2

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname MUMBAI-NSE

!

boot-start-marker

boot system flash c2900-universalk9-mz.SPA.152-1.T4.bin

boot-end-marker

!

no logging on

enable password NNNNNN

!

no aaa new-model

clock timezone IST 5 30

!

no ipv6 cef

!

ip multicast-routing

!

ip flow-cache timeout active 1

ip cef

multilink bundle-name authenticated

!

no mpls ip

!

crypto pki token default removal timeout 0

!

license udi pid CISCO2911/K9 sn FGL151912YC

license boot module c2900 technology-package datak9

!

redundancy

!

ip ftp username itsdc

ip ftp password jhjytg

!

class-map match-all SQOS

match access-group name sgx

class-map match-all qos2

match access-group name file

class-map match-all other

match access-group 121

class-map match-all qos

match access-group 120

!

policy-map FILE

class qos2

bandwidth 800

policy-map BQOS

class qos

bandwidth 40000

queue-limit 1000 packets

class other

bandwidth 5000

queue-limit 10 packets

policy-map SQOS

class SQOS

priority level 1

class other

priority level 2

policy-map SGX

!

interface Embedded-Service-Engine0/0

no ip address

shutdown

!

interface GigabitEthernet0/0

description NSE-BSE

ip address 172.16.18.2 255.255.255.252

ip pim sparse-dense-mode

ip flow ingress

ip flow egress

ip ospf dead-interval minimal hello-multiplier 3

load-interval 30

duplex auto

speed 100

service-policy output BQOS

!

interface GigabitEthernet0/1

description NSE-GGN

ip address 10.95.253.81 255.255.255.252

ip pim sparse-dense-mode

ip flow ingress

ip flow egress

ip ospf dead-interval minimal hello-multiplier 3

load-interval 30

duplex full

speed auto

service-policy output BQOS

!

interface GigabitEthernet0/2

description LOCAL-LAN

ip address 172.25.40.100 255.255.0.0

ip access-group 101 in

ip accounting output-packets

ip pim sparse-dense-mode

ip flow ingress

ip flow egress

ip virtual-reassembly in

ip route-cache same-interface

ip route-cache policy

duplex auto

speed auto

!

interface FastEthernet0/0/0

description NSE-DGCX

ip address 172.16.26.1 255.255.255.0

ip access-group 130 in

ip pim sparse-dense-mode

ip flow ingress

ip flow egress

ip ospf dead-interval minimal hello-multiplier 3

load-interval 30

duplex auto

speed auto

service-policy output SQOS

!

interface FastEthernet0/1/0

description NSE-MCX

ip address 172.16.20.1 255.255.255.0

ip ospf dead-interval minimal hello-multiplier 3

duplex auto

speed auto

!

interface FastEthernet0/1/1

description NSE-SGX

ip address 172.16.27.1 255.255.255.0

ip ospf dead-interval minimal hello-multiplier 3

duplex auto

speed auto

!

interface FastEthernet0/2/0

description NSE-CME

ip address 1.29.75.9 255.255.255.248

duplex auto

speed auto

!

interface FastEthernet0/2/1

no ip address

shutdown

duplex auto

speed auto

!

router ospf 2

network 10.95.253.81 0.0.0.0 area 0

network 172.16.18.0 0.0.0.3 area 0

network 172.16.20.0 0.0.0.3 area 0

network 172.16.20.0 0.0.0.255 area 0

network 172.16.23.0 0.0.0.3 area 0

network 172.16.26.0 0.0.0.255 area 0

network 172.16.27.0 0.0.0.255 area 0

network 172.25.0.0 0.0.255.255 area 0

network 192.168.16.0 0.0.0.255 area 0

network 192.168.150.0 0.0.0.255 area 0

maximum-paths 2

!

ip forward-protocol nd

!

ip pim rp-address 10.95.25.82

ip pim autorp listener

no ip http server

no ip http secure-server

ip flow-export source GigabitEthernet0/1

ip flow-export version 9

ip flow-export template timeout-rate 1

ip flow-export destination 191.191.191.52 9996

ip flow-top-talkers

top 40

sort-by bytes

cache-timeout 20000

!

ip route 1.29.7.0 255.255.255.252 172.16.2.2

ip route 1.50.7.0 255.255.255.248 1.29.7.11

ip route 10.29.7.0 255.255.255.0 1.29.7.11

ip route 192.168.1.10 255.255.255.255 10.95.25.82

ip route 192.168.1.0 255.255.255.0 192.168.1.1

ip route 192.168.6.0 255.255.255.0 10.95.25.82

!

ip access-list extended file

permit tcp any any eq 445

ip access-list extended other

deny udp any any eq 45000

deny udp any any eq 45002

deny udp any any eq 45003

permit ip any any

ip access-list extended sgx

permit udp any any eq 45000

permit udp any any eq 45002

permit udp any any eq 45003

permit tcp any any eq 1801

!

no logging trap

access-list 101 deny udp any any eq 9999

access-list 101 deny udp any any eq 34074

access-list 101 deny udp any any eq 34330

access-list 101 deny udp any any eq 34586

access-list 101 deny udp any any eq 5450

access-list 101 deny udp any any eq 5440

access-list 101 deny udp any any eq 45446 log

access-list 101 deny udp any any eq 80 log

access-list 101 deny udp any any eq 17742 log

access-list 101 deny udp any any eq 50554 log

access-list 101 deny udp any any eq 56955 log

access-list 101 permit ip any any

access-list 110 deny tcp any any eq 3389

access-list 110 deny tcp any any eq 445

access-list 110 permit ip any any

access-list 120 deny ip host 172.25.45.21 any

access-list 120 deny ip host 172.25.45.52 any

access-list 120 deny ip host 172.25.45.18 any

access-list 120 permit ip any any

access-list 120 deny tcp any any log

access-list 120 deny udp any any log

access-list 120 deny ip host 172.25.45.3 any

access-list 121 deny udp any any eq 45000

access-list 121 deny udp any any eq 45002

access-list 121 deny udp any any eq 45003

access-list 121 permit ip any any

access-list 121 permit ip host 172.25.45.5 any

access-list 121 permit ip host 172.25.45.21 any

access-list 121 permit ip host 172.25.45.18 any

access-list 121 permit udp any any

access-list 121 permit udp any any eq 45000

access-list 121 permit udp any any eq 45002

access-list 121 permit udp any any eq 45003

access-list 121 deny udp any any log

access-list 121 deny ip host 172.25.45.8 any

access-list 130 deny udp any any eq 9999

access-list 130 deny udp any any eq 34463

access-list 130 permit ip any any

access-list dynamic-extended

!

snmp-server community public RW

snmp-server ifindex persist

!

control-plane

!

line con 0

line aux 0

line 2

no activation-character

no exec

transport preferred none

transport input all

transport output lat pad telnet rlogin lapb-ta mop udptn v120 ssh

stopbits 1

line vty 0 4

password OKIJ***&%%

login

transport input all

line vty 5 10

password SAD$$#@

login

transport input all

!

scheduler allocate 20000 1000

end

Georg Pauwen · ‎11-02-2018

Hello,

what does the rest of your network/topology look like ? You need to find out first where the broadcast and multicast traffic comes from...

Varun Luthra · ‎11-04-2018

Our company detailing in Stock exchanges, so it works only on Broadcast and Multicast packets. But one thing still i unable to understand, all communications happened with in LAN network only but why router utilisation goes high up to 98% and sometimes hit 100%. We have checked all packets including broadcast & multicast but we cant blocked or deny any because if we do we don't know where it got affected in the applications or business.

My worries how to control Router CPU utilisation because router do not need to do anything for internal communication.

Regards,

Varun

johnd2310 · ‎11-02-2018

Hi,

have you checked what is causing the cpu to spike? Check the TTL of the multicast stream when it hits the router. You might need to capture the traffic as it enters the routes to see the TTL.

How large is the multicast stream passing through that router?

Thanks

John

**Please rate posts you find helpful**

dbeattie · ‎11-02-2018

Obviously, you need to be aware that the 2911 is not a very high throughput router. I note that you are running seven interfaces at 100Mbps or higher, so there is a definite possibility that you could exceed the forwarding capacity of the device.

Hope this helps,

Dave

Varun Luthra · ‎11-04-2018

Yes, it could be possibility. We have 6 different ISP's P2P links from different countries terminated on 2911 router and few links more than 10 Mbps and few 48 Mbps links. And everyone using the same Database and applications running under intranet. All communication happening between LAN network only but why Router CPU utilisation goes high and touch 98% and even sometimes 100%. Still, unable to find the solution.

Appreciate your suggestions.

Thanks & Regards,
Varun Luthra

Georg Pauwen · ‎11-05-2018

Hello,

post the output of:

show processes cpu sorted | ex 0.00

Varun Luthra · ‎11-05-2018

MUMBAI-NSE#sh processes cpu sorted | exclude 0.00
CPU utilization for five seconds: 98%/52%; one minute: 94%; five minutes: 51%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
128 121348928 217679514 557 44.39% 43.28% 22.58% 0 IP Input
302 1857916 36103274 51 0.55% 0.52% 0.50% 0 OSPF-2 Hello
6 512812 112661 4551 0.39% 0.06% 0.05% 0 Check heaps
285 252 117 2153 0.31% 0.11% 0.03% 390 Virtual Exec
99 617608 99619022 6 0.23% 0.20% 0.18% 0 Ethernet Msec Ti
279 92760 8036450 11 0.07% 0.02% 0.01% 0 IGMP Input
280 182696 17186564 10 0.07% 0.06% 0.07% 0 PIM Process
227 128684 24756346 5 0.07% 0.02% 0.01% 0 MMON MENG
95 87756 1584436 55 0.07% 0.02% 0.02% 0 BPSM stat Proces
296 259516 39297692 6 0.07% 0.08% 0.07% 0 MFIB_mrib_write
65 187972 794762 236 0.07% 0.09% 0.08% 0 Per-Second Jobs

---------------------------------------------------------------------------------------------------------

MUMBAI-NSE#sh processes cpu sorted | exclude 0.00
CPU utilization for five seconds: 99%/51%; one minute: 96%; five minutes: 61%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
128 121416644 217697733 557 45.83% 44.54% 27.32% 0 IP Input
302 1858488 36106392 51 0.47% 0.52% 0.50% 0 OSPF-2 Hello
285 344 140 2457 0.31% 0.10% 0.05% 390 Virtual Exec
99 617836 99627824 6 0.23% 0.20% 0.18% 0 Ethernet Msec Ti
280 182780 17188128 10 0.07% 0.06% 0.07% 0 PIM Process
65 188060 794833 236 0.07% 0.09% 0.08% 0 Per-Second Jobs
227 128720 24758559 5 0.07% 0.02% 0.01% 0 MMON MENG
279 92816 8037173 11 0.07% 0.02% 0.01% 0 IGMP Input
296 259592 39301179 6 0.07% 0.08% 0.07% 0 MFIB_mrib_write

Georg Pauwen · ‎11-05-2018

Hello,

as Paul suggested, did you remove all 'log' keywords from your access lists ? Anything logged is process switched, which can cause high CPU and IP Input counter increases...

paul driver · ‎11-05-2018

Hello

Your cpu interrupt level is very high and its related cpu process indicates network traffic flooding the cpu.

Quite a few things that possibly warrant such cpu interrupt utilisation could be your netflow and acl logging, and you have cef enabled but is it applied to the physical interfaces?

suggest:
disable any debugging/span sessions
remove any acl logging
temporary disable netflow/ip accounting

check to make sure you are indeed fast/cef switching

sh debug
sh monitor
sh ip cef
sh cef not-sef switched
sh ip cache
sh ip interface gigx/x | in IP
sh interface gigx/x stats

conf t
un all
no monitor session x

interface GigabitEthernet0/0
no ip flow ingress
no ip flow egress

interface GigabitEthernet0/
no ip flow ingress
no ip flow egress

interface GigabitEthernet0/2
no ip accounting output-packets
no ip route-cache policy
no ip route-cache same-interface
no ip flow ingress
no ip flow egress

no access-list 101
access-list 101 deny udp any any eq 34074
access-list 101 deny udp any any eq 34330
access-list 101 deny udp any any eq 34586
access-list 101 deny udp any any eq 5450
access-list 101 deny udp any any eq 5440
access-list 101 deny udp any any eq 45446
access-list 101 deny udp any any eq 80
access-list 101 deny udp any any eq 17742
access-list 101 deny udp any any eq 50554
access-list 101 deny udp any any eq 56955
access-list 101 permit ip any any

no access-list 120
access-list 120 deny ip host 172.25.45.18 any
access-list 120 deny ip host 172.25.45.3 any
access-list 120 permit ip any any

no access-list 121
access-list 121 deny ip host 172.25.45.8 any
access-list 121 deny udp any any eq 45000
access-list 121 deny udp any any eq 45002
access-list 121 deny udp any any eq 45003
access-list 121 permit ip any any

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Varun Luthra · ‎11-05-2018

Thanks Paul for replying.

Now production is down, so cant do anything. Will check all given commands by tomorrow and reply you after that.

But before implement all this, i have one question. As you mentioned, to apply "no access-list 101", "no access-list 120", "no access-list 121" if i implement this command then i think it will stop communication with specific ports or IP address which may be impact in production environment. Because application team suggest us to block certain ports and IP addresses. So suggest should i proceed or consider something else.

Regards,
Varun Luthra

paul driver · ‎11-05-2018

Hello

@Varun Luthra wrote:
Thanks Paul for replying.

Now production is down, so cant do anything. Will check all given commands by tomorrow and reply you after that.

But before implement all this, i have one question. As you mentioned, to apply "no access-list 101", "no access-list 120", "no access-list 121" if i implement this command then i think it will stop communication with specific ports or IP address which may be impact in production environment. Because application team suggest us to block certain ports and IP addresses. So suggest should i proceed or consider something else.

Yes oh course, that was just a suggestion to amend the acl however if you cannot remove the acl from the interface then you can do it whilst its still applied to the interface but you need to know what ace's to remove or to change.

show ip access-list x/x will show the related numbering to each ace entry within that acl.

example:

sh ip access-lists 101
Extended IP access list 101
10 deny udp any any eq 34074
20 deny udp any any eq 34330
30 deny udp any any eq 34586 log <--------------------- change this to remove the logging
40 deny udp any any eq 5450
50 permit ip any any <---------------------- relocate this to the end of the acl
60 deny udp any any eq 5440
70 deny udp any any eq 45446
80 deny udp any any eq 80 log
90 deny udp any any eq 17742
100 deny udp any any eq 50554
110 deny udp any any eq 56955

conf t

ip access-list extended 101
no 30
30 deny udp any any eq 34586
no 50
120 permit ip any any

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul