11-02-2018 03:41 AM - edited 03-08-2019 04:32 PM
Dear Experts,
Please suggest, we are getting UDP traffic either broadcast or multicast in router 2911 which cause 95%+ utilisation of the router. And we got error in link and business impact due to this. Company has stock exchange business and nano sec downtime worries for us. Please suggest how to control unwanted traffic coming in to cisco router 2911?
Other surprising thing, servers communicated with through LAN network only but why Router CPU utilisation got increased? Sharing router configuration if you found some missing or over configuration which helps to understand better. Highly appreciate your suggestion.
Current configuration : 6715 bytes
!
! Last configuration change at 09:16:50 IST Fri Nov 2 2018
! NVRAM config last updated at 15:08:12 IST Wed Oct 31 2018
! NVRAM config last updated at 15:08:12 IST Wed Oct 31 2018
version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname MUMBAI-NSE
!
boot-start-marker
boot system flash c2900-universalk9-mz.SPA.152-1.T4.bin
boot-end-marker
!
!
no logging on
enable password NNNNNN
!
no aaa new-model
clock timezone IST 5 30
!
no ipv6 cef
!
!
!
ip multicast-routing
!
!
ip flow-cache timeout active 1
ip cef
multilink bundle-name authenticated
!
no mpls ip
!
crypto pki token default removal timeout 0
!
!
license udi pid CISCO2911/K9 sn FGL151912YC
license boot module c2900 technology-package datak9
!
!
!
redundancy
!
!
ip ftp username itsdc
ip ftp password jhjytg
!
class-map match-all SQOS
match access-group name sgx
class-map match-all qos2
match access-group name file
class-map match-all other
match access-group 121
class-map match-all qos
match access-group 120
!
!
policy-map FILE
class qos2
bandwidth 800
policy-map BQOS
class qos
bandwidth 40000
queue-limit 1000 packets
class other
bandwidth 5000
queue-limit 10 packets
policy-map SQOS
class SQOS
priority level 1
class other
priority level 2
policy-map SGX
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description NSE-BSE
ip address 172.16.18.2 255.255.255.252
ip pim sparse-dense-mode
ip flow ingress
ip flow egress
ip ospf dead-interval minimal hello-multiplier 3
load-interval 30
duplex auto
speed 100
service-policy output BQOS
!
interface GigabitEthernet0/1
description NSE-GGN
ip address 10.95.253.81 255.255.255.252
ip pim sparse-dense-mode
ip flow ingress
ip flow egress
ip ospf dead-interval minimal hello-multiplier 3
load-interval 30
duplex full
speed auto
service-policy output BQOS
!
interface GigabitEthernet0/2
description LOCAL-LAN
ip address 172.25.40.100 255.255.0.0
ip access-group 101 in
ip accounting output-packets
ip pim sparse-dense-mode
ip flow ingress
ip flow egress
ip virtual-reassembly in
ip route-cache same-interface
ip route-cache policy
duplex auto
speed auto
!
interface FastEthernet0/0/0
description NSE-DGCX
ip address 172.16.26.1 255.255.255.0
ip access-group 130 in
ip pim sparse-dense-mode
ip flow ingress
ip flow egress
ip ospf dead-interval minimal hello-multiplier 3
load-interval 30
duplex auto
speed auto
service-policy output SQOS
!
interface FastEthernet0/1/0
description NSE-MCX
ip address 172.16.20.1 255.255.255.0
ip ospf dead-interval minimal hello-multiplier 3
duplex auto
speed auto
!
interface FastEthernet0/1/1
description NSE-SGX
ip address 172.16.27.1 255.255.255.0
ip ospf dead-interval minimal hello-multiplier 3
duplex auto
speed auto
!
interface FastEthernet0/2/0
description NSE-CME
ip address 1.29.75.9 255.255.255.248
duplex auto
speed auto
!
interface FastEthernet0/2/1
no ip address
shutdown
duplex auto
speed auto
!
!
router ospf 2
network 10.95.253.81 0.0.0.0 area 0
network 172.16.18.0 0.0.0.3 area 0
network 172.16.20.0 0.0.0.3 area 0
network 172.16.20.0 0.0.0.255 area 0
network 172.16.23.0 0.0.0.3 area 0
network 172.16.26.0 0.0.0.255 area 0
network 172.16.27.0 0.0.0.255 area 0
network 172.25.0.0 0.0.255.255 area 0
network 192.168.16.0 0.0.0.255 area 0
network 192.168.150.0 0.0.0.255 area 0
maximum-paths 2
!
ip forward-protocol nd
!
ip pim rp-address 10.95.25.82
ip pim autorp listener
no ip http server
no ip http secure-server
ip flow-export source GigabitEthernet0/1
ip flow-export version 9
ip flow-export template timeout-rate 1
ip flow-export destination 191.191.191.52 9996
ip flow-top-talkers
top 40
sort-by bytes
cache-timeout 20000
!
ip route 1.29.7.0 255.255.255.252 172.16.2.2
ip route 1.50.7.0 255.255.255.248 1.29.7.11
ip route 10.29.7.0 255.255.255.0 1.29.7.11
ip route 192.168.1.10 255.255.255.255 10.95.25.82
ip route 192.168.1.0 255.255.255.0 192.168.1.1
ip route 192.168.1.0 255.255.255.0 192.168.1.1
ip route 192.168.6.0 255.255.255.0 10.95.25.82
!
ip access-list extended file
permit tcp any any eq 445
ip access-list extended other
deny udp any any eq 45000
deny udp any any eq 45002
deny udp any any eq 45003
permit ip any any
ip access-list extended sgx
permit udp any any eq 45000
permit udp any any eq 45002
permit udp any any eq 45003
permit tcp any any eq 1801
!
no logging trap
access-list 101 deny udp any any eq 9999
access-list 101 deny udp any any eq 34074
access-list 101 deny udp any any eq 34330
access-list 101 deny udp any any eq 34586
access-list 101 deny udp any any eq 5450
access-list 101 deny udp any any eq 5440
access-list 101 deny udp any any eq 45446 log
access-list 101 deny udp any any eq 80 log
access-list 101 deny udp any any eq 17742 log
access-list 101 deny udp any any eq 50554 log
access-list 101 deny udp any any eq 56955 log
access-list 101 permit ip any any
access-list 110 deny tcp any any eq 3389
access-list 110 deny tcp any any eq 445
access-list 110 permit ip any any
access-list 120 deny ip host 172.25.45.21 any
access-list 120 deny ip host 172.25.45.52 any
access-list 120 deny ip host 172.25.45.18 any
access-list 120 deny ip host 172.25.45.18 any
access-list 120 permit ip any any
access-list 120 deny tcp any any log
access-list 120 deny udp any any log
access-list 120 deny ip host 172.25.45.3 any
access-list 121 deny udp any any eq 45000
access-list 121 deny udp any any eq 45002
access-list 121 deny udp any any eq 45003
access-list 121 permit ip any any
access-list 121 permit ip host 172.25.45.5 any
access-list 121 permit ip host 172.25.45.21 any
access-list 121 permit ip host 172.25.45.18 any
access-list 121 permit ip host 172.25.45.18 any
access-list 121 permit udp any any
access-list 121 permit udp any any eq 45000
access-list 121 permit udp any any eq 45002
access-list 121 permit udp any any eq 45003
access-list 121 deny udp any any log
access-list 121 deny ip host 172.25.45.8 any
access-list 130 deny udp any any eq 9999
access-list 130 deny udp any any eq 34463
access-list 130 permit ip any any
access-list dynamic-extended
!
!
!
!
!
snmp-server community public RW
snmp-server ifindex persist
!
control-plane
!
!
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport input all
transport output lat pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
password OKIJ***&%%
login
transport input all
line vty 5 10
password SAD$$#@
login
transport input all
!
scheduler allocate 20000 1000
end
11-02-2018 05:29 AM
Hello,
what does the rest of your network/topology look like ? You need to find out first where the broadcast and multicast traffic comes from...
11-04-2018 11:00 PM
Our company detailing in Stock exchanges, so it works only on Broadcast and Multicast packets. But one thing still i unable to understand, all communications happened with in LAN network only but why router utilisation goes high up to 98% and sometimes hit 100%. We have checked all packets including broadcast & multicast but we cant blocked or deny any because if we do we don't know where it got affected in the applications or business.
My worries how to control Router CPU utilisation because router do not need to do anything for internal communication.
Regards,
Varun
11-02-2018 08:57 AM
Hi,
have you checked what is causing the cpu to spike? Check the TTL of the multicast stream when it hits the router. You might need to capture the traffic as it enters the routes to see the TTL.
How large is the multicast stream passing through that router?
Thanks
John
11-02-2018 09:42 AM
Obviously, you need to be aware that the 2911 is not a very high throughput router. I note that you are running seven interfaces at 100Mbps or higher, so there is a definite possibility that you could exceed the forwarding capacity of the device.
Hope this helps,
Dave
11-04-2018 11:07 PM
11-05-2018 12:46 AM
Hello,
post the output of:
show processes cpu sorted | ex 0.00
11-05-2018 02:41 AM
11-05-2018 05:31 AM
Hello,
as Paul suggested, did you remove all 'log' keywords from your access lists ? Anything logged is process switched, which can cause high CPU and IP Input counter increases...
11-05-2018 03:20 AM - edited 11-05-2018 05:07 AM
Hello
Your cpu interrupt level is very high and its related cpu process indicates network traffic flooding the cpu.
Quite a few things that possibly warrant such cpu interrupt utilisation could be your netflow and acl logging, and you have cef enabled but is it applied to the physical interfaces?
suggest:
disable any debugging/span sessions
remove any acl logging
temporary disable netflow/ip accounting
check to make sure you are indeed fast/cef switching
sh debug
sh monitor
sh ip cef
sh cef not-sef switched
sh ip cache
sh ip interface gigx/x | in IP
sh interface gigx/x stats
conf t
un all
no monitor session x
interface GigabitEthernet0/0
no ip flow ingress
no ip flow egress
interface GigabitEthernet0/
no ip flow ingress
no ip flow egress
interface GigabitEthernet0/2
no ip accounting output-packets
no ip route-cache policy
no ip route-cache same-interface
no ip flow ingress
no ip flow egress
no access-list 101
access-list 101 deny udp any any eq 34074
access-list 101 deny udp any any eq 34330
access-list 101 deny udp any any eq 34586
access-list 101 deny udp any any eq 5450
access-list 101 deny udp any any eq 5440
access-list 101 deny udp any any eq 45446
access-list 101 deny udp any any eq 80
access-list 101 deny udp any any eq 17742
access-list 101 deny udp any any eq 50554
access-list 101 deny udp any any eq 56955
access-list 101 permit ip any any
no access-list 120
access-list 120 deny ip host 172.25.45.18 any
access-list 120 deny ip host 172.25.45.3 any
access-list 120 permit ip any any
no access-list 121
access-list 121 deny ip host 172.25.45.8 any
access-list 121 deny udp any any eq 45000
access-list 121 deny udp any any eq 45002
access-list 121 deny udp any any eq 45003
access-list 121 permit ip any any
11-05-2018 03:54 AM
11-05-2018 05:08 AM - edited 11-05-2018 05:34 AM
Hello
@Varun Luthra wrote:
Thanks Paul for replying.
Now production is down, so cant do anything. Will check all given commands by tomorrow and reply you after that.
But before implement all this, i have one question. As you mentioned, to apply "no access-list 101", "no access-list 120", "no access-list 121" if i implement this command then i think it will stop communication with specific ports or IP address which may be impact in production environment. Because application team suggest us to block certain ports and IP addresses. So suggest should i proceed or consider something else.
Yes oh course, that was just a suggestion to amend the acl however if you cannot remove the acl from the interface then you can do it whilst its still applied to the interface but you need to know what ace's to remove or to change.
show ip access-list x/x will show the related numbering to each ace entry within that acl.
example:
sh ip access-lists 101
Extended IP access list 101
10 deny udp any any eq 34074
20 deny udp any any eq 34330
30 deny udp any any eq 34586 log <--------------------- change this to remove the logging
40 deny udp any any eq 5450
50 permit ip any any <---------------------- relocate this to the end of the acl
60 deny udp any any eq 5440
70 deny udp any any eq 45446
80 deny udp any any eq 80 log
90 deny udp any any eq 17742
100 deny udp any any eq 50554
110 deny udp any any eq 56955
conf t
ip access-list extended 101
no 30
30 deny udp any any eq 34586
no 50
120 permit ip any any
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide