Re: High CPU on 3640

chris · ‎08-20-2005

I have a 3640 with 2 T1s passing nothing more than RTP and some signaling traffic. The CPU for about 350K worth of traffic right now has been around 20%. All processes are low, it is interrupt traffic. During peak traffic of about 2MB the CPU was closer to 75%. At first I had the T1s in MLPPP. I removed that config thinking that the PPP overhead had something to do with the problem. Right now the links balance with CEF on a per-destination method. Removing the QOS helps a little but not enough to make a difference. Here is my config and a sh int. Any ideas are appreciated.

IOS Version: c3640-is-mz.123-14.t3

Memory 128D/32F

Current configuration : 1841 bytes

!

version 12.3

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname host

!

boot-start-marker

boot-end-marker

!

no logging console

no logging monitor

enable secret 5

enable password 7

!

no aaa new-model

!

resource policy

!

ip subnet-zero

!

ip cef

no ip dhcp use vrf connected

!

class-map match-all voice-signaling

match access-group 103

class-map match-all voice-traffic

match access-group 102

!

policy-map VOICE-POLICY

class voice-traffic

priority 900

class voice-signaling

bandwidth 100

class class-default

fair-queue

!

interface Serial0/0

bandwidth 1544

ip address 10.x.x.x 255.255.255.252

ip ospf cost 6

no ip mroute-cache

serial restart-delay 0

service-policy output VOICE-POLICY

!

interface Serial0/1

bandwidth 1544

ip address 10.x.x.x 255.255.255.252

ip ospf cost 6

no ip mroute-cache

serial restart-delay 0

service-policy output VOICE-POLICY

!

interface Serial0/2

no ip address

shutdown

serial restart-delay 0

!

interface Serial0/3

no ip address

shutdown

serial restart-delay 0

!

interface FastEthernet1/0

ip address 10.x.x.x 255.255.255.128

speed 100

full-duplex

!

router ospf 1

log-adjacency-changes

network 10.x.x.x 0.0.0.3 area 0

network 10.x.x.x 0.0.0.127 area 0

!

no ip http server

!

ip classless

!

access-list 102 permit udp any any range 5000 14600

access-list 103 permit tcp any any eq 4000

snmp-server community X RW

snmp-server location X

!

control-plane

!

line con 0

speed 115200

line aux 0

line vty 0 4

password 7

login

!

end

Georg Pauwen · ‎08-21-2005

Hello,

although this will (temporarily) increase your CPU even more, you could try and configure Netflow on your interfaces ('ip route-cache flow'), in order to see which streams and packet sizes are going through your router...

Regards,

GP

chris · ‎08-22-2005

The config has been updated as well as IOS to 12.4. Here is the current CPU with about 2MB of traffic shared on 2 T1s.

CPU utilization for five seconds: 63%/59%; one minute: 60%; five minutes: 59%

Is this level normal for about 3500 pps?

Configuration register is 0x3922

Current configuration : 2017 bytes

!

version 12.4

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname PEN-VoIP-A

!

boot-start-marker

boot-end-marker

!

no logging console

no logging monitor

enable secret 5

enable password 7

!

no aaa new-model

!

resource policy

!

ip subnet-zero

!

ip cef

!

multilink virtual-template 1

!

class-map match-all voice-signaling

match access-group 103

class-map match-all voice-traffic

match access-group 102

!

policy-map VOICE-POLICY

class voice-traffic

priority 900

class voice-signaling

bandwidth 100

class class-default

fair-queue

!

interface Multilink1

no ip address

ip ospf network point-to-point

ip ospf cost 6

shutdown

ppp multilink

ppp multilink fragment disable

ppp multilink group 1

service-policy output VOICE-POLICY

!

interface Serial0/0

bandwidth 1544

ip address 10.X.X.X 255.255.255.252

no ip mroute-cache

serial restart-delay 0

service-policy output VOICE-POLICY

!

interface Serial0/1

bandwidth 1544

ip address 10..X.X.X 255.255.255.252

no ip mroute-cache

serial restart-delay 0

service-policy output VOICE-POLICY

!

interface Serial0/2

no ip address

shutdown

serial restart-delay 0

!

interface Serial0/3

no ip address

shutdown

serial restart-delay 0

!

interface FastEthernet1/0

ip address 10.X.X.X 255.255.255.128

speed 100

full-duplex

!

router ospf 1

log-adjacency-changes

network 10.X.X.X 0.0.0.3 area 0

network 10.X.X.X 0.0.0.127 area 0

!

no ip http server

!

ip classless

!

access-list 102 permit udp any any range 5000 14600

access-list 103 permit tcp any any eq 4000

snmp-server community ****** RW

snmp-server location TempeDataCenter

snmp-server contact

snmp-server host

!

control-plane

!

line con 0

speed 115200

line aux 0

line vty 0 4

password 7

login

!

end

spremkumar · ‎08-22-2005

hi

AS GP pointed out do check out the traffic pattern which is being handled by the router using netflow.

If you see some starnge traffic passing thru or handled in addition with your normal RTP traffic would suggest to block them off or filter it out using the ACLs .

And also need to check out your config register value which is currently 0x3922 not sure though y you have set to that value any specifics attached to it ?ideally it should be 0x2102 ..

hope this link will be of some help to u..

http://www.cisco.com/en/US/products/products_security_advisories_listing.html

regds

chris · ‎08-23-2005

Thanks guys for the replies. Here is the output of the netflows. Please note that the routable IPs are internal. This system has no internet access. The output is attached.

spremkumar · ‎08-23-2005

Hi

From your attachment i observe lots packets being destined to the following ports from your local lan

commplex-main 5000/tcp

commplex-main 5000/udp

commplex-link 5001/tcp

commplex-link 5001/udp

Are you aware of for what purpose the traffic is being sent to these ports or any application other than voip being used which mite use these ports ?

regds

chris · ‎08-24-2005

Those ports are for the VoIP only. We use a proprietary voice platform that uses a range from 5000 - 14600 in our configuration. I traffic should mainly be UDP and not TCP. I will look into that.