Re: 3640 Performance

wwbach · ‎09-15-2006

Relative Performance ? 3640 ? 2821

We are currently using a 3640 with an ATM (NM-1A-T3) interface and a Fast Ethernet interface. Most of the traffic is multicast udp inbound from the ATM/T3 circuit. Under load, we are dropping packets. The traffic is limited to 20mb/s. Everyone upstream swears they are getting sent so we are now focusing on the 3640. Under load, I see pretty high cpu usage using ?show proc cpu hist? ? say 40-60% with peaks up to 90%. Are we just over taxing this router? Is there something in the config that is causing the high load? If I were to look to replace this router with something faster (cost is an issue), how should I determine what router to go with? Thanks. -- Bud

Here is some of our config:

# show tech-support

------------------ show version ------------------

Cisco Internetwork Operating System Software

IOS (tm) 3600 Software (C3640-JS-M), Version 12.3(17a), RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Mon 12-Dec-05 16:29 by evmiller

Image text-base: 0x60008B00, data-base: 0x61F08000

ROM: System Bootstrap, Version 11.1(20)AA2, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

ROM: 3600 Software (C3640-JS-M), Version 12.3(17a), RELEASE SOFTWARE (fc2)

Cisco uptime is 1 week, 2 days, 22 hours, 47 minutes

System returned to ROM by power-on

System image file is "flash:c3640-js-mz.123-17a.bin"

cisco 3640 (R4700) processor (revision 0x00) with 124928K/6144K bytes of memory.

Processor board ID 26878521

R4700 CPU at 100MHz, Implementation 33, Rev 1.0

Bridging software.

X.25 software, Version 3.0.0.

TN3270 Emulation software.

1 FastEthernet/IEEE 802.3 interface(s)

1 ATM network interface(s)

DRAM configuration is 64 bits wide with parity disabled.

125K bytes of non-volatile configuration memory.

32768K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

------------------ show running-config ------------------

Building configuration...

Current configuration : 1813 bytes

!

version 12.3

service tcp-keepalives-in

service tcp-keepalives-out

service timestamps debug uptime

service timestamps log uptime

service password-encryption

!

hostname Cisco

!

boot-start-marker

boot-end-marker

!

logging buffered 20000 debugging

enable secret 5 <removed>

!

no aaa new-model

ip subnet-zero

ip tcp synwait-time 5

!

no ip domain lookup

!

no ip bootp server

ip multicast-routing

ip cef

!

buffers small permanent 200

buffers small max-free 300

buffers middle permanent 300

buffers middle max-free 500

!

interface Loopback0

no ip address

shutdown

!

interface FastEthernet0/0

ip address 172.21.x.x 255.255.255.224

no ip redirects

no ip unreachables

no ip proxy-arp

ip pim neighbor-filter PIMFilter

ip pim sparse-mode

no ip mroute-cache

duplex auto

speed auto

no cdp enable

hold-queue 200 in

!

interface ATM1/0

no ip address

atm scrambling cell-payload

no atm ilmi-keepalive

hold-queue 200 in

!

interface ATM1/0.1 point-to-point

ip address 172.20.x.x 255.255.255.252

ip pim sparse-mode

pvc 1/40

vbr-nrt 44209 44209 1

oam-pvc manage 30

oam ais-rdi 30 3

encapsulation aal5snap

!

no ip http server

ip classless

ip route 65.164.x.x 255.255.255.0 172.20.x.x

ip route 204.167.x.x 255.255.255.0 172.20.x.x

!

ip pim bidir-enable

ip pim rp-address 65.164.x.x WAN

ip pim spt-threshold infinity

!

ip access-list standard PIMFilter

deny any

ip access-list standard WAN

permit 233.119.160.0 0.0.0.63

deny any

no logging trap

no cdp run

!

snmp-server community <removed> RO

!

line con 0

exec-timeout 0 0

login

line aux 0

login

transport input all

line vty 0 5

exec-timeout 120 0

password 7 <removed>

logging synchronous

login

length 0

!

end

jackyoung · ‎09-17-2006

The config. looks fine, I could like to clarify is there any external factor to cause the problem.

Could you please also provide the capture of "sh process cpu" and "sh interface" ?

Moreover, if it is possible, try to upgrade the router to a more updated release (e.g. 12.3(20) and test again.

Hope this helps.

wwbach · ‎09-18-2006

Thanks for looking at this Jack (or anyone else). Here is the info you

asked for. Looks like "IP Input" is the only thing using CPU time.

As for any external factors, nothing I can think of other than a fair

amount of udp multicast traffic which is sometimes bursty as it is

market data. We should be limited to less than 20mb/sec -- should the

3640 be able to handle that? I can try to get a new IOS -- do you

think that is likely to make that much of a difference?

If we were to upgrade the router, any recommendations as to what would

be the most cost effective router to go with to support this application?

I know there are other sites handling this traffic with a 2811 but they

are getting the feed over metro ethernet not atm.

-- Bud

[note: snipped everything with less than 100ms runtime]

Cisco#sh process cpu

CPU utilization for five seconds: 39%/21%; one minute: 41%; five minutes: 48%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

2 740 220513 3 0.00% 0.00% 0.00% 0 Load Meter

3 7056 14331 492 0.00% 0.00% 0.00% 0 IP SNMP

5 532256 112331 4738 0.00% 0.04% 0.03% 0 Check heaps

12 17664 48191 366 0.00% 0.00% 0.00% 0 ARP Input

13 664 275604 2 0.00% 0.00% 0.00% 0 HC Counter Timer

18 1064 1102560 0 0.00% 0.00% 0.00% 0 GraphIt

21 1084 220528 4 0.00% 0.00% 0.00% 0 Net Background

23 1460 1102452 1 0.00% 0.00% 0.00% 0 TTY Background

24 7444 1102589 6 0.00% 0.01% 0.00% 0 Per-Second Jobs

36 864 220514 3 0.00% 0.00% 0.00% 0 Compute load avg

37 380320 18403 20666 0.00% 0.04% 0.00% 0 Per-minute Jobs

41 2388 115301 20 0.00% 0.00% 0.00% 0 ATM Periodic

43 21852 85805 254 0.00% 0.00% 0.00% 0 ATM OAM Input

44 1244 50971 24 0.00% 0.00% 0.00% 0 ATM OAM TIMER

50 25886684 27503210 941 17.75% 19.38% 22.66% 0 IP Input

58 692 1102456 0 0.00% 0.00% 0.00% 0 PI MATM Aging Pr

65 5536 4306922 1 0.00% 0.00% 0.00% 0 SSS Feature Time

66 156 2370 65 0.00% 0.00% 0.00% 0 TCP Timer

72 812 36611 22 0.00% 0.00% 0.00% 0 IP Background

81 5844 548365 10 0.00% 0.00% 0.00% 0 ILMI Timer Proce

85 1532 1102469 1 0.00% 0.00% 0.00% 0 ATMSIG Timer

96 276 18382 15 0.00% 0.00% 0.00% 0 Adj Manager

98 6164 1758280 3 0.00% 0.00% 0.00% 0 CEF process

100 1568 1102473 1 0.00% 0.00% 0.00% 0 RUDPV1 Main Proc

102 568 1102467 0 0.00% 0.00% 0.00% 0 bsm_xmt_proc

112 168 111 1513 0.00% 0.00% 0.00% 130 Virtual Exec

114 100 2 50000 0.00% 0.00% 0.00% 0 CCVPM_HDSPRM

124 2568 7166 358 0.00% 0.00% 0.00% 0 PDU DISPATCHER

129 768 38238 20 0.00% 0.00% 0.00% 0 CEF Scanner

130 1384 1836257 0 0.00% 0.00% 0.00% 0 Inline Power

132 792 1102016 0 0.00% 0.00% 0.00% 0 trunk conditioni

134 26628 1140960 23 0.07% 0.01% 0.00% 0 IGMP Input

135 16676 11016870 1 0.15% 0.02% 0.00% 0 Mwheel Process

136 15380 1230412 12 0.15% 0.02% 0.00% 0 PIM Process

137 12220 11017764 1 0.00% 0.01% 0.00% 0 fastblk backgrou

140 14544 7167 2029 0.00% 0.00% 0.00% 0 SNMP ENGINE

143 256 123 2081 0.00% 0.07% 0.03% 131 Virtual Exec

wwbach · ‎09-18-2006

Cisco#show interface

FastEthernet0/0 is up, line protocol is up

Hardware is AmdFE, address is 0008.2156.0b40 (bia 0008.2156.0b40)

Description: LAN segment

Internet address is 172.21.x.x/27

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 2/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 3d00h

Input queue: 1/200/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 2000 bits/sec, 2 packets/sec

5 minute output rate 799000 bits/sec, 721 packets/sec

944066 packets input, 213139879 bytes

Received 538438 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog

0 input packets with dribble condition detected

11297627 packets output, 1671881286 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

ATM1/0 is up, line protocol is up

Hardware is RS8234 ATM DS3

MTU 4470 bytes, sub MTU 4470, BW 45000 Kbit, DLY 190 usec,

reliability 255/255, txload 1/255, rxload 4/255

Encapsulation ATM, loopback not set

Encapsulation(s): AAL5

1023 maximum active VCs, 1 current VCCs

VC Auto Creation Disabled.

VC idle disconnect time: 300 seconds

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 3d00h

Input queue: 19/200/324555/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: Per VC Queueing

5 minute input rate 785000 bits/sec, 718 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

11334587 packets input, 1651601993 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

425646 packets output, 41411309 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

ATM1/0.1 is up, line protocol is up

Hardware is RS8234 ATM DS3

Description: PVC on WAN

Internet address is 172.20.x.x/30

MTU 4470 bytes, BW 45000 Kbit, DLY 190 usec,

reliability 255/255, txload 1/255, rxload 4/255

Encapsulation ATM

11659142 packets input, 1698107996 bytes

425646 packets output, 41411309 bytes

17490 OAM cells input, 17490 OAM cells output

AAL5 CRC errors : 0

AAL5 SAR Timeouts : 0

AAL5 Oversized SDUs : 0

AAL5 length violation : 0

AAL5 CPI Error : 0

2197880768 packets input, 2680635648 bytes

132062473 packets output, 2680548148 bytes

Last clearing of "show interface" counters 3d00h

Loopback0 is administratively down, line protocol is down

Hardware is Loopback

MTU 1514 bytes, BW 8000000 Kbit, DLY 5000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation LOOPBACK, loopback not set

Last input never, output never, output hang never

Last clearing of "show interface" counters 3d00h

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/0 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

0 packets input, 0 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 packets output, 0 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

jackyoung · ‎09-18-2006

Thanks for the info., it looks like the IP inputs took part of the cpu process but the overall loading is not high. I believe it does not require to upgrade the router at the moment, you can consider if the router loading up to 70% then start to plan to upgrade the router. The router model depends on your future requirement. It is hard to tell.

For the interface capture, I found there are lots of broadcast received from Ethernet. I believe it is the traffic that you mentioned of multicast. There is no abnormal symptom at all interfaces, it is healthy.

The 3640 should be able to handle 20Mbps bandwidth, but the issue is the no. of packet per second that it can handle. Checked CCO, 3640 able to support 50-70kpps that you router should far enough to carry the traffic.

You can use traffic policier to limit the througput of different kind of traffic. Check below for more info.

http://www.cisco.com/en/US/tech/tk543/tk545/technologies_tech_note09186a00800a3a25.shtml

http://www.cisco.com/en/US/products/sw/iosswrel/ps1829/products_feature_guide09186a00801b2404.html

Hope this helps.

wwbach · ‎09-19-2006

Check the attached file. It may show better why I believe the 3640 doesn't have enough juice for this application. We are getting peaks to 100%.

-- Bud

wwbach · ‎09-25-2006

Probabaly way overkill but we upgraded to an 3825 and now we are at 15% CPU max and no dropped packets,

jackyoung · ‎09-25-2006

Sorry for missing your pervious message.

Thanks for the update, this is good to know the upgrade solved the problem. I believe there is hardware accerlation in 38xx to load share the CPU loading. :)