Solved: 7206 Npe G2 high cpu problems

Umit AYDINLI · ‎03-05-2012

Hi,

I have a lot of cisco 7206 vxr deviceses. I have a high cpu problem.

I have a 7206 vxr g2. I used to use ebgp for three upstream carrier. I used to bgp full route table method. I have about 800 Mbps active traffics and behing the router about 1600 active customers.

Could you advice me how can I trouble shoot this high cpu problem?

#sh int gigabitEthernet 0/1

GigabitEthernet0/1 is up, line protocol is up

Hardware is MV64460 Internal MAC, address is 000c.cf1d.d01b (bia 000c.cf1d.d01b)

Description: ***** GW Interface *****

Internet address is 213.144.96.1/24

MTU 1700 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 31/255, rxload 45/255

Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set

Keepalive set (10 sec)

Full Duplex, 1000Mbps, media type is RJ45

output flow-control is XON, input flow-control is unsupported

ARP type: ARPA, ARP Timeout 03:20:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 4w4d

Input queue: 0/4096/4860/1347216 (size/max/drops/flushes); Total output drops: 13800835

Queueing strategy: fifo

Output queue: 0/4096 (size/max)

30 second input rate 177466000 bits/sec, 35236 packets/sec

30 second output rate 125457000 bits/sec, 33418 packets/sec

2016938526 packets input, 1849362236 bytes, 4794 no buffer

Received 50788313 broadcasts (15690571 IP multicasts)

0 runts, 0 giants, 0 throttles

51133 input errors, 44 CRC, 0 frame, 0 overrun, 51089 ignored

0 watchdog, 27438132 multicast, 0 pause input

1827169002 packets output, 1537189058 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

#

#sh int gigabitEthernet 0/2

GigabitEthernet0/2 is up, line protocol is up

Hardware is MV64460 Internal MAC, address is 000c.cf1d.d01a (bia 000c.cf1d.d01a)

Description: ***** Internet Access Port *****

MTU 1700 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 95/255, rxload 84/255

Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set

Keepalive set (10 sec)

Full Duplex, 1000Mbps, media type is RJ45

output flow-control is XON, input flow-control is unsupported

ARP type: ARPA, ARP Timeout 03:20:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 4w4d

Input queue: 0/4096/15826/2230509 (size/max/drops/flushes); Total output drops: 17298

Queueing strategy: fifo

Output queue: 0/4096 (size/max)

30 second input rate 332999000 bits/sec, 63563 packets/sec

30 second output rate 373472000 bits/sec, 71761 packets/sec

1133386686 packets input, 849294076 bytes, 17917 no buffer

Received 37718109 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

421 input errors, 0 CRC, 0 frame, 0 overrun, 421 ignored

0 watchdog, 2554 multicast, 0 pause input

2675409271 packets output, 2343996450 bytes, 0 underruns

4 output errors, 0 collisions, 4 interface resets

0 babbles, 0 late collision, 0 deferred

4 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

#

#sh int gigabitEthernet 0/3

GigabitEthernet0/3 is up, line protocol is up

Hardware is MV64460 Internal MAC, address is 000c.cf1d.d019 (bia 000c.cf1d.d019)

Description: ***** Metro Ethernet *****

MTU 1700 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 64/255, rxload 59/255

Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set

Keepalive set (10 sec)

Full Duplex, 1000Mbps, media type is RJ45

output flow-control is XON, input flow-control is unsupported

ARP type: ARPA, ARP Timeout 03:20:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 4w4d

Input queue: 0/4096/99466/2578258 (size/max/drops/flushes); Total output drops: 45184295

Queueing strategy: fifo

Output queue: 0/4096 (size/max)

30 second input rate 234300000 bits/sec, 51018 packets/sec

30 second output rate 251936000 bits/sec, 44635 packets/sec

1015412816 packets input, 1352601046 bytes, 41095 no buffer

Received 157083858 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

296819 input errors, 413 CRC, 0 frame, 0 overrun, 296406 ignored

0 watchdog, 157681185 multicast, 0 pause input

2426528064 packets output, 508387291 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

#

#sh ver

Cisco IOS Software, 7200 Software (C7200P-SPSERVICESK9-M), Version 12.2(33)SRD2, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 20-May-09 21:32 by prod_rel_team

ROM: System Bootstrap, Version 12.4(4r)XD5, RELEASE SOFTWARE (fc1)

7206_VXR_G2 uptime is 26 weeks, 5 days, 19 hours, 17 minutes

System returned to ROM by power-on

System restarted at 15:58:49 GMT+2 Tue Aug 30 2011

System image file is "disk2:/c7200p-spservicesk9-mz.122-33.SRD2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

Cisco 7206VXR (NPE-G2) processor (revision A) with 917504K/65536K bytes of memory.

Processor board ID 29813402

MPC7448 CPU at 1666Mhz, Implementation 0, Rev 2.2

6 slot VXR midplane, Version 2.7

Last reset from power-on

PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points.

Current configuration on bus mb1 has a total of 90 bandwidth points.

This configuration is within the PCI bus capacity and is supported.

PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points.

Current configuration on bus mb2 has a total of 0 bandwidth points.

This configuration is within the PCI bus capacity and is supported.

Please refer to the following document "Cisco 7200 Series Port Adaptor

Hardware Configuration Guidelines" on Cisco.com <http://www.cisco.com>

for c7200 bandwidth points oversubscription and usage guidelines.

1 FastEthernet interface

3 Gigabit Ethernet interfaces

3 Serial interfaces

8 Channelized E1/PRI ports

1 Channelized E3 port

2045K bytes of NVRAM.

250880K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes).

65536K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

#

#sh processes cpu sorted

CPU utilization for five seconds: 75%/62%; one minute: 71%; five minutes: 69%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

246 371404700 1324881 280347 7.91% 2.23% 1.71% 0 BGP Scanner

249 834493961271325781 65 0.79% 0.94% 0.96% 0 EIGRP-IPv4: PDM

85 1989393561828373765 108 0.79% 0.84% 0.80% 0 IP Input

245 35318844 989199499 35 0.47% 0.59% 0.54% 0 EIGRP-IPv4: PDM

125 85488388 108901503 785 0.47% 0.55% 0.56% 0 IP RIB Update

133 90593692 95054980 953 0.47% 0.56% 0.56% 0 static

241 30375216 679746761 44 0.31% 0.32% 0.32% 0 EIGRP-IPv4: PDM

153 254854208 212933212 1196 0.31% 0.61% 0.68% 0 BGP Router

24 42637432 339131952 125 0.23% 0.20% 0.22% 0 ARP Input

239 30037548 233726388 128 0.23% 0.11% 0.11% 0 EIGRP-IPv4: PDM

3 164 625 262 0.15% 0.04% 0.00% 2 SSH Process

214 9704812 36008323 269 0.15% 0.07% 0.07% 0 SNMP ENGINE

48 2342552 292826 7999 0.15% 0.01% 0.00% 0 Per-minute Jobs

12 19224 16121638 1 0.07% 0.00% 0.00% 0 IPC Periodic Tim

128 5841603820724583 0 0.07% 0.08% 0.07% 0 HQF Shaper Backg

261 12606892 460359880 27 0.07% 0.07% 0.07% 0 EIGRP-IPv4: HELL

221 410440 488309363 0 0.07% 0.04% 0.05% 0 PPP manager

236 10576081928546169 0 0.07% 0.09% 0.07% 0 IP SLAs Event Pr

129 4362563820729036 0 0.07% 0.07% 0.07% 0 HQF Input Shaper

250 7143556 346717390 20 0.07% 0.05% 0.06% 0 EIGRP-IPv4: HELL

117 1054272 22516667 46 0.07% 0.06% 0.07% 0 CEF: IPv4 proces

247 838676 42421526 19 0.07% 0.00% 0.00% 0 EIGRP-IPv4: HELL

101 1316444 65438354 20 0.07% 0.00% 0.00% 0 Spanning Tree

89 14028036 117415925 119 0.07% 0.06% 0.07% 0 ADJ resolve proc

222 357576 488309506 0 0.07% 0.02% 0.02% 0 PPP Events

212 6874068 69959410 98 0.07% 0.05% 0.06% 0 IP SNMP

26 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

27 0 1 0 0.00% 0.00% 0.00% 0 ATM ASYNC PROC

25 285844 16814593 16 0.00% 0.00% 0.00% 0 ARP Background

30 0 4 0 0.00% 0.00% 0.00% 0 Entity MIB API

28 0 5 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

29 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

33 116 16995 6 0.00% 0.00% 0.00% 0 Serial Backgroun

31 5372 160564 33 0.00% 0.00% 0.00% 0 EEM ED Syslog

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

35 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify Wa

36 0 19 0 0.00% 0.00% 0.00% 0 DDR Timers

37 0 2 0 0.00% 0.00% 0.00% 0 SMART

32 784 5 156800 0.00% 0.00% 0.00% 0 PrstVbl

39 17824 16121633 1 0.00% 0.00% 0.00% 0 GraphIt

40 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

15 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager

42 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

43 2817912 4977553 566 0.00% 0.03% 0.01% 0 Net Background

23 3352 3232113 1 0.00% 0.00% 0.00% 0 ALARM_TRIGGER_SC

22 0 1 0 0.00% 0.00% 0.00% 0 Exception contro

34 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers

38 0 1 0 0.00% 0.00% 0.00% 0 RFS server proce

14 12304 16121640 0 0.00% 0.00% 0.00% 0 IPC Deferred Por

49 20 84 238 0.00% 0.00% 0.00% 0 IF-MGR control p

50 20 393 50 0.00% 0.00% 0.00% 0 IF-MGR event pro

51 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

52 0 2 0 0.00% 0.00% 0.00% 0 INTR MGR PROCESS

53 9712 5276901 1 0.00% 0.00% 0.00% 0 Transport Port A

54 51772 4807188 10 0.00% 0.00% 0.00% 0 HC Counter Timer

41 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

56 0 1 0 0.00% 0.00% 0.00% 0 POS APS Event Pr

57 0 1 0 0.00% 0.00% 0.00% 0 SONET alarm time

58 0 2 0 0.00% 0.00% 0.00% 0 Hawkeye Backgrou

59 0 1 0 0.00% 0.00% 0.00% 0 CSP Timer

60 0 2 0 0.00% 0.00% 0.00% 0 FPD Management P

44 0 2 0 0.00% 0.00% 0.00% 0 IDB Work

62 159304 919190 173 0.00% 0.00% 0.00% 0 Collection proce

63 0 2 0 0.00% 0.00% 0.00% 0 VNM DSPRM MAIN

64 0 2 0 0.00% 0.00% 0.00% 0 Flash MIB Update

65 8 6 1333 0.00% 0.00% 0.00% 0 Flash Card Oir

66 0 1 0 0.00% 0.00% 0.00% 0 CES Line Conditi

45 4968 342666 14 0.00% 0.00% 0.00% 0 Logger

68 0 1 0 0.00% 0.00% 0.00% 0 Async write proc

46 27764 16121595 1 0.00% 0.00% 0.00% 0 TTY Background

70 0 1 0 0.00% 0.00% 0.00% 0 Async write proc

#

#sh processes memory sorted

Processor Pool Total: 850287212 Used: 508707208 Free: 341580004

I/O Pool Total: 67108864 Used: 6691312 Free: 60417552

Transient Pool Total: 16777216 Used: 122356 Free: 16654860

PID TTY Allocated Freed Holding Getbufs Retbufs Process

153 0 2167646800 3258059216 369532500 52448 0 BGP Router

125 0 2566262772 823460356 78550656 0 0 IP RIB Update

0 0 51236456 7371184 39439020 0 0 *Init*

0 0 3116389520 2968229116 7423404 3661344 17420 *Dead*

5 0 358534536 332228156 4457884 133743676 135805988 Pool Manager

239 0 2924544240 2945053956 4331500 19108 719136 EIGRP-IPv4: PDM

0 0 0 0 2033364 0 0 *MallocLite*

249 0 1396961512 1353257724 1081536 146924 1013924 EIGRP-IPv4: PDM

96 0 796772 47276 710012 0 0 PPP SIP

84 0 535788 25248 526860 0 0 IP ARP Adjacency

207 0 446176 0 507300 0 0 CCPROXY_CT

105 0 580832 88200 500112 0 0 SSS Manager

222 0 499536 13776 494768 780 0 PPP Events

124 0 449952 107988 342492 0 0 IP Background

100 0 311152 188 324088 0 0 SSM connection m

133 0 820972 615892 323388 0 0 static

31 0 279992 0 289504 113400 0 EEM ED Syslog

89 0 1027472104 1153134984 265224 6600880 30557704 ADJ resolve proc

241 0 3208006648 3210118876 224048 48436 392504 EIGRP-IPv4: PDM

62 0 3442292 1015964 210036 0 0 Collection proce

1 0 37852324 37808240 182816 0 0 Chunk Manager

188 0 142428 9180 148940 0 0 EEM Server

85 0 731131288 534554412 119264 44456588 10697448 IP Input

164 0 65848 0 90972 0 0 QOS_MODULE_MAIN

117 0 90752 76 90752 0 0 CEF: IPv4 proces

43 0 436500 248 83176 65260 0 Net Background

256 0 285140 11248 82884 0 0 Tag Control

115 0 65752 0 78876 0 0 CEF background p

167 0 111976 3912 67600 0 0 Crypto ACL

82 0 49404 0 62528 0 0 IPAM Manager

24 0 3298366036 3366734816 62384 0 13954704 ARP Input

#

Message was edited by: Umit AYDINLI because i added sh tech output.

Message was edited by: Umit AYDINLI

Edison Ortiz · ‎03-05-2012

Your interfaces are reporting a high number of input and output drops for the 4week period.

Can you make sure the duplex/speed are matching at either end and also verify cabling?

Packet drops in the router can create high CPU conditions.

What other services you have enabled in the router?

Can you post your sanitize config?

Regards,

Edison

View solution in original post

Vishnu Asok · ‎03-05-2012

Hi Umit

With the information you provided, we cannot arrive at a conclusion. The CPU utilisation will depend on several factors like-

1)features configured

2)Switching path taken by packets

3)Broadcasts and multicast packets received

4)Packets direct to the router

From the outputs, I see that you have 62 % interrupts and remaining CPU is accounted by various process which is pretty normal.

CPU utilization for five seconds: 75%/62%; one minute: 71%; five minutes: 69%

I notice that you have significant traffic flowing through the router. I would say you are hitting the hardware limitation with the current flow of traffic.

As per documents NPE-G2 can handle up to 1024 Mbps traffic, but never expect more than 800 Mbps with heavy configuration.

As you add ACL's, encryption, QOS, etc - performance will decline significantly from the given numbers, unless it is a hardware-assisted platform, such as the ASR 1000, 7600 or 12000

You can compare the router performance in below link-

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

The only logical thing we can do now is to perform a CPU profiling and identify the functions that are interrupting the CPU. Once the function is identified, we can try tweaking/ optimizing the associated configuration . However in your case I doubt there is anything more we can do to reduce CPU other than redirecting some traffic.

Regards

Vishnu

View solution in original post

Edison Ortiz · ‎03-05-2012

Your interfaces are reporting a high number of input and output drops for the 4week period.

Can you make sure the duplex/speed are matching at either end and also verify cabling?

Packet drops in the router can create high CPU conditions.

What other services you have enabled in the router?

Can you post your sanitize config?

Regards,

Edison

Joseph W. Doherty · ‎03-05-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

From what you've posted, you might just be occasionally offering too much load to this router. A G2 can struggle with two gig links.

Joseph W. Doherty · ‎03-08-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

A G2 can struggle with two gig links.

BTW, reason for this statement, gig Ethernet can require up to 1.488 Mpps. So two gig links could require up to about 3 Mpps, and your three gig links could require up to about 4.5 Mpps. Since a -G2 is rated at 2 Mpps, see the reason why it can struggle?

Even the "fastest" 7200 "engine", the discontinued NSE-1, rated at 3.5 Mpps (PXF), might struggle with your three gig links. A 5 Mpps ASR would probably be the most suitable choice for your current usage.

Looking at your stats, there is some "low hanging fruit", i.e. some configuration options that might help. These include: insuring your BGP peering is using MTU, not just 576; insuring buffer usage is just "hits"; enabling compiled ACLs (if supported and assuming you're using more than a few ACLs); tuning SPD; and perhaps reducing your TCP window size to only accept 2 to 4 packets (to throttle BGP updates).

Vishnu Asok · ‎03-05-2012

Hi Umit

With the information you provided, we cannot arrive at a conclusion. The CPU utilisation will depend on several factors like-

1)features configured

2)Switching path taken by packets

3)Broadcasts and multicast packets received

4)Packets direct to the router

From the outputs, I see that you have 62 % interrupts and remaining CPU is accounted by various process which is pretty normal.

CPU utilization for five seconds: 75%/62%; one minute: 71%; five minutes: 69%

I notice that you have significant traffic flowing through the router. I would say you are hitting the hardware limitation with the current flow of traffic.

As per documents NPE-G2 can handle up to 1024 Mbps traffic, but never expect more than 800 Mbps with heavy configuration.

As you add ACL's, encryption, QOS, etc - performance will decline significantly from the given numbers, unless it is a hardware-assisted platform, such as the ASR 1000, 7600 or 12000

You can compare the router performance in below link-

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

The only logical thing we can do now is to perform a CPU profiling and identify the functions that are interrupting the CPU. Once the function is identified, we can try tweaking/ optimizing the associated configuration . However in your case I doubt there is anything more we can do to reduce CPU other than redirecting some traffic.

Regards

Vishnu