03-05-2012 01:49 AM - edited 03-04-2019 03:32 PM
Hi,
I have a lot of cisco 7206 vxr deviceses. I have a high cpu problem.
I have a 7206 vxr g2. I used to use ebgp for three upstream carrier. I used to bgp full route table method. I have about 800 Mbps active traffics and behing the router about 1600 active customers.
Could you advice me how can I trouble shoot this high cpu problem?
#sh int gigabitEthernet 0/1
GigabitEthernet0/1 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is 000c.cf1d.d01b (bia 000c.cf1d.d01b)
Description: ***** GW Interface *****
Internet address is 213.144.96.1/24
MTU 1700 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 31/255, rxload 45/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, media type is RJ45
output flow-control is XON, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 03:20:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 4w4d
Input queue: 0/4096/4860/1347216 (size/max/drops/flushes); Total output drops: 13800835
Queueing strategy: fifo
Output queue: 0/4096 (size/max)
30 second input rate 177466000 bits/sec, 35236 packets/sec
30 second output rate 125457000 bits/sec, 33418 packets/sec
2016938526 packets input, 1849362236 bytes, 4794 no buffer
Received 50788313 broadcasts (15690571 IP multicasts)
0 runts, 0 giants, 0 throttles
51133 input errors, 44 CRC, 0 frame, 0 overrun, 51089 ignored
0 watchdog, 27438132 multicast, 0 pause input
1827169002 packets output, 1537189058 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
#
#sh int gigabitEthernet 0/2
GigabitEthernet0/2 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is 000c.cf1d.d01a (bia 000c.cf1d.d01a)
Description: ***** Internet Access Port *****
MTU 1700 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 95/255, rxload 84/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, media type is RJ45
output flow-control is XON, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 03:20:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 4w4d
Input queue: 0/4096/15826/2230509 (size/max/drops/flushes); Total output drops: 17298
Queueing strategy: fifo
Output queue: 0/4096 (size/max)
30 second input rate 332999000 bits/sec, 63563 packets/sec
30 second output rate 373472000 bits/sec, 71761 packets/sec
1133386686 packets input, 849294076 bytes, 17917 no buffer
Received 37718109 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
421 input errors, 0 CRC, 0 frame, 0 overrun, 421 ignored
0 watchdog, 2554 multicast, 0 pause input
2675409271 packets output, 2343996450 bytes, 0 underruns
4 output errors, 0 collisions, 4 interface resets
0 babbles, 0 late collision, 0 deferred
4 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
#
#sh int gigabitEthernet 0/3
GigabitEthernet0/3 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is 000c.cf1d.d019 (bia 000c.cf1d.d019)
Description: ***** Metro Ethernet *****
MTU 1700 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 64/255, rxload 59/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, media type is RJ45
output flow-control is XON, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 03:20:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 4w4d
Input queue: 0/4096/99466/2578258 (size/max/drops/flushes); Total output drops: 45184295
Queueing strategy: fifo
Output queue: 0/4096 (size/max)
30 second input rate 234300000 bits/sec, 51018 packets/sec
30 second output rate 251936000 bits/sec, 44635 packets/sec
1015412816 packets input, 1352601046 bytes, 41095 no buffer
Received 157083858 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
296819 input errors, 413 CRC, 0 frame, 0 overrun, 296406 ignored
0 watchdog, 157681185 multicast, 0 pause input
2426528064 packets output, 508387291 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
#
#sh ver
Cisco IOS Software, 7200 Software (C7200P-SPSERVICESK9-M), Version 12.2(33)SRD2, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Wed 20-May-09 21:32 by prod_rel_team
ROM: System Bootstrap, Version 12.4(4r)XD5, RELEASE SOFTWARE (fc1)
7206_VXR_G2 uptime is 26 weeks, 5 days, 19 hours, 17 minutes
System returned to ROM by power-on
System restarted at 15:58:49 GMT+2 Tue Aug 30 2011
System image file is "disk2:/c7200p-spservicesk9-mz.122-33.SRD2.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
Cisco 7206VXR (NPE-G2) processor (revision A) with 917504K/65536K bytes of memory.
Processor board ID 29813402
MPC7448 CPU at 1666Mhz, Implementation 0, Rev 2.2
6 slot VXR midplane, Version 2.7
Last reset from power-on
PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points.
Current configuration on bus mb1 has a total of 90 bandwidth points.
This configuration is within the PCI bus capacity and is supported.
PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points.
Current configuration on bus mb2 has a total of 0 bandwidth points.
This configuration is within the PCI bus capacity and is supported.
Please refer to the following document "Cisco 7200 Series Port Adaptor
Hardware Configuration Guidelines" on Cisco.com <http://www.cisco.com>
for c7200 bandwidth points oversubscription and usage guidelines.
1 FastEthernet interface
3 Gigabit Ethernet interfaces
3 Serial interfaces
8 Channelized E1/PRI ports
1 Channelized E3 port
2045K bytes of NVRAM.
250880K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes).
65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102
#
#sh processes cpu sorted
CPU utilization for five seconds: 75%/62%; one minute: 71%; five minutes: 69%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
246 371404700 1324881 280347 7.91% 2.23% 1.71% 0 BGP Scanner
249 834493961271325781 65 0.79% 0.94% 0.96% 0 EIGRP-IPv4: PDM
85 1989393561828373765 108 0.79% 0.84% 0.80% 0 IP Input
245 35318844 989199499 35 0.47% 0.59% 0.54% 0 EIGRP-IPv4: PDM
125 85488388 108901503 785 0.47% 0.55% 0.56% 0 IP RIB Update
133 90593692 95054980 953 0.47% 0.56% 0.56% 0 static
241 30375216 679746761 44 0.31% 0.32% 0.32% 0 EIGRP-IPv4: PDM
153 254854208 212933212 1196 0.31% 0.61% 0.68% 0 BGP Router
24 42637432 339131952 125 0.23% 0.20% 0.22% 0 ARP Input
239 30037548 233726388 128 0.23% 0.11% 0.11% 0 EIGRP-IPv4: PDM
3 164 625 262 0.15% 0.04% 0.00% 2 SSH Process
214 9704812 36008323 269 0.15% 0.07% 0.07% 0 SNMP ENGINE
48 2342552 292826 7999 0.15% 0.01% 0.00% 0 Per-minute Jobs
12 19224 16121638 1 0.07% 0.00% 0.00% 0 IPC Periodic Tim
128 5841603820724583 0 0.07% 0.08% 0.07% 0 HQF Shaper Backg
261 12606892 460359880 27 0.07% 0.07% 0.07% 0 EIGRP-IPv4: HELL
221 410440 488309363 0 0.07% 0.04% 0.05% 0 PPP manager
236 10576081928546169 0 0.07% 0.09% 0.07% 0 IP SLAs Event Pr
129 4362563820729036 0 0.07% 0.07% 0.07% 0 HQF Input Shaper
250 7143556 346717390 20 0.07% 0.05% 0.06% 0 EIGRP-IPv4: HELL
117 1054272 22516667 46 0.07% 0.06% 0.07% 0 CEF: IPv4 proces
247 838676 42421526 19 0.07% 0.00% 0.00% 0 EIGRP-IPv4: HELL
101 1316444 65438354 20 0.07% 0.00% 0.00% 0 Spanning Tree
89 14028036 117415925 119 0.07% 0.06% 0.07% 0 ADJ resolve proc
222 357576 488309506 0 0.07% 0.02% 0.02% 0 PPP Events
212 6874068 69959410 98 0.07% 0.05% 0.06% 0 IP SNMP
26 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
27 0 1 0 0.00% 0.00% 0.00% 0 ATM ASYNC PROC
25 285844 16814593 16 0.00% 0.00% 0.00% 0 ARP Background
30 0 4 0 0.00% 0.00% 0.00% 0 Entity MIB API
28 0 5 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
29 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
33 116 16995 6 0.00% 0.00% 0.00% 0 Serial Backgroun
31 5372 160564 33 0.00% 0.00% 0.00% 0 EEM ED Syslog
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
35 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify Wa
36 0 19 0 0.00% 0.00% 0.00% 0 DDR Timers
37 0 2 0 0.00% 0.00% 0.00% 0 SMART
32 784 5 156800 0.00% 0.00% 0.00% 0 PrstVbl
39 17824 16121633 1 0.00% 0.00% 0.00% 0 GraphIt
40 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
15 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager
42 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
43 2817912 4977553 566 0.00% 0.03% 0.01% 0 Net Background
23 3352 3232113 1 0.00% 0.00% 0.00% 0 ALARM_TRIGGER_SC
22 0 1 0 0.00% 0.00% 0.00% 0 Exception contro
34 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers
38 0 1 0 0.00% 0.00% 0.00% 0 RFS server proce
14 12304 16121640 0 0.00% 0.00% 0.00% 0 IPC Deferred Por
49 20 84 238 0.00% 0.00% 0.00% 0 IF-MGR control p
50 20 393 50 0.00% 0.00% 0.00% 0 IF-MGR event pro
51 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest
52 0 2 0 0.00% 0.00% 0.00% 0 INTR MGR PROCESS
53 9712 5276901 1 0.00% 0.00% 0.00% 0 Transport Port A
54 51772 4807188 10 0.00% 0.00% 0.00% 0 HC Counter Timer
41 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
56 0 1 0 0.00% 0.00% 0.00% 0 POS APS Event Pr
57 0 1 0 0.00% 0.00% 0.00% 0 SONET alarm time
58 0 2 0 0.00% 0.00% 0.00% 0 Hawkeye Backgrou
59 0 1 0 0.00% 0.00% 0.00% 0 CSP Timer
60 0 2 0 0.00% 0.00% 0.00% 0 FPD Management P
44 0 2 0 0.00% 0.00% 0.00% 0 IDB Work
62 159304 919190 173 0.00% 0.00% 0.00% 0 Collection proce
63 0 2 0 0.00% 0.00% 0.00% 0 VNM DSPRM MAIN
64 0 2 0 0.00% 0.00% 0.00% 0 Flash MIB Update
65 8 6 1333 0.00% 0.00% 0.00% 0 Flash Card Oir
66 0 1 0 0.00% 0.00% 0.00% 0 CES Line Conditi
45 4968 342666 14 0.00% 0.00% 0.00% 0 Logger
68 0 1 0 0.00% 0.00% 0.00% 0 Async write proc
46 27764 16121595 1 0.00% 0.00% 0.00% 0 TTY Background
70 0 1 0 0.00% 0.00% 0.00% 0 Async write proc
#
#sh processes memory sorted
Processor Pool Total: 850287212 Used: 508707208 Free: 341580004
I/O Pool Total: 67108864 Used: 6691312 Free: 60417552
Transient Pool Total: 16777216 Used: 122356 Free: 16654860
PID TTY Allocated Freed Holding Getbufs Retbufs Process
153 0 2167646800 3258059216 369532500 52448 0 BGP Router
125 0 2566262772 823460356 78550656 0 0 IP RIB Update
0 0 51236456 7371184 39439020 0 0 *Init*
0 0 3116389520 2968229116 7423404 3661344 17420 *Dead*
5 0 358534536 332228156 4457884 133743676 135805988 Pool Manager
239 0 2924544240 2945053956 4331500 19108 719136 EIGRP-IPv4: PDM
0 0 0 0 2033364 0 0 *MallocLite*
249 0 1396961512 1353257724 1081536 146924 1013924 EIGRP-IPv4: PDM
96 0 796772 47276 710012 0 0 PPP SIP
84 0 535788 25248 526860 0 0 IP ARP Adjacency
207 0 446176 0 507300 0 0 CCPROXY_CT
105 0 580832 88200 500112 0 0 SSS Manager
222 0 499536 13776 494768 780 0 PPP Events
124 0 449952 107988 342492 0 0 IP Background
100 0 311152 188 324088 0 0 SSM connection m
133 0 820972 615892 323388 0 0 static
31 0 279992 0 289504 113400 0 EEM ED Syslog
89 0 1027472104 1153134984 265224 6600880 30557704 ADJ resolve proc
241 0 3208006648 3210118876 224048 48436 392504 EIGRP-IPv4: PDM
62 0 3442292 1015964 210036 0 0 Collection proce
1 0 37852324 37808240 182816 0 0 Chunk Manager
188 0 142428 9180 148940 0 0 EEM Server
85 0 731131288 534554412 119264 44456588 10697448 IP Input
164 0 65848 0 90972 0 0 QOS_MODULE_MAIN
117 0 90752 76 90752 0 0 CEF: IPv4 proces
43 0 436500 248 83176 65260 0 Net Background
256 0 285140 11248 82884 0 0 Tag Control
115 0 65752 0 78876 0 0 CEF background p
167 0 111976 3912 67600 0 0 Crypto ACL
82 0 49404 0 62528 0 0 IPAM Manager
24 0 3298366036 3366734816 62384 0 13954704 ARP Input
#
Message was edited by: Umit AYDINLI because i added sh tech output.
Message was edited by: Umit AYDINLI
Solved! Go to Solution.
03-05-2012 08:06 AM
Your interfaces are reporting a high number of input and output drops for the 4week period.
Can you make sure the duplex/speed are matching at either end and also verify cabling?
Packet drops in the router can create high CPU conditions.
What other services you have enabled in the router?
Can you post your sanitize config?
Regards,
Edison
03-05-2012 01:38 PM
Hi Umit
With the information you provided, we cannot arrive at a conclusion. The CPU utilisation will depend on several factors like-
1)features configured
2)Switching path taken by packets
3)Broadcasts and multicast packets received
4)Packets direct to the router
From the outputs, I see that you have 62 % interrupts and remaining CPU is accounted by various process which is pretty normal.
CPU utilization for five seconds: 75%/62%; one minute: 71%; five minutes: 69%
I notice that you have significant traffic flowing through the router. I would say you are hitting the hardware limitation with the current flow of traffic.
As per documents NPE-G2 can handle up to 1024 Mbps traffic, but never expect more than 800 Mbps with heavy configuration.
As you add ACL's, encryption, QOS, etc - performance will decline significantly from the given numbers, unless it is a hardware-assisted platform, such as the ASR 1000, 7600 or 12000
You can compare the router performance in below link-
http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf
The only logical thing we can do now is to perform a CPU profiling and identify the functions that are interrupting the CPU. Once the function is identified, we can try tweaking/ optimizing the associated configuration . However in your case I doubt there is anything more we can do to reduce CPU other than redirecting some traffic.
Regards
Vishnu
03-05-2012 08:06 AM
Your interfaces are reporting a high number of input and output drops for the 4week period.
Can you make sure the duplex/speed are matching at either end and also verify cabling?
Packet drops in the router can create high CPU conditions.
What other services you have enabled in the router?
Can you post your sanitize config?
Regards,
Edison
03-05-2012 01:06 PM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
From what you've posted, you might just be occasionally offering too much load to this router. A G2 can struggle with two gig links.
03-08-2012 06:13 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
A G2 can struggle with two gig links.
BTW, reason for this statement, gig Ethernet can require up to 1.488 Mpps. So two gig links could require up to about 3 Mpps, and your three gig links could require up to about 4.5 Mpps. Since a -G2 is rated at 2 Mpps, see the reason why it can struggle?
Even the "fastest" 7200 "engine", the discontinued NSE-1, rated at 3.5 Mpps (PXF), might struggle with your three gig links. A 5 Mpps ASR would probably be the most suitable choice for your current usage.
Looking at your stats, there is some "low hanging fruit", i.e. some configuration options that might help. These include: insuring your BGP peering is using MTU, not just 576; insuring buffer usage is just "hits"; enabling compiled ACLs (if supported and assuming you're using more than a few ACLs); tuning SPD; and perhaps reducing your TCP window size to only accept 2 to 4 packets (to throttle BGP updates).
03-05-2012 01:38 PM
Hi Umit
With the information you provided, we cannot arrive at a conclusion. The CPU utilisation will depend on several factors like-
1)features configured
2)Switching path taken by packets
3)Broadcasts and multicast packets received
4)Packets direct to the router
From the outputs, I see that you have 62 % interrupts and remaining CPU is accounted by various process which is pretty normal.
CPU utilization for five seconds: 75%/62%; one minute: 71%; five minutes: 69%
I notice that you have significant traffic flowing through the router. I would say you are hitting the hardware limitation with the current flow of traffic.
As per documents NPE-G2 can handle up to 1024 Mbps traffic, but never expect more than 800 Mbps with heavy configuration.
As you add ACL's, encryption, QOS, etc - performance will decline significantly from the given numbers, unless it is a hardware-assisted platform, such as the ASR 1000, 7600 or 12000
You can compare the router performance in below link-
http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf
The only logical thing we can do now is to perform a CPU profiling and identify the functions that are interrupting the CPU. Once the function is identified, we can try tweaking/ optimizing the associated configuration . However in your case I doubt there is anything more we can do to reduce CPU other than redirecting some traffic.
Regards
Vishnu
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide