Hello.Thanks for output.TAC

me_noori · ‎11-29-2014

Hi everybody,

our cisco 7613 has about 4.5 Gbps Tx/Rx IP traffic in total, and we run ospf with other cisco cloud for routing I list in the following some our router show.what is your idea about our high cpu usage .Is it in normal range with the listed cards and modules.How can I tune the rsp720 and other SIP-200,400,600 for better performances

why our interrupt rate is high ,and one thing more the total sum of 5sec in separate rows not equal to cpu utilization for five second 50%

show proc cpu sor

CPU utilization for five seconds: 50%/46%; one minute: 54%; five minutes: 59%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
8 196795220 12640741 15568 1.51% 0.38% 0.26% 0 Check heaps

224 1048610528 4169501364 0 1.19% 1.45% 1.44% 0 IP Input

13 374006320 3155162661 0 0.23% 0.26% 0.24% 0 ARP Input

217 119862004 985030884 121 0.15% 0.32% 0.25% 0 ADJ resolve pro
c
185 537716 1825736183 0 0.07% 0.03% 0.02% 0 ACE Tunnel Task

260 1550992 2983272818 0 0.07% 0.13% 0.15% 0 Ethernet Msec T
i
305 38186336 58050485 657 0.07% 0.02% 0.00% 0 XDR mcast

34 67208 11707798 5 0.07% 0.00% 0.00% 0 IPC Loadometer

27 232776 57160812 4 0.07% 0.01% 0.00% 0 IPC Periodic Ti
m
325 17539200 92894502 188 0.07% 0.15% 0.15% 0 CEF: IPv4 proce
s
195 7406636 43782487 169 0.07% 0.00% 0.00% 0 esw_vlan_stat_p
r

show ip route summ

IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source Networks Subnets Replicates Overhead Memory (bytes)
static 1 120 0 7620 20812
connected 0 313 0 18860 53836
ospf 98 17 4892 0 589020 863984
Intra-area: 89 Inter-area: 383 External-1: 0 External-2: 0
NSSA External-1: 0 NSSA External-2: 4437
bgp 12880 0 1 0 60 172
External: 1 Internal: 0 Local: 0
ospf 410 0 269 0 16220 47344
Intra-area: 1 Inter-area: 0 External-1: 0 External-2: 268
NSSA External-1: 0 NSSA External-2: 0
internal 137 260544
Total 155 5595 0 631780 1246692

sh module

Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 0 4-subslot SPA Interface Processor-200 7600-SIP-200
2 0 4-subslot SPA Interface Processor-400 7600-SIP-400
3 24 CEF720 24 port 1000mb SFP WS-X6724-SFP
6 1 1-subslot SPA Interface Processor-600 7600-SIP-600
7 2 Route Switch Processor 720 (Active) RSP720-3CXL-GE
8 2 Route Switch Processor 720 (Cold) RSP720-3CXL-GE

show ver

System image file is "bootdisk:c7600rsp72043-adventerprisek9-mz.122-33.SRE2.bin"

1 SIP-200 controller .
1 SIP-400 controller (1 Channelized OC3/STM-1).
1 SIP-600 controller (1 TenGigabitEthernet).
2 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces
1 Ten Gigabit Ethernet interface
1 Channelized STM-1 port
1 Channelized STM-1 port

show int vlan 1

Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 2d23h
Input queue: 0/75/2886/1830 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2380531000 bits/sec, 287383 packets/sec
5 minute output rate 422133000 bits/sec, 254113 packets/sec
L2 Switched: ucast: 1200869468 pkt, 101172643240 bytes - mcast: 253599 pkt, 78
873415 bytes
L3 in Switched: ucast: 60947040633 pkt, 68919665115039 bytes - mcast: 0 pkt, 0
bytes mcast
L3 out Switched: ucast: 52594517004 pkt, 9869168832783 bytes mcast: 0 pkt, 0 b
ytes
62147839148 packets input, 69016175499764 bytes, 0 no buffer
Received 257634 broadcasts (0 IP multicasts)
0 runts, 0 giants, 15 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
53647248858 packets output, 10292998021217 bytes, 0 underruns
0 output errors, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

Vasilii Mikhailovskii · ‎12-03-2014

Hello.

Your CPU is busy with interrupts.

Please provide:

- show int summ;

- show ibc (collect it 2-4 times with interval of 1 minute);

- show ip traffic (collect it 3-4 times with interval of 1 minute).

Try to capture traffic that was punted to the RP (debug netdr).

me_noori · ‎12-03-2014

Thank you for your hints and replying

These are our show ibc in 1 min interval

Interface information:
Interface IBC0/0
5 minute rx rate 20045000 bits/sec, 30183 packets/sec
5 minute tx rate 47394000 bits/sec, 60212 packets/sec
19879272237 packets input, 4006174536193 bytes
19835355282 broadcasts received
19808585787 packets output, 3981305571968 bytes
90548 broadcasts sent
0 Bridge Packet loopback drops
19756362091 Packets CEF Switched, 1320184 Packets Fast Switched
0 Packets SLB Switched, 0 Packets CWAN Switched
Label switched pkts dropped: 0 Pkts dropped during dma: 339549
Invalid pkts dropped: 0 Pkts dropped(not cwan consumed): 0
IPSEC pkts dropped: 635184
Xconnect pkts processed: 0, dropped: 0
Xconnect pkt reflection drops: 0
Total paks copied for process level 0
Total short paks sent in route cache 2605317676
Total throttle drops 265338 Input queue drops 5831090
total spd packets classified (120217214 low, 174503 medium, 3073 high)
total spd packets dropped (339549 low, 0 medium, 0 high)
spd prio pkts allowed in due to selective throttling (0 med, 0 high)
IBC resets = 1; last at 23:52:49.004 Sat Jan 19 2013

Driver Level Counters: (Cumulative, Zeroed only at Reset)
Frames Bytes
Rx(0) 26537712 3421085217
Rx(1) 3449063135 2838813650
Tx(0) 3390340306 2016620276

Input Drop Frame Count
Rx0 = 0 Rx1 = 2488435
Per Queue Receive Errors:
FRME OFLW BUFE NOENP DISCRD DISABLE BADCOUNT
Rx0 0 0 0 0 0 0 0
Rx1 0 0 0 3633 0 0 0

Tx Errors/State:
One Collision Error = 0 More Collisions = 0
No Encap Error = 0 Deferred Error = 0
Loss Carrier Error = 0 Late Collision Error = 0
Excessive Collisions = 0 Buffer Error = 0
Tx Freeze Count = 0 Tx Intrpt Serv timeout= 1
Tx Flow State = FLOW_ON
Tx Flow Off Count = 0 Tx Flow On Count = 0

Counters collected at Idb:
Is input throttled = 0 Throttle Count = 0
Rx Resource Errors = 0 Input Drops = 2488435
Input Errors = 194243
Output Drops = 0 Giants/Runts = 0/0
Dma Mem Error = 0 Input Overrun = 0

Hash match table for multicast (in use 0, maximum 64 entries):

show ibc

Interface information:
Interface IBC0/0
5 minute rx rate 20194000 bits/sec, 30412 packets/sec
5 minute tx rate 47753000 bits/sec, 60663 packets/sec
19891125514 packets input, 4007158118761 bytes
19847185365 broadcasts received
19820407164 packets output, 3982279276274 bytes
90576 broadcasts sent
0 Bridge Packet loopback drops
19768178233 Packets CEF Switched, 1321008 Packets Fast Switched
0 Packets SLB Switched, 0 Packets CWAN Switched
Label switched pkts dropped: 0 Pkts dropped during dma: 339549
Invalid pkts dropped: 0 Pkts dropped(not cwan consumed): 0
IPSEC pkts dropped: 635574
Xconnect pkts processed: 0, dropped: 0
Xconnect pkt reflection drops: 0
Total paks copied for process level 0
Total short paks sent in route cache 2606549061
Total throttle drops 265338 Input queue drops 5831090
total spd packets classified (120252754 low, 174531 medium, 3074 high)
total spd packets dropped (339549 low, 0 medium, 0 high)
spd prio pkts allowed in due to selective throttling (0 med, 0 high)
IBC resets = 1; last at 23:52:49.004 Sat Jan 19 2013

Driver Level Counters: (Cumulative, Zeroed only at Reset)
Frames Bytes
Rx(0) 26550723 3422835145
Rx(1) 3461063605 176652699
Tx(0) 3402319442 3368513724

Input Drop Frame Count
Rx0 = 0 Rx1 = 2490155
Per Queue Receive Errors:
FRME OFLW BUFE NOENP DISCRD DISABLE BADCOUNT
Rx0 0 0 0 0 0 0 0
Rx1 0 0 0 3633 0 0 0

Tx Errors/State:
One Collision Error = 0 More Collisions = 0
No Encap Error = 0 Deferred Error = 0
Loss Carrier Error = 0 Late Collision Error = 0
Excessive Collisions = 0 Buffer Error = 0
Tx Freeze Count = 0 Tx Intrpt Serv timeout= 1
Tx Flow State = FLOW_ON
Tx Flow Off Count = 0 Tx Flow On Count = 0

Counters collected at Idb:
Is input throttled = 0 Throttle Count = 0
Rx Resource Errors = 0 Input Drops = 2490155
Input Errors = 194358
Output Drops = 0 Giants/Runts = 0/0
Dma Mem Error = 0 Input Overrun = 0

Hash match table for multicast (in use 0, maximum 64 entries):

and sorry what is your idea about total sum of 5sec in separate rows not equal to cpu utilization for five second 50%

Vasilii Mikhailovskii · ‎12-04-2014

Hello.

The output of CPU utilization shows "CPU utilization for five seconds: 50%/46%"

It means that 50% is an overall utilization, of which 46% is by interrupts.

So, summarizing CPU consumption per process, you will be able to get about 4% only (50-46 = 4).

So, you have a lot of interrupts, it probably means you have a lot of CPU punted traffic, that is why I asked for "sh ibc" and other commands.

Could you please provide:

- show int summ;
- show ibc (collect it 2-4 times with interval of 1 minute);
- show ip traffic (collect it 3-4 times with interval of 1 minute);
- show int switching (collect it 3-4 times with interval of 1 minute)

And run them during a single session; please save the outputs to txt file and attach to the discussion.

From your recent "sh ibc" I see that a lot of traffic (broadcasts?) is going to your RP and this is a problem. You need to investigate what is the traffic is why is it punted to RP. There might be some loop in your network.

To capture traffic you may use debug netdr commands.

But I would suggest you to open the case in Cisco TAC, as for investigation your need to provide "show tech" and other "private" network information that might not be shared here.

me_noori · ‎12-04-2014

Thanks a lot vasilli

please find the show commands in attached file, please help you know TAC is very slow

and please give us some help about netdr ,we have about 4.5 Gig IP traffic if we use that for capturing how change our cpu utilization is it enough for netdr

Thanks again

Vasilii Mikhailovskii · ‎12-05-2014

Hello.

Thanks for output.

TAC is responsive and could help you troubleshooting the case online.

---

But,

I had no chance to find the traffic pattern from the output.

netdr is completely safe for your box. Please try to capture traffic with netdr - I think this should clarify the traffic pattern.

Could you also provide " show interface" and "sh span-tree" output?

high CPU usage in cisco 7613 with rsp720-3cxl