High CPU Interrupt Rate on 7206vxr

Nidal Safadi · ‎01-18-2014

Hello Everybody;

I had a very stange problem we suffer from serveral months, searching all internet sites disccusions with no luck to get a solution, here are the details:

I'm a service provider which provide ADSL service, we have 5 LNSs "BRAS" to terminat PPP users via LAC of Upper service provider.

Each LNS type is Cisco 7206vxr-NPE-G2 ( c7200p-adventerprisek9-mz.151-4.M1.bin ) handle 1800 PPP sessions and also each LNS connected to 13 LAC via vpdn over l2tp tunnels, and each LNS handle (400 - 500 Mbps) of traffic which make total of 2.5 Gbps to Internet Router.

LNS configuration not include any additional configuration else it runs OSPF to collect connected routes and send it to Internet Router.

The problem is we suffer from High cpu utlization of Interrupts. Each LNS have CPU rate of : 90/70.

Please could any one give me any advice to solve this or it's a normal thing?

Note: ip cef enabled

i can provide you any show results

Here some of show results:

LNS#show cef drop

% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Drop Statistics

Slot Encap_fail Unresolved Unsupported No_route No_adj ChkSum_Err

RP 0 0 2562691697 90 0 12160

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

LNS#show cef not-cef-switched

% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Packets passed on to next switching layer

Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag

RP 0 0 2562929756 0 68566096 0 126945 0

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

LNS#show interfaces switching

GigabitEthernet0/1 " Interface face upper service provider "

Throttle count 0

Drops RP 176 SP 0

SPD Flushes Fast 471823 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 1924470 Drops 0

Protocol IP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 718302296 4126662098 1112298943 1451433415

Cache misses 0 - - -

Fast 3305818606 1149552345 661100592 1370347133

Auton/SSE 0 0 0 0

Protocol DEC MOP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 0 0 1738 133826

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol ARP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 4780 286800 1526 91560

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol CDP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 17399 8247126 19362 7163940

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol Other

Switching path Pkts In Chars In Pkts Out Chars Out

Process 0 0 104308 6258480

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

GigabitEthernet0/2 " Inerface connected to Internet Router "

Throttle count 0

Drops RP 959104 SP 0

SPD Flushes Fast 15007215 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 1364797 Drops 0

Protocol IP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 999401649 1612699701 712502786 2118640235

Cache misses 0 - - -

Fast 1083143063 4057627019 3036792652 3002157150

Auton/SSE 0 0 0 0

Protocol DEC MOP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 0 0 1738 133826

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol ARP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 75 4500 1254 75240

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol CDP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 19355 8245225 19354 7160980

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol Other

Switching path Pkts In Chars In Pkts Out Chars Out

Process 0 0 104314 6258840

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Some of Virtual-Access Users result:

Virtual-Access10

Throttle count 0

Drops RP 0 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 25451 Drops 0

Protocol IP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 385798 16849653 457847 635790618

Cache misses 0 - - -

Fast 681084 63626366 26634169 1192416118

Auton/SSE 0 0 0 0

Protocol Other

Switching path Pkts In Chars In Pkts Out Chars Out

Process 3 96 86610 1386434

Cache misses 0 - - -

Fast 19576850 2329991433 0 0

Auton/SSE 0 0 0 0

Virtual-Access12

Throttle count 0

Drops RP 0 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 8543 Drops 0

Protocol IP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 15848 8018299 21948 2215706

Cache misses 0 - - -

Fast 310092 46529741 22224211 1403568072

Auton/SSE 0 0 0 0

Protocol Other

Switching path Pkts In Chars In Pkts Out Chars Out

Process 4 128 85123 1362688

Cache misses 0 - - -

Fast 17113093 2030440478 0 0

Auton/SSE 0 0 0 0

Virtual-Access19

Throttle count 0

Drops RP 0 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 8123 Drops 0

Protocol IP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 1637093 139266284 2423448 3162258987

Cache misses 0 - - -

Fast 0 0 28885932 2671057232

Auton/SSE 0 0 0 0

Protocol Other

Switching path Pkts In Chars In Pkts Out Chars Out

Process 0 0 85303 1365572

Cache misses 0 - - -

Fast 20771820 2246068842 0 0

Auton/SSE 0 0 0 0

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

LNS#show ip cef switching statistics

Path Reason Drop Punt Punt2Host

RP LES Packet destined for us 0 35893991 0

RP LES TTL expired 0 0 170

RP LES Fragmentation failed, DF 389 0 0

RP LES Features 9615849 0 0

RP LES Unclassified reason 9 997284686 0

RP LES Neighbor resolution req 0 1 0

RP LES Fragmentation no pak 0 0 3441

RP LES Total 9616247 1033178678 3611

RP PAS No route 90 0 1

RP PAS Packet destined for us 2919 32696467 216

RP PAS Incomplete adjacency 21772 0 0

RP PAS Bad checksum 12161 0 0

RP PAS TTL expired 0 0 11118798

RP PAS Bad IP packet length 37 0 0

RP PAS Features 964811261 557882479 131388

RP PAS Unclassified reason 294249 997284686 0

RP PAS Neighbor resolution req 1 0 0

RP PAS Total 965142490 1587863632 11250403

All Total 974758737 2621042310 11254014

2 seconds after another show command

LNS#show ip cef switching statistics

Path Reason Drop Punt Punt2Host

RP LES Packet destined for us 0 35894734 0

RP LES TTL expired 0 0 170

RP LES Fragmentation failed, DF 389 0 0

RP LES Features 9615935 0 0

RP LES Unclassified reason 9 997291274 0

RP LES Neighbor resolution req 0 1 0

RP LES Fragmentation no pak 0 0 3442

RP LES Total 9616333 1033186009 3612

RP PAS No route 90 0 1

RP PAS Packet destined for us 2919 32696508 216

RP PAS Incomplete adjacency 21772 0 0

RP PAS Bad checksum 12161 0 0

RP PAS TTL expired 0 0 11118890

RP PAS Bad IP packet length 37 0 0

RP PAS Features 964819941 557889345 131389

RP PAS Unclassified reason 294250 997291274 0

RP PAS Neighbor resolution req 1 0 0

RP PAS Total 965151171 1587877127 11250496

All Total 974767504 2621063136 11254108

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

LNS#show ip cef switching statistics feature

IPv4 CEF input features:

Path Feature Drop Consume Punt Punt2Host Gave route

RP LES uRPF 17 0 0 0 0

RP LES CAR 74919 0 0 0 0

RP PAS Virtual Fragment 54857 6440840 0 4470 0

RP PAS Access List 117919 0 0 126642 0

RP PAS uRPF 247018 0 0 0 0

RP PAS CAR 11870139 0 0 0 0

Total 12364869 6440840 0 131112 0

IPv4 CEF output features:

Path Feature Drop Consume Punt Punt2Host New i/f

RP LES CAR 9541552 0 0 0 0

RP PAS Access List 415655 0 0 303 0

RP PAS TCP Adjust MSS 0 0 557939001 0 0

RP PAS CAR 952181909 0 0 0 0

Total 962139116 0 557939001 303 0

IPv4 CEF post-encap features:

Path Feature Drop Consume Punt Punt2Host New i/f

Total 0 0 0 0 0

IPv4 CEF for us features:

Path Feature Drop Consume Punt Punt2Host New i/f

Total 0 0 0 0 0

IPv4 CEF punt features:

Path Feature Drop Consume Punt Punt2Host New i/f

Total 0 0 0 0 0

IPv4 CEF local features:

Path Feature Drop Consume Punt Punt2Host Gave route

Total 0 0 0 0 0

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

LNS#show ip traffic

IP statistics:

Rcvd: 1731030398 total, 29323905 local destination

7 format errors, 230 checksum errors, 9755726 bad hop count

0 unknown protocol, 0 not a gateway

0 security failures, 0 bad options, 9238659 with options

Opts: 0 end, 0 nop, 0 basic security, 0 loose source route

0 timestamp, 0 extended security, 0 record route

0 stream ID, 0 strict source route, 9238659 alert, 0 cipso, 0 ump

0 other

Frags: 9717546 reassembled, 6040 timeouts, 0 couldn't reassemble

443666552 fragmented, 887330011 fragments, 778 couldn't fragment

Bcast: 90024 received, 0 sent

Mcast: 1362936 received, 685736 sent

Sent: 154123399 generated, 1231263328 forwarded

Drop: 913 encapsulation failed, 0 unresolved, 0 no adjacency

0 no route, 250302 unicast RPF, 0 forced drop

0 options denied

Drop: 0 packets with source IP address zero

Drop: 0 packets with internal loop back IP address

0 physical broadcast

Reinj: 0 in input feature path, 0 in output feature path

ICMP statistics:

Rcvd: 0 format errors, 147 checksum errors, 0 redirects, 22 unreachable

104596 echo, 2 echo reply, 0 mask requests, 0 mask replies, 0 quench

0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other

0 irdp solicitations, 0 irdp advertisements

10662 time exceeded, 0 info replies

Sent: 0 redirects, 132461 unreachable, 0 echo, 104596 echo reply

0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies

0 info reply, 188205 time exceeded, 0 parameter problem

0 irdp solicitations, 0 irdp advertisements

TCP statistics:

Rcvd: 25409 total, 3 checksum errors, 59 no port

Sent: 24035 total

BGP statistics:

Rcvd: 0 total, 0 opens, 0 notifications, 0 updates

0 keepalives, 0 route-refresh, 0 unrecognized

Sent: 0 total, 0 opens, 0 notifications, 0 updates

0 keepalives, 0 route-refresh

EIGRP-IPv4 statistics:

Rcvd: 0 total

Sent: 0 total

PIMv2 statistics: Sent/Received

Total: 0/0, 0 checksum errors, 0 format errors

Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0

Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0

Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0

Queue drops: 0

State-Refresh: 0/0

IGMP statistics: Sent/Received

Total: 0/0, Format errors: 0/0, Checksum errors: 0/0

Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0

DVMRP: 0/0, PIM: 0/0

Queue drops: 0

UDP statistics:

Rcvd: 27819278 total, 0 checksum errors, 83282 no port

Sent: 152988495 total, 0 forwarded broadcasts

OSPF statistics:

Last clearing of OSPF traffic counters never

Rcvd: 1363785 total, 0 checksum errors

110059 hello, 172 database desc, 1 link state req

1113254 link state updates, 140294 link state acks

Sent: 685982 total

110015 hello, 173 database desc, 52 link state req

285043 link state updates, 290701 link state acks

ARP statistics:

Rcvd: 30911 requests, 365 replies, 0 reverse, 0 other

Sent: 370 requests, 34367 replies (0 proxy), 0 reverse

Drop due to input queue full: 0

443666552 fragmented, 887330011 fragments and reassembled Increases Hevaily in IP Statistics

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

interface Virtual-Template1

ip unnumbered GigabitEthernet0/1

ip access-group denysomething out

no ip redirects

no ip unreachables

no ip proxy-arp

ip mtu 1492

ip virtual-reassembly in

ip verify unicast reverse-path

ip tcp adjust-mss 1436

no logging event link-status

peer default ip address pool ADSL-POOL

no ipv6 redirects

ppp authentication pap vpdn

ppp authorization vpdn

ppp accounting vpdn

ppp ipcp dns xx.xx.xx.xx xx.xx.xx.xx

ppp ipcp wins reject

end

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

Thanks !! waiting for your experience to solve with me this problem.

Regards

Nidal Safadi · ‎01-22-2014

Is there any one can help please ?

Dr.X · ‎01-22-2014

well ,

remove

ip access-group denysomething out

under ur virtual-template and tell me how much cpu will decrease ?

also ,

show config under ur interfaces such as gi0/1 gi0/2 gi0/3

regards

Nidal Safadi · ‎01-22-2014

This accesslist is just to deny anyone to access my customers CPE however i removed it and nothing happen to my CPU.

Here are the interfaces config

interface GigabitEthernet0/1

ip address yyy.yyy.yyy.yyy zzz.zzz.zzz.zzz

no ip redirects

no ip unreachables

ip flow ingress

ip virtual-reassembly in

ip route-cache same-interface

duplex auto

speed auto

media-type rj45

negotiation auto

hold-queue 4096 in

hold-queue 4096 out

end

interface GigabitEthernet0/2

description Internet

ip address 10.20.10.2 255.255.255.252

no ip redirects

no ip unreachables

ip route-cache same-interface

duplex auto

speed auto

media-type rj45

negotiation auto

hold-queue 4096 in

hold-queue 4096 out

end

interface GigabitEthernet0/3

ip address xxx.xxx.xxx.xxx 255.255.255.0

duplex auto

speed auto

media-type rj45

negotiation auto

end

First one connected to my service provider with their LACs.

Second one connected to my edge router to provide internet.

Third one to for AAA services.

Dr.X · ‎01-22-2014

hi ,

the acl on ur virtual templae must be processing more cpu , ur cpu must be decreased at least some percentage

well ,

u seem performed nat and removed it ,

anyway

plz do the following :

interface gi0/1

no ip flow ingress

no ip virtual-reassembly in

no ip route-cache same-interface

no hold-queue 4096 in

no hold-queue 4096 out

interface gi0/2

no ip route-cache same-interface

no negotiation auto

no hold-queue 4096 in

no hold-queue 4096 ou

tell me with sh process cpu

and sh process cpu sorted

regards

Nidal Safadi · ‎01-22-2014

I never used NAT is my LNS ever. all my customers have a vaild public IP.

all the above you mentioned is entered in my configuration early because of High CPU but still no changes, and know i removed them with the same result.

Just to be in the situation, my aaa send rate-limit attribut to users vi access interface to limit their download to a specific rate, these may be when the user reach his limit, the rate-limit will drop any packets exceeded the accepted rate which make all drops go to interrupt cpu level.

one more thing is i didn't see if my virtual-access interfaces are cef switching, because all traffic is cpu proccess, how can i make them to be cef switiched however it's globally enabled cef and under each interface enabled by default that's why they didn't appear in my config, however if i do ( show run all ) i see under all my interfaces cef enabled by default.

Any suggestions !!

Dr.X · ‎05-18-2014

did u fix the problem ??

was it routing issue ?? IOS bug issue ??

looking forward to hear form you.

with my best regards

Nidal Safadi · ‎01-22-2014

show proc cpu sort | ex 0.00

CPU utilization for five seconds: 95%/77%; one minute: 94%; five minutes: 93%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

109 260408828 1205848927 215 10.80% 9.85% 9.72% 0 IP Input

314 111212364 1105392453 100 4.24% 3.83% 3.78% 0 L2X Data Daemon

66 48352752 1208092 40024 1.76% 1.73% 1.74% 0 Compute load avg

139 9484452 2106286 4502 0.39% 0.42% 0.42% 0 CEF: IPv4 proces

320 3267236 46016038 71 0.31% 0.32% 0.31% 0 PPP Events

311 3783708 12534293 301 0.15% 0.16% 0.15% 0 RADIUS

94 782348 3398167 230 0.15% 0.16% 0.15% 0 ACCT Periodic Pr

319 306292 42998756 7 0.07% 0.07% 0.07% 0 PPP manager

315 2621088 3284933 797 0.07% 0.04% 0.05% 0 L2TP mgmt daemon

316 1686088 2468047 683 0.07% 0.02% 0.01% 0 L2TUN Applicatio

334 726460 3644903 199 0.07% 0.03% 0.01% 0 OSPF-1 Router

93 1104480 3735866 295 0.07% 0.07% 0.07% 0 AAA ACCT Proc

127 2138968 2226764 960 0.07% 0.04% 0.05% 0 SSS Manager

201 877896 13560475 64 0.07% 0.15% 0.15% 0 IPHC Admin

332 2837540 861892 3292 0.07% 0.08% 0.08% 0 VTEMPLATE Backgr

Dr.X · ‎01-22-2014

u have high differnee between hardware & software processing ,

95-77 is high value

how many users and how mush bw ur router loaded ?

also ,

give# sh run for ur routing table .

i mean sh run | i route

also ,

#sh ip arp

also

#show log

regards

Nidal Safadi · ‎01-22-2014

NOW the concurrent session users is 1350.

The total cuonsumed traffic in the interface of gig 0/1 is 500 Mbps

and for interface gig 0/2 is 520 Mbps

in my show arp, it just appear the devices directly connected to my LNS is contains 14 address

show log result

Jan 22 10:33:13.729: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Virtual-Access808: the fragment table has reached its maximum threshold 16

Jan 22 10:35:22.401: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Virtual-Access808: the fragment table has reached its maximum threshold 16

Jan 22 10:37:38.401: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Virtual-Access808: the fragment table has reached its maximum threshold 16

Jan 22 10:41:00.937: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Virtual-Access808: the fragment table has reached its maximum threshold 16

Jan 22 10:47:48.289: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Virtual-Access808: the fragment table has reached its maximum threshold 16

Jan 22 10:53:34.321: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Virtual-Access808: the fragment table has reached its maximum threshold 16

Jan 22 12:05:39.841: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Virtual-Access70: the fragment table has reached its maximum threshold 16

Jan 22 12:44:52.537: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Virtual-Access950: the fragment table has reached its maximum threshold 16

Jan 22 16:07:35.016: %IP_VFR-3-OVERLAP_FRAGMENTS: Virtual-Access1429: from the host xxxxxxxx destined to 168.83.77.205

Jan 22 16:31:08.592: %IP_VFR-3-OVERLAP_FRAGMENTS: Virtual-Access713: from the host yyyyyyyyyyy destined to 2.89.62.26

Jan 22 17:06:47.504: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Virtual-Access1350: the fragment table has reached its maximum threshold 1

LNS#show run | i route

no ip source-route

router ospf 1

router-id 127.1.1.1

ip route 0.0.0.0 0.0.0.0 10.20.10.1

ip route 2222222222222.0 255.255.255.0 next hop of giga0/1 interface

ip route 3333333333333.0 255.255.255.0 next hop of giga0/1 interface

Dr.X · ‎01-22-2014

hi ,

go to virtual-temp 1

and

no ip virtual-reassembly in

and tell me

i faced this issue before ,

tell me what happen when u remove it

regards

Nidal Safadi · ‎01-22-2014

the fragmnetation error stopped but still high cpu without any changes.

could you provide me your config of your LNS, my be i can match some differeneces

Dr.X · ‎01-22-2014

well ,

give me some topology , u may have routing loop

i mean that give me some info about topology

LNS----------------------------------???

regards

Dr.X · ‎01-24-2014

any new news ???

Nidal Safadi · ‎01-24-2014

Still have the problem, i don't know what to do.

about topolgy

ISP<--------------->EdgeR <------------->LNS<--------------->LAC

I don't think that i have any routing issue because in non rush hours, the cpu slow down to 20%

Could you provide me your configuration please ? may be can help