Cisco 867VAE-W-A-K9 Gradual High CPU Usage

[harri.phu]

Hi all,

I'm not quite familiar with how to troubleshoot an issue I'm having with my 867 modem at home. It will be fine after a reload but after a day or two I'll see very high CPU utilisation to the point where pings are lost and there is significant lag trying to connect via SSH - this makes it pretty much impossible to do any debugging and I end up having to restart the modem.

I've got a tunnel set up to Azure and I'm suspecting config.

The below aren't stats from when cpu usage is 100%, but uptime has been for about a day so it will probably happen soon.

Any light you can shed on this would be of great help!

Config:

Building configuration...

Current configuration : 7089 bytes
!
version 15.5
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname nvm-hq-gw01
!
boot-start-marker
boot-end-marker
!
aqm-register-fnf
!
logging buffered 51200 warnings
!
no aaa new-model
bsd-client server url https://cloudsso.cisco.com/as/token.oauth2
ppp packet throttle 100 1 5
wan mode dsl
!
!
!
ip dhcp excluded-address 10.1.1.1 10.1.1.20
ip dhcp excluded-address 10.1.1.201 10.1.1.254
!
ip dhcp pool ccp-pool
import all
network 10.1.1.0 255.255.255.0
default-router 10.1.1.254
dns-server 10.1.2.10 10.1.2.11 203.0.176.191 203.215.29.191
lease 0 2
!
!
!
ip domain name *removed*
ip name-server 203.0.178.191
ip name-server 203.215.29.191
ip cef
no ipv6 cef
!
!
flow record nbar-appmon
match ipv4 source address
match ipv4 destination address
match application name
collect interface output
collect counter bytes
collect counter packets
collect timestamp absolute first
collect timestamp absolute last
!
!
flow monitor application-mon
cache timeout active 60
record nbar-appmon
!
parameter-map type inspect global
max-incomplete low 18000
max-incomplete high 20000
nbar-classify
!
!
!
!
!
!
!
!
!
crypto pki trustpoint TP-self-signed-153958242
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-153958242
revocation-check none
rsakeypair TP-self-signed-153958242
!
!
crypto pki certificate chain TP-self-signed-153958242
certificate self-signed 01
30820229 30820192 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
30312E30 2C060355 04031325 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 31353339 35383234 32301E17 0D313630 37313130 34313234
395A170D 32303031 30313030 30303030 5A303031 2E302C06 03550403 1325494F
532D5365 6C662D53 69676E65 642D4365 72746966 69636174 652D3135 33393538
32343230 819F300D 06092A86 4886F70D 01010105 0003818D 00308189 02818100
BB34CCD7 CE6E0446 4505436D 3A5A0B91 93B03087 9AA36EC2 0B1C3384 4EFD08A2
06BC3607 438793EA 2629B4FA F16231E9 DCC7D5F2 DB72A63A 1F7BA8DE FA880AB4
F563E619 8CD465B1 0433193A 527B6234 05281694 5546A08F D90343D1 17159B78
E375EB5E 769CA13A 10D0039B E6ADBC10 03E7755A 462BFC7E 1AFDB8F9 81FE4877
02030100 01A35330 51300F06 03551D13 0101FF04 05300301 01FF301F 0603551D
23041830 16801400 81D7FE2F 83BA182E 1C47763D 8C7D5A59 A3846D30 1D060355
1D0E0416 04140081 D7FE2F83 BA182E1C 47763D8C 7D5A59A3 846D300D 06092A86
4886F70D 01010505 00038181 00807909 09CD6F6A 244D24D1 426D793C 6D3DB3CE
143EE912 E1CF28C7 71A51A75 598CE807 BB390214 03AE0BB1 53F7298E 6B066247
8519586D C9A1EC2E C70B3B3C 198D2CAF 63711C40 0EA5F84A C8DC549C 04A5E926
295357A0 7026C8F2 9429B8C6 414DE51B 4A0B4824 026FD6E7 B236C831 2AD7DC3E
A149BF26 6934EB60 1767AAF6 AE
quit
!
!
object-group service INTERNAL_UTM_SERVICE
!
object-group network local_cws_net
!
object-group network local_lan_subnets
any
!
object-group network vpn_remote_subnets
any
!
username iinet privilege 15 secret 5 *removed*
username admin privilege 15 secret 5 *removed*
!
crypto key pubkey-chain rsa
named-key realm-cisco.pub signature
key-string
30820122 300D0609 2A864886 F70D0101 01050003 82010F00 3082010A 02820101
00C19E93 A8AF124A D6CC7A24 5097A975 206BE3A2 06FBA13F 6F12CB5B 4E441F16
17E630D5 C02AC252 912BE27F 37FDD9C8 11FC7AF7 DCDD81D9 43CDABC3 6007D128
B199ABCB D34ED0F9 085FADC1 359C189E F30AF10A C0EFB624 7E0764BF 3E53053E
5B2146A9 D7A5EDE3 0298AF03 DED7A5B8 9479039D 20F30663 9AC64B93 C0112A35
FE3F0C87 89BCB7BB 994AE74C FA9E481D F65875D6 85EAF974 6D9CC8E3 F0B08B85
50437722 FFBE85B9 5E4189FF CC189CB9 69C46F9C A84DFBA5 7A0AF99E AD768C36
006CF498 079F88F8 A3B3FB1F 9FB7B3CB 5539E1D1 9693CCBB 551F78D2 892356AE
2F56D826 8918EF3C 80CA4F4D 87BFCA3B BFF668E9 689782A5 CF31CB6E B4B094D3
F3020301 0001
quit
!
!
controller VDSL 0
no cdp run
!
zone security LAN
zone security WAN
zone security VPN
zone security DMZ
!
!
crypto ikev2 proposal Azure-Proposal
encryption aes-cbc-256 aes-cbc-128 3des
integrity sha1
group 2
!
crypto ikev2 policy Azure-Policy
proposal Azure-Proposal
!
crypto ikev2 keyring Azure-Keyring
peer *removed*
address *removed*
pre-shared-key *removed*
!
!
!
crypto ikev2 profile Azure-Profile
match address local interface Dialer1
match identity remote address *removed* 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local Azure-Keyring
!
!
!
!
crypto ipsec profile Azure-IPSec-Profile
set ikev2-profile Azure-Profile
!
!
!
!
!
!
!
interface Tunnel1
ip address 169.254.0.1 255.255.255.0
ip tcp adjust-mss 1350
tunnel source Dialer1
tunnel mode ipsec ipv4
tunnel destination *removed*
tunnel protection ipsec profile Azure-IPSec-Profile
!
interface ATM0
no ip address
no atm ilmi-keepalive
!
interface ATM0.1 point-to-point
description iiNet
pvc 8/35
pppoe-client dial-pool-number 1
!
!
interface Ethernet0
no ip address
shutdown
!
interface FastEthernet0
no ip address
!
interface FastEthernet1
no ip address
!
interface FastEthernet2
no ip address
!
interface GigabitEthernet0
no ip address
!
interface GigabitEthernet1
no ip address
!
interface GigabitEthernet2
no ip address
ip tcp adjust-mss 1412
shutdown
duplex auto
speed auto
!
interface Wlan-GigabitEthernet0
description Internal switch interface connecting to the embedded AP
no ip address
!
interface Vlan1
description $ETH_LAN$
ip address 10.1.2.254 255.255.255.0 secondary
ip address 10.1.3.254 255.255.255.0 secondary
ip address 10.1.5.254 255.255.255.0 secondary
ip address 10.1.1.254 255.255.255.0
ip nat inside
ip virtual-reassembly in
ip tcp adjust-mss 1412
!
interface Dialer1
ip address negotiated
ip mtu 1452
ip nat outside
ip virtual-reassembly in
encapsulation ppp
dialer pool 1
dialer-group 1
ppp authentication chap pap callin
ppp chap hostname *removed*
ppp chap password 7 *removed*
ppp pap sent-username *removed* password 7 *removed*
ppp ipcp dns request
no cdp enable
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
!
no ip nat service sip udp port 5060
ip nat inside source list nat-list interface Dialer1 overload
ip nat inside source static tcp 10.1.2.50 80 interface Dialer1 80
ip nat inside source static tcp 10.1.2.50 443 interface Dialer1 443
ip nat inside source static tcp 10.1.2.50 3389 interface Dialer1 33890
ip route 0.0.0.0 0.0.0.0 Dialer1
ip route 10.1.253.0 255.255.255.0 Tunnel1
ip route 10.1.254.0 255.255.255.0 Tunnel1
!
ip access-list extended nat-list
permit ip object-group local_lan_subnets any
!
dialer-list 1 protocol ip permit
mac-address-table aging-time 10
!
!
!
line con 0
login local
no modem enable
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport input all
stopbits 1
line vty 0 4
privilege level 15
login local
transport input ssh
!
scheduler allocate 60000 1000
!
end

The IP Input process has shown be high CPU when modem stops responding.

nvm-hq-gw01#show processes cpu sorted
CPU utilization for five seconds: 36%/26%; one minute: 78%; five minutes: 59%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
109 7536628 467421 16123 8.61% 39.84% 28.22% 0 IP Input
117 1052 98 10734 0.31% 0.98% 0.24% 3 SSH Process
340 11156 2642805 4 0.15% 0.04% 0.06% 0 PPP manager
243 7196 1333084 5 0.15% 0.02% 0.04% 0 PERFMON ASYNC TI
329 16320 184080 88 0.07% 0.06% 0.07% 0 Per-Second Jobs
105 7388 2622607 2 0.07% 0.02% 0.05% 0 IPAM Manager
242 7724 1333086 5 0.07% 0.03% 0.05% 0 MMA DP TIMER
28 7352 34244 214 0.07% 0.00% 0.00% 0 ARP Input
203 7228 1333076 5 0.07% 0.01% 0.01% 0 MMA DB TIMER
177 804 175193 4 0.07% 0.00% 0.00% 0 CCE DP URLF cach
341 7232 2642812 2 0.07% 0.02% 0.02% 0 PPP Events
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
73 2728 425909 6 0.07% 0.00% 0.00% 0 COLLECT STAT COU
12 0 1 0 0.00% 0.00% 0.00% 0 Exception contro
11 0 1 0 0.00% 0.00% 0.00% 0 Crash writer
10 0 13 0 0.00% 0.00% 0.00% 0 WATCH_AFS
9 0 2 0 0.00% 0.00% 0.00% 0 Timers
13 84 18350 4 0.00% 0.00% 0.00% 0 IPC Event Notifi
14 4 1535 2 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
19 0 1 0 0.00% 0.00% 0.00% 0 IPC Process leve
15 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv
21 8 5254 1 0.00% 0.00% 0.00% 0 IPC Check Queue
16 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
23 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat TX Cont
17 716 88694 8 0.00% 0.00% 0.00% 0 IPC Periodic Tim
25 180 18356 9 0.00% 0.00% 0.00% 0 IPC Loadometer
8 0 1 0 0.00% 0.00% 0.00% 0 DiscardQ Backgro
27 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage
7 11708 1421268 8 0.00% 0.04% 0.05% 0 Pool Manager
18 508 88695 5 0.00% 0.00% 0.00% 0 IPC Deferred Por
20 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager
31 0 1 0 0.00% 0.00% 0.00% 0 ATM ASYNC PROC
32 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
33 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
22 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat RX Cont
35 0 2 0 0.00% 0.00% 0.00% 0 Entity MIB API
36 428 11 38909 0.00% 0.00% 0.00% 0 PrstVbl
37 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
38 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API
24 28 9203 3 0.00% 0.00% 0.00% 0 IPC Keep Alive M
40 0 2 0 0.00% 0.00% 0.00% 0 SMART
41 0 1 0 0.00% 0.00% 0.00% 0 RFS server proce
42 1512 34244 44 0.00% 0.00% 0.00% 0 ARP Snoop
26 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Deta
44 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
45 28704 5287 5429 0.00% 0.00% 0.00% 0 crypto sw pk pro
46 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client
29 820 94637 8 0.00% 0.00% 0.00% 0 ARP Background
48 984 20134 48 0.00% 0.00% 0.00% 0 Net Background
49 4 3 1333 0.00% 0.00% 0.00% 0 IDB Work
50 28 357 78 0.00% 0.00% 0.00% 0 Logger
51 1108 90781 12 0.00% 0.00% 0.00% 0 TTY Background
52 208 46013 4 0.00% 0.00% 0.00% 0 Reset button det
30 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
54 0 20 0 0.00% 0.00% 0.00% 0 IF-MGR event pro
6 261656 18719 13978 0.00% 0.21% 0.22% 0 Check heaps

This command only shows processes inside the IOS daemon.
Please use 'show processes cpu platform sorted'
to show processes from the underlying operating system.

Here's a copy of the switching stats:

nvm-hq-gw01#show interfaces switching
ATM0
Throttle count 0
Drops RP 6 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0

Protocol PPPoE
Switching path Pkts In Chars In Pkts Out Chars Out
Process 217491 118724187 797767 615553747
Cache misses 0 - - -
Fast 6686102 794008264 4977476 1303058332
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 2 158
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
Dialer1
Throttle count 0
Drops RP 1863 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 217479 111343412 807451 641988138
Cache misses 0 - - -
Fast 6670383 565780361 5016062 1141540259
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 9238 129332
Cache misses 0 - - -
Fast 0 0 4615 64610
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
Interface Ethernet0 is disabled

FastEthernet0

All statistics for this interface are zero.
FastEthernet1

All statistics for this interface are zero.
FastEthernet2

All statistics for this interface are zero.
GigabitEthernet0

All statistics for this interface are zero.
GigabitEthernet1

Protocol Spanning Tree
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 45967 2758020
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 9249 554940
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
Interface GigabitEthernet2 is disabled

Interface NVI0 is disabled

Tunnel1

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 111831 87150023 0 0
Cache misses 0 - - -
Fast 512212 235033830 20066 18789856
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
Virtual-Access1

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 217484 111329374 807482 641992471
Cache misses 0 - - -
Fast 1863 706283 5017812 1143112536
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 9247 129485
Cache misses 0 - - -
Fast 6684239 565974513 4616 64624
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
Vlan1 $ETH_LAN$
Throttle count 36
Drops RP 0 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 35885 Drops 0

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 206629 31244248 165752 98735998
Cache misses 0 - - -
Fast 5840189 1799227181 6788329 633120287
Auton/SSE 0 0 0 0

Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 35883 2152980 47626 2857560
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 3248 706211 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
Wlan-GigabitEthernet0 Internal switch interface connecting to the embedded AP

Protocol Spanning Tree
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 0 0
Cache misses 0 - - -
Fast 0 0 45967 2758020
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.

[Leo Laohoo]

109 7536628 467421 16123 8.61% 39.84% 28.22% 0 IP Input

What is your WAN speed? 

I don't see any signs of traffic shaping policy in the config.

[harri.phu]

ADSL2 and not a very good one at that.

[Leo Laohoo]

I think that's what's killing the CPU (IP Input):  The DSL interface could be dropping packets.

You might want to consider configuring Traffic Shaping.