Extreme and constant 2000-3000 ms Latency through ISR C1111

[TheGoob]

Hello

So I have DSL 40/5 Service and for the past 2 years I have had no issues. As of this past 3 weeks I have these insane lag spikes. It could stay normal for 30 minutes or 5 hours, then it happens. My 30ms goes to 2000-3000 ms. It either will not go back down, or will after a while. I have removed every device except the ISRC1111 and the PC plugged into it, same thing. Currently my DSL Modem is in Bridge mode and ISR is doing PPPoE.

To test it, I eliminated everything Cisco and simply used my DSL Modem in PPPoE and 2 days went by 0 issues and my continuous ping stayed 30-60 ms. With that said, when using my ISR C1111 as PPPoE and the lag starts, if I reboot the ISR it all goes back to normal...for 30 mins maybe or several hours.

I really do not know how to troubleshoot this, especially when all in same exact config has been fine. Also, even doing the traceroute and ping ON the ISR, same madness.

What all would be needed from me to help in helping me figure it out?

[MHM Cisco World]

Try reboot ISR after wr config 

MHM

[TheGoob]

I will do this in a bit when I get the chance to.. Is this a common thing or? I am just curious..

[MHM Cisco World]

Yes friend 

MHM

[liviu.gheorghe]

Hello @TheGoob ,

before rebooting the device, can you collect the following outputs into a file and attaching it here:

term len 0

show proc cpu

show logg

show ip nat translations

[TheGoob]

I am sorry, I had done the write and reboot prior to reading this. Is it still relevant now to continue to give you those logs? 

[liviu.gheorghe]

No problem. If the observed delays are not seen after reload, I would say the logs are not relevant.

If the delays come back, then yes, please send the logs.

[TheGoob]

Roger that, thank you.

Not sure if running a continuous Ping via Windows shell is good or bad, but it seems to maintain 40-50ms Ping since 'write' and seems to maintain.

So, was this a bug or a known issue?

[MHM Cisco World]

It not bug 

Why I am suggest immediately reboot 

1- show version 

Give you how long this device is run' if it long time then there is some process or memory is stuck and need reboot to clean it

2- I follow your previous posts (many post) you try many config' this some times make router stuck in one point' so clean it is the effect solution. 

MHM

[TheGoob]

Hey

Well, it has been months since I have made any changes, so I found it odd this to happen all of a sudden. Also, I have rebooted this ISR many times during this issue. Either way, I will do those logs as requested if/when it happens again.

[TheGoob]

So it seems to have done it again. A lot of stuff here, but here we go. Also;

Version 17.14.1a

And here is my current running-config that as I said has not changed in a few months and this 2000-3000 5+ times a day has only started approx 2 weeks ago. With that said, being I work for the ISP, I also changed the ports/trees for both DSL Pairs in the dslam and also when using JUST my CenturyLink modem, no issues.

Current configuration : 15108 bytes
!
! Last configuration change at 02:32:13 UTC Tue Jul 2 2024 by admin
!
version 17.14
service timestamps debug datetime msec
service timestamps log datetime msec
service call-home
platform qfp utilization monitor load 80
platform hardware throughput crypto 50000
!
hostname HoM
!
boot-start-marker
boot system bootflash:c1100-universalk9.17.14.01a.SPA1.bin
boot-end-marker
!
!
no logging monitor
no aaa new-model
!
ip vrf mgmt
!
ip name-server 205.171.3.65 205.171.2.65
no ip domain lookup
!
!
!
!
!
!
!
!
!
no ip igmp snooping
login on-success log
!
!
!
!
!
!
!
subscriber templating
!
!
!
!
!
vtp domain ''
vtp mode transparent
!
!
!
crypto pki trustpoint TP-self-signed-4284067838
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-4284067838
 revocation-check none
 rsakeypair TP-self-signed-4284067838
 hash sha256
!
crypto pki trustpoint SLA-TrustPoint
 enrollment terminal
 revocation-check crl
 hash sha256
!
!
crypto pki certificate chain TP-self-signed-4284067838
 certificate self-signed 01
  30820330 30820218 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
  31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
  69666963 6174652D 34323834 30363738 3338301E 170D3234 30323037 30303033
  34305A17 0D333430 32303630 30303334 305A3031 312F302D 06035504 03132649
  4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D34 32383430
  36373833 38308201 22300D06 092A8648 86F70D01 01010500 0382010F 00308201
  0A028201 0100CF63 E76384AF 6078E295 B087349B E465A89A B84A8E90 D13E52C5
  CB28BEF5 39387B19 1036EE98 89053B3D D42D6EB3 C5F305ED 9B2FD78A C699EA02
  3FE0C2F1 23F4A538 6278551D 3717D703 13024BB1 3D9BD85F 18310A3C 83F38191
  EA11D0D6 E35C16E7 F21E507D 2A94276A 8310E595 C88EB804 05166E4A 251A654B
  82A77BF3 D6AE009A 57B0783A 90D525D3 F6DA5080 7A05528B 1C4455C3 EFFFFBBD
  55859475 D26FCD7C 04F305EB 19733ED2 3FABFF22 5549BD82 2FFF0C8E BD81F2F8
  13615860 BB6EB874 FBBBD392 C0F3EAB8 8CF66214 34354F70 69A52D4F 922DE35E
  8964E54D C946A7E6 142E9C41 0458E6C3 FD6A8FCA A0EBE66B 87FFD40F 06DA3EC0
  CC4B739F BC410203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF
  301F0603 551D2304 18301680 14D5EFD5 A40B0A02 5F830483 14D21A7C A9759BDD
  04301D06 03551D0E 04160414 D5EFD5A4 0B0A025F 83048314 D21A7CA9 759BDD04
  300D0609 2A864886 F70D0101 05050003 82010100 036BBEA4 BDEDE57A 0FD35041
  B30A2394 B79A8A01 2C87EBD4 D9A80DB7 E571FDD7 4275FDA1 55278B72 EF3236AC
  2FC6CDB5 22E67299 6079B347 E8E8F454 48AC7032 312AAC4E 02D415DC DB4D5D91
  C5490AE2 F653B0C4 A32E6369 734DBF79 98263F72 5B5F534E 06AB0049 FAC1D563
  763CB160 74093ACF 549423BB 0F5B5A6B 2B3C0802 E7C83861 ACE6E040 24A3D259
  55BCA7EC F446157C 6A6B270C EB91874B 41A4A2E9 F5C9A5AF 39E34112 EEBFB1C7
  BE0A215B 4586E7ED 20496190 A93FE5E1 63EFA300 B74DED30 E159573C B429A790
  9A2E9F1C E1A2A852 C9DC74C6 935D878A 7785C339 EEA6D219 172B13EE DB79986E
  C98E60B6 7899E8BA 3191ABE3 ED52432E 264B0F12
        quit
crypto pki certificate chain SLA-TrustPoint
 certificate ca 01
  30820321 30820209 A0030201 02020101 300D0609 2A864886 F70D0101 0B050030
  32310E30 0C060355 040A1305 43697363 6F312030 1E060355 04031317 43697363
  6F204C69 63656E73 696E6720 526F6F74 20434130 1E170D31 33303533 30313934
  3834375A 170D3338 30353330 31393438 34375A30 32310E30 0C060355 040A1305
  43697363 6F312030 1E060355 04031317 43697363 6F204C69 63656E73 696E6720
  526F6F74 20434130 82012230 0D06092A 864886F7 0D010101 05000382 010F0030
  82010A02 82010100 A6BCBD96 131E05F7 145EA72C 2CD686E6 17222EA1 F1EFF64D
  CBB4C798 212AA147 C655D8D7 9471380D 8711441E 1AAF071A 9CAE6388 8A38E520
  1C394D78 462EF239 C659F715 B98C0A59 5BBB5CBD 0CFEBEA3 700A8BF7 D8F256EE
  4AA4E80D DB6FD1C9 60B1FD18 FFC69C96 6FA68957 A2617DE7 104FDC5F EA2956AC
  7390A3EB 2B5436AD C847A2C5 DAB553EB 69A9A535 58E9F3E3 C0BD23CF 58BD7188
  68E69491 20F320E7 948E71D7 AE3BCC84 F10684C7 4BC8E00F 539BA42B 42C68BB7
  C7479096 B4CB2D62 EA2F505D C7B062A4 6811D95B E8250FC4 5D5D5FB8 8F27D191
  C55F0D76 61F9A4CD 3D992327 A8BB03BD 4E6D7069 7CBADF8B DF5F4368 95135E44
  DFC7C6CF 04DD7FD1 02030100 01A34230 40300E06 03551D0F 0101FF04 04030201
  06300F06 03551D13 0101FF04 05300301 01FF301D 0603551D 0E041604 1449DC85
  4B3D31E5 1B3E6A17 606AF333 3D3B4C73 E8300D06 092A8648 86F70D01 010B0500
  03820101 00507F24 D3932A66 86025D9F E838AE5C 6D4DF6B0 49631C78 240DA905
  604EDCDE FF4FED2B 77FC460E CD636FDB DD44681E 3A5673AB 9093D3B1 6C9E3D8B
  D98987BF E40CBD9E 1AECA0C2 2189BB5C 8FA85686 CD98B646 5575B146 8DFC66A8
  467A3DF4 4D565700 6ADF0F0D CF835015 3C04FF7C 21E878AC 11BA9CD2 55A9232C
  7CA7B7E6 C1AF74F6 152E99B7 B1FCF9BB E973DE7F 5BDDEB86 C71E3B49 1765308B
  5FB0DA06 B92AFE7F 494E8A9E 07B85737 F3A58BE1 1A48A229 C37C1E69 39F08678
  80DDCD16 D6BACECA EEBC7CF9 8428787B 35202CDC 60E4616A B623CDBD 230E3AFB
  418616A9 4093E049 4D10AB75 27E86F73 932E35B5 8862FDAE 0275156F 719BB2F0
  D697DF7F 28
        quit
!
crypto pki certificate pool
 cabundle nvram:ios_core.p7b
!
!
diagnostic bootup level minimal
!
no license feature hseck9
license udi pid C1111-8PLTEEAWB sn FGL223493AJ
license smart url https://smartreceiver.cisco.com/licservice/license
license smart url smart https://smartreceiver.cisco.com/licservice/license
license smart transport smart
license smart usage interval 365
memory free low-watermark processor 71826
!
spanning-tree extend system-id
!
enable secret 9 $9$b/g5KPM9Y12dQU$crUmYT6b1Kd47wyvwsA8UgtHlwfVZ6GW21mtMTDrrG6
enable password [password]
!
username admin privilege 15 password 0 [password]
!
redundancy
 mode none
!
!
!
!
!
controller Cellular 0/2/0
!
!
vlan internal allocation policy ascending
!
vlan 8-9
!
vlan 10
 name DMZ
!
!
class-map type inspect match-all INSIDE-TO-OUTSIDE-CLASS
 match access-group name INSIDE-TO-OUTSIDE
class-map type inspect match-all OUTSIDE-TO-INSIDE-CLASS
 match access-group name OUTSIDE-TO-INSIDE
class-map type inspect match-all INSIDE-TO-DMZ-CLASS
 match access-group name INSIDE-TO-DMZ
class-map type inspect match-all DMZ-TO-INSIDE-CLASS
 match access-group name DMZ-TO-INSIDE
class-map type inspect match-all OUTSIDE-TO-DMZ-CLASS
 match access-group name OUTSIDE-TO-DMZ
class-map type inspect match-all DMZ-TO-OUTSIDE-CLASS
 match access-group name DMZ-TO-OUTSIDE
!
policy-map type inspect INSIDE-TO-DMZ-POLICY
 class type inspect INSIDE-TO-DMZ-CLASS
  inspect
 class class-default
  drop log
policy-map type inspect INSIDE-TO-OUTSIDE-POLICY
 class type inspect INSIDE-TO-OUTSIDE-CLASS
  inspect
 class class-default
  drop log
policy-map type inspect DMZ-TO-OUTSIDE-POLICY
 class type inspect DMZ-TO-OUTSIDE-CLASS
  inspect
 class class-default
  drop log
policy-map type inspect OUTSIDE-TO-INSIDE-POLICY
 class type inspect OUTSIDE-TO-INSIDE-CLASS
  inspect
 class class-default
  drop log
policy-map type inspect OUTSIDE-TO-DMZ-POLICY
 class type inspect OUTSIDE-TO-DMZ-CLASS
  inspect
 class class-default
  drop log
policy-map type inspect DMZ-TO-INSIDE-POLICY
 class type inspect DMZ-TO-INSIDE-CLASS
  inspect
 class class-default
  drop log
!
zone security INSIDE
zone security OUTSIDE
zone security DMZ
zone-pair security DMZ-TO-IN source DMZ destination INSIDE
 service-policy type inspect DMZ-TO-INSIDE-POLICY
zone-pair security DMZ-TO-OUT source DMZ destination OUTSIDE
 service-policy type inspect DMZ-TO-OUTSIDE-POLICY
zone-pair security IN-TO-DMZ source INSIDE destination DMZ
 service-policy type inspect INSIDE-TO-DMZ-POLICY
zone-pair security IN-TO-OUT source INSIDE destination OUTSIDE
 service-policy type inspect INSIDE-TO-OUTSIDE-POLICY
zone-pair security OUT-TO-DMZ source OUTSIDE destination DMZ
 service-policy type inspect OUTSIDE-TO-DMZ-POLICY
zone-pair security OUT-TO-IN source OUTSIDE destination INSIDE
 service-policy type inspect OUTSIDE-TO-INSIDE-POLICY
!
!
interface GigabitEthernet0/0/0
 description WAN
 no ip address
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip mtu 1460
 ip tcp adjust-mss 1412
 negotiation auto
 no cdp enable
 pppoe enable group global
 pppoe-client dial-pool-number 1
!
interface GigabitEthernet0/0/1
 no ip address
 negotiation auto
!
interface GigabitEthernet0/1/0
 switchport mode access
 shutdown
!
interface GigabitEthernet0/1/1
 shutdown
!
interface GigabitEthernet0/1/2
 shutdown
!
interface GigabitEthernet0/1/3
 shutdown
!
interface GigabitEthernet0/1/4
 description ceyea
 switchport access vlan 10
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/1/5
 description FPR-WAN
 switchport access vlan 8
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/1/6
 description Management
 switchport
 switchport access vlan 9
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/1/7
 switchport
 shutdown
!
interface Wlan-GigabitEthernet0/1/8
 shutdown
!
interface Cellular0/2/0
 no ip address
 shutdown
!
interface Cellular0/2/1
 no ip address
 shutdown
!
interface Vlan1
 description ISR default LAN
 ip address 192.168.20.1 255.255.255.0
 ip nat inside
 no ip virtual-reassembly
!
interface Vlan8
 description Link _To_FPR
 ip address 172.16.1.1 255.255.255.0
 ip nat inside
 zone-member security INSIDE
!
interface Vlan9
 description management
 ip vrf forwarding mgmt
 ip address 10.0.0.1 255.255.255.0
!
interface Vlan10
 description DMZ
 ip address 192.168.10.1 255.255.255.0
 ip nat inside
 zone-member security DMZ
!
interface Dialer1
 mtu 1492
 ip address negotiated
 no ip redirects
 ip mtu 1460
 ip nat outside
 zone-member security OUTSIDE
 encapsulation ppp
 ip tcp adjust-mss 1412
 dialer pool 1
 dialer idle-timeout 0
 dialer persistent
 dialer-group 1
 no cdp enable
 ppp authentication chap pap callin
 ppp chap hostname [username]
 ppp chap password 0 [password]
 ppp pap sent-username [username] password 0 [password]
 ppp ipcp dns request
 ppp ipcp route default
!
ip forward-protocol nd
no ip http server
ip http authentication local
ip http secure-server
ip http secure-trustpoint TP-self-signed-4284067838
ip http timeout-policy idle 600 life 180 requests 25
!
ip nat translation tcp-timeout 600
ip nat translation udp-timeout 300
ip nat translation syn-timeout 5
ip nat translation dns-timeout 10
ip nat translation icmp-timeout 30
ip nat translation max-entries 200000
ip nat pool 177 207.108.121.177 207.108.121.177 prefix-length 30
ip nat pool 178 207.108.121.178 207.108.121.178 prefix-length 30
ip nat pool 179 207.108.121.179 207.108.121.179 prefix-length 30
ip nat pool 182 207.108.121.182 207.108.121.182 prefix-length 30
ip nat pool 181 207.108.121.181 207.108.121.181 prefix-length 30
ip nat pool 180 207.108.121.180 207.108.121.180 prefix-length 30
ip nat pool DMZ_OUT 207.108.121.179 207.108.121.179 netmask 255.255.255.252
ip nat inside source static tcp 192.168.10.179 25 207.108.121.179 25 extendable
ip nat inside source static tcp 192.168.10.179 80 207.108.121.179 80 extendable
ip nat inside source static tcp 192.168.10.179 993 207.108.121.179 993 extendable
ip nat inside source static tcp 192.168.1.180 25 207.108.121.180 25 extendable
ip nat inside source static tcp 192.168.1.180 80 207.108.121.180 80 extendable
ip nat inside source static tcp 192.168.1.180 993 207.108.121.180 993 extendable
ip nat inside source static tcp 192.168.1.180 2280 207.108.121.180 2280 extendable
ip nat inside source static tcp 192.168.2.181 80 207.108.121.181 80 extendable
ip nat inside source static tcp 192.168.2.181 443 207.108.121.181 443 extendable
ip nat inside source list 1 interface Dialer1 overload
ip nat inside source list 4 pool 179 overload
ip nat inside source list 5 pool 178 overload
ip nat inside source list 6 pool 182 overload
ip nat inside source list 7 pool 177 overload
ip nat inside source list 8 pool 181 overload
ip nat inside source list 9 pool 180 overload
ip nat inside source list DMZ-TO-OUTSIDE pool DMZ_OUT overload
ip route 0.0.0.0 0.0.0.0 Dialer1
ip route 192.168.1.0 255.255.255.0 172.16.1.2
ip route 192.168.2.0 255.255.255.0 172.16.1.2
ip route 192.168.3.0 255.255.255.0 172.16.1.2
ip route 192.168.4.0 255.255.255.0 172.16.1.2
ip route 192.168.5.0 255.255.255.0 172.16.1.2
ip route 192.168.6.0 255.255.255.0 172.16.1.2
ip route vrf mgmt 192.168.5.0 255.255.255.0 10.0.0.3
ip ssh bulk-mode 131072
!
ip access-list extended DMZ-TO-INSIDE
 10 deny tcp host 192.168.10.179 host 192.168.2.181 eq 22
 21 permit tcp host 192.168.10.179 host 192.168.2.181 eq 445
ip access-list extended DMZ-TO-OUTSIDE
 10 permit ip 192.168.10.0 0.0.0.255 any
ip access-list extended INSIDE-TO-DMZ
 10 permit tcp 192.168.0.0 0.0.255.255 host 192.168.10.179 eq smtp
 11 permit tcp 192.168.0.0 0.0.255.255 host 192.168.10.179 eq 993
 20 permit tcp 192.168.0.0 0.0.255.255 host 192.168.10.179 eq 22
ip access-list extended INSIDE-TO-OUTSIDE
 10 permit ip 192.168.1.0 0.0.0.255 any
 20 permit ip 192.168.2.0 0.0.0.255 any
 30 permit ip 192.168.3.0 0.0.0.255 any
 40 permit ip 192.168.4.0 0.0.0.255 any
 50 permit ip 192.168.5.0 0.0.0.255 any
 60 permit ip 192.168.6.0 0.0.0.255 any
 70 permit udp host 172.16.2.2 any eq domain
 71 permit udp host 172.16.2.2 any eq ntp
ip access-list extended OUTSIDE-TO-DMZ
 10 permit tcp any host 192.168.10.179 eq smtp
 20 permit tcp any host 192.168.10.179 eq 22
 30 permit tcp any host 192.168.10.179 eq 993
 40 deny tcp any host 192.168.10.179 eq www
ip access-list extended OUTSIDE-TO-INSIDE
 10 permit icmp any 192.168.0.0 0.0.255.255
 11 deny ip 10.0.0.0 0.255.255.255 any
 12 deny ip 127.0.0.0 0.255.255.255 any
 13 deny ip 172.16.0.0 0.15.255.255 any
 14 deny ip 192.168.0.0 0.0.255.255 any
 15 deny ip 224.0.0.0 31.255.255.255 any
 16 deny ip 192.0.2.0 0.0.0.255 any
 17 deny ip host 0.0.0.0 any
 18 deny tcp any any eq 22 log
 19 deny tcp any any eq telnet log
 20 deny ip 85.209.0.0 0.0.255.255 any
 30 permit tcp any host 192.168.1.180 eq 2280
 40 permit tcp any host 192.168.2.181 eq 443
 50 permit tcp any host 192.168.2.181 eq www
 58 permit tcp any host 192.168.1.180 eq www
 59 permit tcp any host 192.168.1.180 eq 993
 60 permit tcp any host 192.168.1.180 eq smtp
!
ip access-list standard 1
 20 permit 172.16.1.0 0.0.0.255
ip access-list standard 4
 10 permit 192.168.3.0 0.0.0.255
ip access-list standard 5
 10 permit 192.168.4.0 0.0.0.255
ip access-list standard 6
 10 permit 192.168.5.0 0.0.0.255
ip access-list standard 7
 10 permit 192.168.6.0 0.0.0.255
ip access-list standard 8
 10 permit 192.168.2.0 0.0.0.255
ip access-list standard 9
 10 permit 192.168.1.0 0.0.0.255
dialer-list 1 protocol ip permit
!
snmp-server community public RO
!
!
control-plane
!
!
line con 0
 transport input none
 stopbits 1
line vty 0 4
 password [password]
 login
 length 0
 transport input ssh
line vty 5 30
 login
 transport input ssh
!
call-home
 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
 ! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
 contact-email-addr sch-smart-licensing@cisco.com
 profile "CiscoTAC-1"
  active
  destination transport-method http
!
!
!
!
!
!
end

[liviu.gheorghe]

Hello, I see you are running a newer IOS version than in April when it was 17.13.01a.  When did you upgrade the IOS to 17.14.01a?

[TheGoob]

I’ll answer this one first as it is easiest…. I normally do not upgrade the OS. I upgraded to this one last week thinking maybe a newer OS would fix it. 

[MHM Cisco World]

 654      181852      130321       1395  0.00%  0.00%  0.00%   0 SNMP ENGINE <<- this issue 

SNMP is eating your CPU, disable SNMP and see if latency is normal or not  
SNMP in all network device if it not config correctly can led to high CPU utilize and memory issue 

MHM

[liviu.gheorghe]

I'm not sure this is the culprit. Ok, the SNMP ENGINE process has been invoked a lot of times, but the CPU overhead generated by this process is 0.00%

@TheGoob , do you have in your network a software which polls the router using SNMP? Have you installed this app about the same time you started having problems with the latency? If the answer to any of the questions is yes, can you stop the app and see if the latency issue dissapears?

Alternatively, you can disable the snap on the router with the command no snmp-server community public