Solved: Extreme and constant 2000-3000 ms Latency through ISR C1111 - Page 2

TheGoob · ‎06-29-2024

Hello

So I have DSL 40/5 Service and for the past 2 years I have had no issues. As of this past 3 weeks I have these insane lag spikes. It could stay normal for 30 minutes or 5 hours, then it happens. My 30ms goes to 2000-3000 ms. It either will not go back down, or will after a while. I have removed every device except the ISRC1111 and the PC plugged into it, same thing. Currently my DSL Modem is in Bridge mode and ISR is doing PPPoE.

To test it, I eliminated everything Cisco and simply used my DSL Modem in PPPoE and 2 days went by 0 issues and my continuous ping stayed 30-60 ms. With that said, when using my ISR C1111 as PPPoE and the lag starts, if I reboot the ISR it all goes back to normal...for 30 mins maybe or several hours.

I really do not know how to troubleshoot this, especially when all in same exact config has been fine. Also, even doing the traceroute and ping ON the ISR, same madness.

What all would be needed from me to help in helping me figure it out?

TheGoob · ‎07-02-2024

Alright, so to answer this, no, I have not installed anything at all like that on any host or any patch or software. I know it’s never the answer that makes sense, but truly I have not changed anything. Which is why the first week or so it as back and forth with my ISP thinking maybe they had dslam issues etc. I’ll disable what you both mentioned and we can go that route.

liviu.gheorghe · ‎07-02-2024

Ok, but don't overlook the IOS upgrade from 17.13.01a to 17.14.01a. When did that take place?

Regards, LG
*** Please Rate All Helpful Responses ***

TheGoob · ‎07-02-2024

Truly could not tell specifics, but it was because of the 2000-3000 ms latency that I upgraded to begin with. I would never have even thought about upgraded had I not had an issue. Cause I know “newer” just means new bugs .

Leo Laohoo · ‎06-29-2024

What firmware is the router on?

TheGoob · ‎07-02-2024

Darn. Still doing it.

TheGoob · ‎07-04-2024

Downgraded again, still Same issue. I mean at this stage it either has to be 1. Defective ISR. 2. Being attacked at random (like 20 times a day) or 3. something in my config.

I can not wrap my head around this and for the life of me I can not figure it out but can anyone refresh my mind or assist me in this;

I want to “unplug” everything out of my ISR and make GE 0/1/7 have a network of let’s say 192.168.20.0 and use the WAN IP (dialer 1?) so I can eliminate everything and see if just the ISR has issues. I’ve done NAT’s different ACL’s etc. I can’t believe it’s so simple but I can’t figure it out!, based on my running-config showed earlier

MHM Cisco World · ‎07-04-2024

do

show cpu history
check if there is peak in history

MHM

TheGoob · ‎07-04-2024

show processes cpu history




      1     11111
  100
   90
   80
   70
   60
   50
   40
   30
   20
   10
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)




                              2
       111 1 111  11  111111117  11 1111   2 1  111 113 1 111 11
  100
   90
   80
   70
   60
   50
   40
   30                         *
   20                         *
   10                         #
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%




      2 47 2     7
      734735222452
  100
   90
   80    *
   70    *       *
   60    *       *
   50    *       *
   40   **       *
   30 * ** *     *
   20 * ** *     *
   10 * ** *    **
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

MHM Cisco World · ‎07-04-2024

That answer you

Cpu is up to 80% make latency' and I suggest before disable snmp did you do that ?

MHM

TheGoob · ‎07-04-2024

I did yes, I did it the moment you mentioned it.

TheGoob · ‎07-04-2024

This just happened like 10 minutes ago 1000ms lag for like 5 minutes.

      111 11  1 211  111 111 1 111  11  111111117  11 1111   2 1
  100
   90
   80
   70
   60
   50
   40
   30                                           *
   20                                           *
   10                                           #
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

MHM Cisco World · ‎07-04-2024

Do

Show ip traffic

Twice

See which count is increasing rapidly'

I think it ARP to make also sure check show arp see how many arp incomplete in table

MHM

TheGoob · ‎07-04-2024

Well currently system "running fine" but here are both.

show arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  172.16.1.1              -   00ea.bded.c4f4  ARPA   Vlan8
Internet  172.16.1.2             48   d0e0.42d9.6e48  ARPA   Vlan8
Internet  192.168.10.1            -   00ea.bded.c4f4  ARPA   Vlan10
Internet  192.168.10.179         27   5254.00f0.439e  ARPA   Vlan10
Internet  192.168.20.1            -   00ea.bded.c4f4  ARPA   Vlan1


show ip traffic
IP statistics:
  Rcvd:  21185157 total, 39955 local destination
         0 format errors, 0 checksum errors, 1 bad hop count
         0 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 0 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other, 0 ignored
  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
         95 fragmented, 190 fragments, 37 couldn't fragment
         0 invalid hole
  Bcast: 0 received, 0 sent
  Mcast: 108 received, 0 sent
  Sent:  21122 generated, 14693943 forwarded
  Drop:  0 encapsulation failed, 0 unresolved, 1453 no adjacency
         0 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr
         0 options denied, 0 source IP address zero

ICMP statistics:
  Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 3 unreachable
        29014 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 86 irdp advertisements
        0 time exceeded, 0 info replies
  Sent: 0 redirects, 6437 unreachable, 0 echo, 29014 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp repl                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  ies
        0 info reply, 1 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

UDP statistics:
  Rcvd: 21831 total, 0 checksum errors, 21831 no port 0 finput
  Sent: 0 total, 0 forwarded broadcasts

TCP statistics:
  Rcvd: 18032 total, 1 checksum errors, 3642 no port
  Sent: 14718 total

PIMv2 statistics: Sent/Received
  Total: 0/0, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 0/0
  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  Queue drops: 0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
  DVMRP: 0/0, PIM: 0/0
  Queue drops: 0
  Report VRF mismatch drops: 0

EIGRP-IPv4 statistics:
  Rcvd: 0 total
  Sent: 0 total

NHRP statistics:
   Sent: Total 0
         0 Resolution Request  0 Resolution Reply  0 Registration Request
         0 Registration Reply  0 Purge Request  0 Purge Reply
         0 Error Indication  0 Traffic Indication  0 Redirect Suppress
   Rcvd: Total 0
         0 Resolution Request  0 Resolution Reply  0 Registration Request
         0 Registration Reply  0 Purge Request  0 Purge Reply
         0 Error Indication  0 Traffic Indication  0 Redirect Suppress

OSPF statistics:
  Last clearing of OSPF traffic counters never
  Rcvd: 0 total, 0 checksum errors
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks
  Sent: 0 total
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

BGP statistics:
  Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh, 0 unrecognized
  Sent: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh

ARP statistics:
  Rcvd: 153 requests, 5 replies, 0 reverse, 0 other
  Sent: 6 requests, 171 replies (0 proxy), 0 reverse
  Drop due to input queue full: 0

HoM#show ip traffic
IP statistics:
  Rcvd:  21197205 total, 39985 local destination
         0 format errors, 0 checksum errors, 1 bad hop count
         0 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 0 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other, 0 ignored
  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
         95 fragmented, 190 fragments, 37 couldn't fragment
         0 invalid hole
  Bcast: 0 received, 0 sent
  Mcast: 108 received, 0 sent
  Sent:  21154 generated, 14699671 forwarded
  Drop:  0 encapsulation failed, 0 unresolved, 1453 no adjacency
         0 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr
         0 options denied, 0 source IP address zero

ICMP statistics:
  Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 3 unreachable
        29014 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 86 irdp advertisements
        0 time exceeded, 0 info replies
  Sent: 0 redirects, 6437 unreachable, 0 echo, 29014 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
        0 info reply, 1 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

UDP statistics:
  Rcvd: 21831 total, 0 checksum errors, 21831 no port 0 finput
  Sent: 0 total, 0 forwarded broadcasts

TCP statistics:
  Rcvd: 18062 total, 1 checksum errors, 3642 no port
  Sent: 14750 total

PIMv2 statistics: Sent/Received
  Total: 0/0, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 0/0
  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  Queue drops: 0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
  DVMRP: 0/0, PIM: 0/0
  Queue drops: 0
  Report VRF mismatch drops: 0

EIGRP-IPv4 statistics:
  Rcvd: 0 total
  Sent: 0 total

NHRP statistics:
   Sent: Total 0
         0 Resolution Request  0 Resolution Reply  0 Registration Request
         0 Registration Reply  0 Purge Request  0 Purge Reply
         0 Error Indication  0 Traffic Indication  0 Redirect Suppress
   Rcvd: Total 0
         0 Resolution Request  0 Resolution Reply  0 Registration Request
         0 Registration Reply  0 Purge Request  0 Purge Reply
         0 Error Indication  0 Traffic Indication  0 Redirect Suppress

OSPF statistics:
  Last clearing of OSPF traffic counters never
  Rcvd: 0 total, 0 checksum errors
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks
  Sent: 0 total
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

BGP statistics:
  Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh, 0 unrecognized
  Sent: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh

ARP statistics:
  Rcvd: 153 requests, 5 replies, 0 reverse, 0 other
  Sent: 6 requests, 171 replies (0 proxy), 0 reverse
  Drop due to input queue full: 0

MHM Cisco World · ‎07-04-2024

ARP is OK
there is ICMP count and UDP conut
for UDP count
check
show ip nat statistics <<- share it here
for ICMP there is high number of ICMP toward router ? are you run any IP SLA or EEM ?

MHM

TheGoob · ‎07-04-2024

In truth I do not know what IP SLA or EEM is, so I would have to say no. Only "server" type things I am running are several email servers.

Total active translations: 124 (9 static, 115 dynamic; 124 extended)
Outside interfaces:
  Dialer1, Virtual-Access2
Inside interfaces:
  Vlan1, Vlan8, Vlan10
Hits: 25561608  Misses: 37922
Expired translations: 37807
Dynamic mappings:
-- Inside Source
[Id: 1] access-list 1 interface Dialer1 refcount 0
[Id: 2] access-list 4 pool 179 refcount 2
 pool 179: id 3, netmask 255.255.255.252
        start 207.108.121.179 end 207.108.121.179
        type generic, total addresses 1, allocated 1 (100%), misses 0
[Id: 3] access-list 5 pool 178 refcount 0
 pool 178: id 2, netmask 255.255.255.252
        start 207.108.121.178 end 207.108.121.178
        type generic, total addresses 1, allocated 0 (0%), misses 0
[Id: 4] access-list 6 pool 182 refcount 102
 pool 182: id 4, netmask 255.255.255.252
        start 207.108.121.182 end 207.108.121.182
        type generic, total addresses 1, allocated 1 (100%), misses 0
[Id: 5] access-list 7 pool 177 refcount 2
 pool 177: id 1, netmask 255.255.255.252
        start 207.108.121.177 end 207.108.121.177
        type generic, total addresses 1, allocated 1 (100%), misses 0
[Id: 6] access-list 8 pool 181 refcount 4
 pool 181: id 5, netmask 255.255.255.252
        start 207.108.121.181 end 207.108.121.181
        type generic, total addresses 1, allocated 1 (100%), misses 0
[Id: 7] access-list 9 pool 180 refcount 0
 pool 180: id 6, netmask 255.255.255.252
        start 207.108.121.180 end 207.108.121.180
        type generic, total addresses 1, allocated 0 (0%), misses 0
[Id: 8] access-list DMZ-TO-OUTSIDE pool DMZ_OUT refcount 0
 pool DMZ_OUT: id 7, netmask 255.255.255.252
        start 207.108.121.179 end 207.108.121.179
        type generic, total addresses 1, allocated 0 (0%), misses 0
nat-limit statistics:
 max entry: max allowed 200000, used 115, missed 0
In-to-out drops: 2058  Out-to-in drops: 1046
Pool stats drop: 0  Mapping stats drop: 0
Port block alloc fail: 0
IP alias add fail: 0
Limit entry add fail: 0